Skip to content

[v0.8] Add Claim Evidence Schema v3 and semantic review contract #91

Description

@Stahl-G

Parent: #89

Summary

Extend the Claim Ledger schema so v0.8 can represent evidence readiness dimensions without breaking existing v1/v2 claim ledgers. Add a separate semantic review contract for agent-written reviewer outputs.

Motivation

Current Claim entries can trace a statement to a source and carry epistemic fields, but they do not yet encode the dimensions needed for evidence readiness:

  • claim category;
  • source authority;
  • support status;
  • metric scope;
  • structured limitations;
  • verification path;
  • release eligibility.

Without these fields, downstream gates can see that a source exists, but cannot tell whether the source is appropriate for the claim type.

Proposed Claim fields

Add backward-compatible optional/default fields to src/multi_agent_brief/core/schemas.py:

claim_category: str = "general"
source_authority: str = "unknown"
support_status: str = "not_checked"
evidence_status: str = "not_checked"
release_eligibility: str = "eligible_with_warning"
raw_excerpt: str = ""
source_locator: dict[str, Any] = field(default_factory=dict)
metric_scope: dict[str, Any] = field(default_factory=dict)
verification_path: str = ""
structured_limitations: list[dict[str, Any]] = field(default_factory=list)

Keep existing fields, including:

limitations: list[str]
epistemic_type
evidence_relation

Do not force-migrate old ledgers.

Suggested controlled vocabularies

claim_category

- macro_statistic
- official_price_statistic
- exchange_inventory
- market_price_snapshot
- market_fund_flow
- company_event
- policy_event
- legal_trade_remedy
- import_export_statistic
- analyst_synthesis
- branding_or_audience_statement
- general

source_authority

- official_statistical
- official_exchange
- official_company_filing
- official_legal_regulatory
- licensed_terminal
- industry_data_vendor
- reputable_financial_media
- third_party_price_snapshot
- secondary_reprint
- user_supplied
- unknown

support_status

- supported
- partially_supported
- context_only
- unsupported
- needs_official_source
- not_checked

evidence_status

- pass
- conditional
- warning
- blocked
- not_checked

release_eligibility

- eligible
- eligible_with_warning
- blocked_for_research_review
- blocked_for_formal_release

Semantic review contract

Add:

src/multi_agent_brief/semantic_review/__init__.py
src/multi_agent_brief/semantic_review/schemas.py
src/multi_agent_brief/semantic_review/contract.py
tests/test_semantic_review_contract.py

The semantic review contract should validate output/intermediate/semantic_review_report.json.

Required top-level shape:

{
  "schema_version": "multi-agent-brief-semantic-review/v1",
  "release_mode": "research_review",
  "review_status": "completed",
  "judges": [],
  "findings": []
}

Each finding must include:

{
  "finding_id": "SR_001",
  "judge_id": "source_authority_judge",
  "finding_type": "source_authority_insufficient",
  "severity": "high",
  "claim_id": "...",
  "rating": 1,
  "rating_label": "insufficient_for_formal_release",
  "reason": "...",
  "evidence_spans": [],
  "verification_path": "...",
  "release_effect": "block_formal_release"
}

Supported finding types

- source_authority_insufficient
- source_claim_support_partial
- source_claim_support_missing
- metric_scope_missing
- metric_scope_conflict
- official_source_missing
- latest_official_check_missing
- branding_authorization_missing
- legal_or_policy_scope_overbroad
- release_boundary_unclear

Acceptance criteria

  • Existing v1/v2 claim_ledger.json fixtures still load.
  • New Claim fields export to JSON and import from JSON.
  • Unknown/missing new fields do not break legacy runs.
  • Semantic review report contract validates required top-level fields.
  • Semantic review findings require judge_id, finding_type, rating, reason, verification_path, and release_effect.
  • Unknown finding types fail validation.
  • Tests cover valid report, malformed report, missing fields, unknown finding type, and legacy Claim compatibility.

Non-goals

  • Do not implement judge prompts here.
  • Do not implement evidence scoring here.
  • Do not mutate audited_brief.md or claim_ledger.json based on semantic review.
  • Do not claim semantic truth verification.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions