feat(eval): SOTU gold fixture 2026-07-10 + span/attribution audit (gold-side)#23
Open
aRealGem wants to merge 1 commit into
Open
feat(eval): SOTU gold fixture 2026-07-10 + span/attribution audit (gold-side)#23aRealGem wants to merge 1 commit into
aRealGem wants to merge 1 commit into
Conversation
…ld-side) Integrates the architect's 15-claim manually-verified SOTU gold fixture (written VERBATIM) with hygiene tooling. Gold-side ONLY — must never enter model context; verified no loader globs the claim-set dir into prompts (contamination guard holds). - sotu_gold_fixture_2026-07-10.json — payload verbatim (15 claims, 7-value interim enum). - resolve_fixture_spans.py + .offsets.json — read-only span resolution (15/15 resolved; trump2026-03 paragraph-level, trump2026-05 confirmed cross-speech misattribution -> biden_2022:0325) and a 277-set attribution audit (277/277 match; the Dominican-officers claim is correctly under biden_2022, so NO repo metadata fix is needed). - HANDOFF_*.md — taxonomy mapping (interim -> VerdictLabel v0.2.0; normative_with_true_premise does not map cleanly), citation-hygiene table (resolved primary URLs), contamination-guard note, and proposed follow-up cards (not created). No merges (Jackie), no kanban writes, no corpus mutations. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GNg63eCE6Q7dNJLm7hxSev
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Integrates the architect's manually-verified 15-claim SOTU gold fixture (written verbatim) plus read-only hygiene tooling. Gold-side only — never enters model context. No merge (Jackie's call).
Deliverables (full detail in
HANDOFF_sotu_gold_fixture_2026-07-10.md)eval/benchmarks/claim-set/sotu_gold_fixture_2026-07-10.json(verbatim). Contamination guard verified: no loader globs the claim-set dir into prompts.VerdictLabel(6-label). Clean for 5 values;true_at_utterance/unverifiable_personal_testimonyare lossy;normative_with_true_premisedoes not map (canonical is factual-only) — flagged for Precious.resolve_fixture_spans.py+.offsets.json): 14 exact sentence-level;trump2026-03paragraph-level ("1826" in adjacent sentence);trump2026-05cross-speech →biden_2022:0325(confirms the misattribution flag).biden_2022:0325, correctly attributed — no repo metadata fix needed; the misattribution is fixture-side only.resolve_url:trueentries (statutes/treaty/congress/CBO/BEA/BMJ confident; a few flagged for exact-issue confirmation). Proposed in the HANDOFF, not written into the verbatim fixture.Proposed (not created) follow-up cards: fixture↔PR#22
verdict_goldreconciliation; fixture r2 (re-key trump2026-05, fold URLs); wire the attribution audit into CI.Suite unaffected (data + standalone gold-side script).
🤖 Generated with Claude Code