feat(eval): expand verdict-gold to 15 sourced rows (Layer B accuracy scoring)#22
Open
aRealGem wants to merge 2 commits into
Open
feat(eval): expand verdict-gold to 15 sourced rows (Layer B accuracy scoring)#22aRealGem wants to merge 2 commits into
aRealGem wants to merge 2 commits into
Conversation
…scoring) Grows the 3-row seed to 15 so Layer B is meaningfully accuracy-scorable: 10 TRUE, 3 MISLEADING, 2 UNVERIFIABLE (10 biden_2022 + 5 trump_2026 check-worthy claims). Every decidable row carries an authoritative source + rationale per the schema; 6 of the source URLs were re-fetched and confirmed directly (Senate roll call, FactCheck.org 2026 SOTU, PGPF/BEA, history.com, WH ARP fact sheet, STAT/BMJ insulin), the rest came from web research and are marked needs_review=true pending multi-annotator adjudication. Conservative by design: dropped claims whose sourcing was shaky or whose events are recent/contested (e.g. "53 record highs", Iran "obliterated", oil/Venezuela, shutdown GDP), and skipped forcing verdicts. Follow-up: the set skews TRUE/MISLEADING — add well-sourced FALSE examples and more UNVERIFIABLE/adjudicated rows. Also makes test_seed_gold_scores_against_devlot_artifact seed-specific (filters to the 3 seed sids) so it survives gold growth. Suite green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GNg63eCE6Q7dNJLm7hxSev
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
…truism)
biden_2022:0210 ('let Medicare negotiate ... like the VA already does') is a normative
proposal -> belongs in Layer A as opinion, not a Layer B verdict. trump_2026:0700
('Thomas Jefferson drew his last breath') is an undisputed truism -> unimportant. Both
were Layer A false positives (see the A2 prompt fix); removing them from the check-worthy
verdict-gold. Gold now 13 rows (8 TRUE / 3 MISLEADING / 2 UNVERIFIABLE).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GNg63eCE6Q7dNJLm7hxSev
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Grows the 3-row seed verdict-gold to 15 rows so Layer B is meaningfully accuracy-scorable. Left for your review — not auto-merged (verdict judgments on politically-sensitive claims; every row is
needs_review=truepending multi-annotator adjudication per the schema).What's in it
15 rows over check-worthy TRAIN claims (10
biden_2022+ 5trump_2026):Every decidable (TRUE/FALSE/MISLEADING) row carries an authoritative source + rationale per
verdict_gold.SCHEMA.md. The 2 UNVERIFIABLE rows are private-anecdote claims (no authoritative source exists — schema-valid, emptysources).Sourcing rigor
Research was done with web search/fetch under strict "real-URL-only" rules, then I personally re-fetched and confirmed 6 of the load-bearing source URLs: the Senate roll-call (H.R.1, 0 Democratic Yea), FactCheck.org's 2026 SOTU page, PGPF/BEA GDP, history.com (Jefferson), the WH ARP fact sheet, and STAT/BMJ insulin costs. The remaining sources came from the research pass and are flagged
needs_review.Conservative by design: I dropped claims with shaky sourcing or recent/contested events ("53 record highs", Iran "obliterated", oil/Venezuela, shutdown GDP, MFN drug prices) rather than force a verdict.
Please sanity-check before merge
trump_2026verdicts especially (0132 all-Dems-voted-against = TRUE via Senate roll call; 0208 inflation = MISLEADING; 0700 Jefferson = TRUE).Also makes
test_seed_gold_scores_against_devlot_artifactseed-specific so it survives gold growth. Suite: 865 passed.🤖 Generated with Claude Code