fix(checkworthy): sharpen A2 prompt for dominant speech-act + truisms#24
Open
aRealGem wants to merge 1 commit into
Open
fix(checkworthy): sharpen A2 prompt for dominant speech-act + truisms#24aRealGem wants to merge 1 commit into
aRealGem wants to merge 1 commit into
Conversation
Layer A (claude-haiku, single-pass, no few-shot examples) let two false positives
through into the check-worthy pool: a normative proposal with an embedded true premise
("let Medicare negotiate ... like the VA already does" -> should be opinion) and an
undisputed truism ("Thomas Jefferson drew his last breath" -> should be unimportant).
Root cause = prompt: it over-weighted "is there a verifiable fact here?" and under-applied
the dominant-speech-act and importance/triviality tests, with zero worked examples.
Fix (prompt-only, still cheap-tier haiku):
- check-worthy now requires NON-OBVIOUS + disputable + consequential.
- opinion explicitly covers proposals/"should"/"let X do Y" and says to judge the
DOMINANT speech-act even when a factual premise is embedded (premise != assertion).
- unimportant explicitly covers undisputed truisms (well-known dates, ceremonial).
- Added 5 few-shot examples incl. the two real failure cases + positive contrasts.
Behavioral validation is the live Layer A eval (next); this is the prompt lever before
any model-tier change (escalate only the ambiguous band to sonnet if it persists).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GNg63eCE6Q7dNJLm7hxSev
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root-causes the two Layer A false positives Jackie caught in the gold set: a normative proposal with an embedded true premise ("let Medicare negotiate ... like the VA already does" → should be opinion) and an undisputed truism ("Thomas Jefferson drew his last breath" → should be unimportant) were both labeled
check-worthy.Diagnosis (evidence in the thread): Layer A runs on claude-haiku (cheap tier), single-pass (the
_labels_*files are shards, not per-claim votes — no adjudication), with a prompt that had zero few-shot examples. It over-weighted "is there a verifiable fact here?" and under-applied the dominant speech-act and importance tests. → a prompt problem, not a model-capability one.Fix (prompt-only, still cheap haiku):
check-worthynow requires NON-OBVIOUS + disputable + consequential.opinionexplicitly covers proposals / "should" / "let X do Y", and says to judge the dominant speech-act even when a factual premise is embedded.unimportantexplicitly covers truisms (well-known dates, ceremonial statements).Regression test pins the guidance + example cases. Behavioral validation is the live Layer A eval (next step) — this is the prompt lever before any model-tier change; if nuanced misses persist we escalate only the ambiguous band to sonnet, not blanket-upgrade (Layer A runs on every sentence). Suite: 866 passed.
🤖 Generated with Claude Code