Extend the AI-coding-verifier corpus: email tool, CI-gate removal, suppression#155
Merged
Merged
Conversation
…ppression The benchmark/ai-coding-verifier corpus deliberately asserts base/head scenarios against the real engine (no fragile golden trees). It covered refund + policy-edit + two docs-only cases; add three canonical capability transitions: - agent_adds_email_tool: an external-communication action is a gated capability change (action_added detected; not auto-mergeable). - agent_removes_ci_gate: deleting the Shipgate CI workflow touches a trust root / weakens policy and routes to human review — the gate cannot be removed to self-merge (the flagship anti-bypass case). - agent_adds_suppression: adding a checks.ignore touches a trust root; the agent cannot silently suppress and self-merge. (Surfaces as trust_root_touched, not policy_weakened, because the suppressed check has no active blocker here.) All assertions reflect real engine output (confirmed by running). README table updated. Test-only + docs; no engine change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses review of #155: the three new scenarios passed on generic signals (trust_root_touched, "email" in subject), so a regression in the specific check each scenario is named for would not be caught. Tightened to the actual check each transition fires (confirmed by probing the real engine): - agent_adds_email_tool: merge_verdict == blocked + blocker SHIP-ACTION-EXTERNAL-COMMUNICATION-AUDIT-MISSING. - agent_removes_ci_gate (renamed _blocks): merge_verdict == blocked + blocker SHIP-VERIFY-CI-GATE-REMOVED. - agent_adds_suppression: merge_verdict == human_review_required + review_item SHIP-VERIFY-BASELINE-OR-WAIVER-EXPANDED + policy_broadened change naming suppression:SHIP-POLICY-APPROVAL-MISSING. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Extends the
benchmark/ai-coding-verifiercorpus (the deterministic base/head merge-verdict scenarios intests/test_verifier_scenarios.py) with three canonical capability transitions the product claims to handle but the corpus didn't yet cover.The corpus deliberately asserts semantic verifier.json fields against the real engine rather than committing golden trees (per its README: "rather than committing fragile golden trees") — so this follows that pattern, not a parallel golden-file one.
New scenarios
agent_adds_email_toolmessaging.send_customer_emailtoolaction_added;can_merge_without_human: falseagent_removes_ci_gate.github/workflows/agents-shipgate.ymltrust_root_touched/policy_weakened; not auto-mergeable — the flagship anti-bypass caseagent_adds_suppressionchecks.ignoretoshipgate.yamltrust_root_touched; the agent can't silently suppress and self-mergeAll three assertions reflect the real engine output (confirmed by running). One honest note, recorded in the suppression test: adding a
checks.ignorefor a check with no active blocker surfaces astrust_root_touched, notpolicy_weakened— defensible (the effective gate isn't weakened), and still routed to a human. If you'd want a suppression of an active blocker to register aspolicy_weakened, that's a small follow-up worth considering.Verification
tests/test_verifier_scenarios.py— 7 scenarios pass; ruff clean. Test-only + README; no engine change.🤖 Generated with Claude Code