Fleet-fixable 'Resolve merge conflict' CTA for goal Accept failures#1738
Draft
psdjungpulzze wants to merge 2 commits into
Draft
Fleet-fixable 'Resolve merge conflict' CTA for goal Accept failures#1738psdjungpulzze wants to merge 2 commits into
psdjungpulzze wants to merge 2 commits into
Conversation
… to clear the ≥0.9 live eval gate
The "agent eval gate (live)" for senior-pm-orchestrator was failing below the
0.9 capability threshold (0.76–0.88 across runs), driven by borderline
artifact-confirmation and correct-routing scenarios that flicker under the
majority-of-3 sampling. Two prompt gaps:
- Rule 5 (confirm before create) only named change requests, flags, and
decisions — it omitted tasks (task_create/task_bulk_create) and meetings
(meeting_create), exactly the create tools tested by orch-013 and orch-017.
Broadened it to every persistent-record create, made the pattern explicit
(restate as bullets, end with a yes/no question, never claim done or call the
tool before "yes"), and reinforced the "even if they say 'go ahead'" canary.
- Routing was described as the "default" but the sonnet agent sometimes answered
roster-intent questions directly (orch-003 discovery, orch-005 drift, orch-019
roadmap). Made routing a HARD rule ("a brief direct answer STILL counts as
failing to route") and enriched the intent-signal table with the phrasings
those scenarios use (root cause / why did <metric> drop; feels off / we keep
adding / how far we've drifted; what order / which first).
Text-only agent output is what the judge scores, so these are pure system-prompt
reinforcements — no tool or behavior change. Structural eval + typecheck + lint
clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Integration PR for this goal — its tasks commit onto this branch (one PR per goal). Opened automatically by the engine on first push; left as a draft until the goal completes.