fix(agents): senior-pm delegation reply shape — routing floor 0.75→0.9 (G#393)#1774
Merged
psdjungpulzze merged 3 commits intoJul 3, 2026
Conversation
…never stall; floor 0.75→0.9 (G#393) Live triage (EVAL_SAMPLES=5 per scenario) showed the borderline routing family (orch-002/003/005/019) failing on STALL replies — 'I'll pull the current project state first, then route this to the right specialist' with no specialist named (orch-002 0/5, orch-003 0/5) — not on wrong-specialist choice. The judge sees only the agent output, so an unnamed delegation is unscoreable. Fix: a 'Delegation reply shape' block in ROUTING_RULES (name the exact specialist, one-line reason, delegation marker; no pre-routing project_query; no fabricated delegation narration or invented specialist output) plus tie-breakers for the roadmap-vs-prioritizer and drift-vs-roadmap overlaps, and a routing exception on the session-start project_query habit. Validation: targeted rerun 20/20 samples on the four scenarios; 3 full live runs at EVAL_SAMPLES=5 measured capability 1.00 / 1.00 / 0.95 (canaries 100% in all three; single flip orch-008 = judge-blindness artifact). Floor raised to the 0.9 default with the evidence trail in the note.
…r-pm-routing-adherence-for-roadmap-discovery
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Goal
G#393 (
6eeb182f-e1ae-45f7-89a3-f0ccd883aa47): Strengthen senior-pm routing adherence for roadmap/discovery/drift/phase-sequencing intents; raise the eval capability floor from 0.75 toward 0.9. Follow-up to G#352 (PR #1751) and G#350 (PR #1736).Root cause (measured, not guessed)
Targeted live triage (orch-002/003/005/019, 5 samples each, judge reasoning read) showed the routing family failing on stall replies, not wrong-specialist choice:
project_queryhabit (rule 1).<function_calls>syntax and invented the specialist's entire deliverable.The eval judge sees only the agent output (never the scenario input), so a delegation that doesn't name its specialist is unscoreable — and in production the same stall is simply a worse reply.
Fix (
src/server/agents/configs/senior-pm.ts, additive — G#352 invariants untouched)*[Delegated to …]*marker; bans stall replies, pre-routingproject_query, and fabricated delegation narration / invented specialist output.Validation (paid live evals, CI parity)
EVAL_SAMPLES=5: capability 1.00 / 1.00 / 0.95, canaries 100% in all three. orch-002/003/005/019: zero flips across all runs (done-when tolerated one).systemContextsupplies "Phase 2 is active", but the judge can't see inputs, so restating it reads as data invention. Documented in the floor note; recommended follow-up: pass scenario input to the judge prompt.Floor
evals/capability-floors.jsonsenior-pm-orchestrator 0.75 → 0.9 (the default — fully reached), note rewritten with the evidence trail. Measured low 0.95 leaves one-flip headroom on the 20-scenario suite.Agent-maintenance note
This change is the agent config update (prompt-only; no tools added/removed, no
TOOL_NAMESor config-sync changes needed).Tests
config-sync/capability-floors/eval-agents-resolversuites: 49 + 7 pass.npm run typecheckclean.