Skip to content

fix(agents): senior-pm delegation reply shape — routing floor 0.75→0.9 (G#393)#1774

Merged
psdjungpulzze merged 3 commits into
mainfrom
goal/strengthen-senior-pm-routing-adherence-for-roadmap-discovery
Jul 3, 2026
Merged

fix(agents): senior-pm delegation reply shape — routing floor 0.75→0.9 (G#393)#1774
psdjungpulzze merged 3 commits into
mainfrom
goal/strengthen-senior-pm-routing-adherence-for-roadmap-discovery

Conversation

@psdjungpulzze

Copy link
Copy Markdown
Contributor

Goal

G#393 (6eeb182f-e1ae-45f7-89a3-f0ccd883aa47): Strengthen senior-pm routing adherence for roadmap/discovery/drift/phase-sequencing intents; raise the eval capability floor from 0.75 toward 0.9. Follow-up to G#352 (PR #1751) and G#350 (PR #1736).

Root cause (measured, not guessed)

Targeted live triage (orch-002/003/005/019, 5 samples each, judge reasoning read) showed the routing family failing on stall replies, not wrong-specialist choice:

  • orch-002: 0/5 — every sample was "I'll pull the current project state first, then route this to the right specialist" (no specialist named, no delegation visible). Induced by the prompt's session-start project_query habit (rule 1).
  • orch-003: 0/5 — same unnamed-specialist stall; one sample fabricated <function_calls> syntax and invented the specialist's entire deliverable.
  • orch-005 / orch-019: 4/5 each — flips were the same stall pattern.

The eval judge sees only the agent output (never the scenario input), so a delegation that doesn't name its specialist is unscoreable — and in production the same stall is simply a worse reply.

Fix (src/server/agents/configs/senior-pm.ts, additive — G#352 invariants untouched)

  • New "Delegation reply shape" block in ROUTING_RULES: the first reply IS the delegation — name the exact specialist, one-line reason, *[Delegated to …]* marker; bans stall replies, pre-routing project_query, and fabricated delegation narration / invented specialist output.
  • Tie-breakers for overlapping intents: sequencing-over-time vs value-ranking (roadmap vs prioritizer), metric-drop → discovery, baseline-vs-current scope → drift analyst even when a phase is named.
  • Rule 1 gains an explicit routing exception (delegate first; the specialist gathers its own context).

Validation (paid live evals, CI parity)

  • Targeted post-fix rerun: 20/20 samples across the four scenarios (was 9/20).
  • 3 full live runs at EVAL_SAMPLES=5: capability 1.00 / 1.00 / 0.95, canaries 100% in all three. orch-002/003/005/019: zero flips across all runs (done-when tolerated one).
  • The single flip (orch-008, majority 2/5, run 3 only) is a judge-blindness artifact: the scenario's systemContext supplies "Phase 2 is active", but the judge can't see inputs, so restating it reads as data invention. Documented in the floor note; recommended follow-up: pass scenario input to the judge prompt.

Floor

evals/capability-floors.json senior-pm-orchestrator 0.75 → 0.9 (the default — fully reached), note rewritten with the evidence trail. Measured low 0.95 leaves one-flip headroom on the 20-scenario suite.

Agent-maintenance note

This change is the agent config update (prompt-only; no tools added/removed, no TOOL_NAMES or config-sync changes needed).

Tests

  • config-sync / capability-floors / eval-agents-resolver suites: 49 + 7 pass.
  • npm run typecheck clean.

…never stall; floor 0.75→0.9 (G#393)

Live triage (EVAL_SAMPLES=5 per scenario) showed the borderline routing family
(orch-002/003/005/019) failing on STALL replies — 'I'll pull the current project
state first, then route this to the right specialist' with no specialist named
(orch-002 0/5, orch-003 0/5) — not on wrong-specialist choice. The judge sees
only the agent output, so an unnamed delegation is unscoreable.

Fix: a 'Delegation reply shape' block in ROUTING_RULES (name the exact
specialist, one-line reason, delegation marker; no pre-routing project_query;
no fabricated delegation narration or invented specialist output) plus
tie-breakers for the roadmap-vs-prioritizer and drift-vs-roadmap overlaps, and
a routing exception on the session-start project_query habit.

Validation: targeted rerun 20/20 samples on the four scenarios; 3 full live
runs at EVAL_SAMPLES=5 measured capability 1.00 / 1.00 / 0.95 (canaries 100%
in all three; single flip orch-008 = judge-blindness artifact). Floor raised
to the 0.9 default with the evidence trail in the note.
…r-pm-routing-adherence-for-roadmap-discovery
@psdjungpulzze psdjungpulzze merged commit 6afb4ba into main Jul 3, 2026
9 checks passed
@psdjungpulzze psdjungpulzze deleted the goal/strengthen-senior-pm-routing-adherence-for-roadmap-discovery branch July 3, 2026 23:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant