Recalibrate the live agent-eval gate: capability-only metric + measured per-agent floors

[psdjungpulzze]

Problem

The agent eval gate (live) job hard-blocked every PR touching senior-pm.ts (#1731, G#333's PR). Evidence from four consecutive runs of one unchanged commit: scores 0.80/0.84/0.88/0.86-ish against the 0.9 bar, with nine different scenarios failing across runs (each passing in at least one other run) and canaries at 100% every time — judge variance around a true baseline that sits below the bar, not regressions.

Root causes (three, compounding)

1. The "capability" metric wasn't capability-only. passRate divides all passes by all 25 scenarios, so the 5 always-must-pass canaries inflate it ~4 points; the real capability-only scores were 15–17/20 (0.75–0.85).
2. The absolute 0.9 bar sits above the senior-pm prompt's measured baseline. All 5 artifact-confirmation scenarios + 4 routing scenarios are per-sample borderline; majority-of-3 (already on) can't stabilize p≈0.5–0.8 scenarios.
3. The designed regression-baseline safety net never functioned. evals/nightly.ts crashes with ENOENT on a stale AGENTS entry (calendar-detection vs the actual calendar-customer-interview-detection scenario file) before ever writing evals/results/ — so the directory is empty, no baseline was ever recorded, and the 5%-regression comparison never ran.

Fix

• capabilityPassRate (capability scenarios only) added to the runner; the gate asserts it against a per-agent calibrated floor from the new evals/capability-floors.json — default stays 0.9, senior-pm-orchestrator floored at 0.70 with the full measurement rationale inline (meant to be raised once confirm-before-create adherence improves in the prompt). Canaries keep their strict 100% gate.
• EVAL_SAMPLES 3 → 5 (samples run concurrently — no wall-time cost); the $0.50/scenario cost-cap test now normalizes per sample since the runner sums sample costs.
• Nightly repair: agents filtered through evalAssetsExist (same skip-with-warn as the PR gate) so one stale entry can't kill the run before writeFileSync; the summary's passedScenarios/capabilityCount mismatch fixed. The next scheduled nightly should finally seed evals/results/ and activate regression detection.
• Tests: structural validation of the floors file (known agents only, notes required, 0.9 default) + a capability-floors unit suite.

Follow-up worth its own goal

All five artifact-confirmation scenarios being borderline is a real behavioral signal about the (much-grown) senior-pm prompt — strengthening confirm-before-create adherence would let the 0.70 floor be raised back toward 0.9.

Agent config

No agent-visible data or tools changed — CI/eval-harness only, so no agent config update is needed.

Test plan

• npx vitest run tests/agent-eval/ — 281 passed (structural + new unit suites)
• npx tsc --noEmit clean; eslint clean on changed files
• This PR itself changes evals/judge/ + evals/lib/, so the harness-canary rule runs the live senior-pm eval against the new gate — a live validation of the calibrated floor.

Closes G#350.