feat(harness): lb scenario tier — simulate tier/failover/breaker/affinity

[0x0079]

Follow-up to #1223, now rebased onto main (the core affinity/tier fix merged in #1223). The diff is harness-only.

What

Our tests could not realistically verify load-balancing dynamics. This adds a scenario simulator and a harness lb CLI tier so tier selection, mid-request failover, the circuit breaker (trip + timed recovery), health-based exclusion, and session-affinity pin movement can be driven against programmable fake upstreams over a request sequence — and watched request by request.

• internal/server/lbsim.go — shared engine LBSimulator, driving the real ServiceSelector.Select → dispatchWithPriorityFailover path with a deterministic clock. Each attempt feeds both production feedback channels exactly as a real request would: the breaker recorder and Server.reportHealthStatus (status-classified: 429 → rate-limit, 401/403 → immediate auth-unhealthy, 5xx/other → 3-strike).
• Shared clock seam — SetClock/nowFn in loadbalance (breaker.go + health_monitor.go) and routing (affinity TTL). One fake clock drives breaker recovery, health recovery, and strict affinity-TTL expiry together. Production is unchanged (defaults to time.Now).
• cli/harness lb — --file scenario.yaml or --example cascade|flat|grid|single|regression|ratelimit|authflip|crossmodel. Default output is a pencil graph (per-request failover hops + each svc's breaker/health + affinity pin); --table and --json also available.

Alignment with the merged load-balance fixes

Rebasing surfaced two behaviours that landed alongside #1223/#1233; the simulator now models both:

• Strict affinity TTL (bugfix(routing): implement strict affinity TTL and tier-aware session pinning #1223 merged): a lock expires exactly at LockedAt + affinity_secs and an in-window request does not slide it. The affinity time check rides the same fake clock as the breaker/health, so TestLBScenario_AffinityStrictTTLRelock asserts the strict expiry-and-re-lock deterministically (added routing.SetClock/routing.Now; the store's Get/GC and postProcess now use it).
• Cross-model failover (bugfix(failover, tier): missing model info while failover #1233): with different models per tier, a failover hop must dispatch the fallback's own model, not reuse the primary's. TestLBScenario_CrossModelFailover + --example crossmodel show t0/model-a → t1/model-b in the attempt trace.
• Realigned TestFailoverLogging to the renamed failover log stages (failover_retry / failover_exhausted) introduced by bugfix(failover, tier): missing model info while failover #1233 — this test was already red on main.

Example

#3  s1   t0/gpt-4 ✗500  →  t1/gpt-4 ✓200   →  client=200
       state: t0/gpt-4=open/unhealthy   t1/gpt-4=closed/healthy   pin=t0/gpt-4
#4  s1   t1/gpt-4 ✓200   →  client=200
       state: t0/gpt-4=open/unhealthy   t1/gpt-4=closed/healthy   pin=t1/gpt-4

Tests

internal/server/lb_scenario_test.go — A/B/C/D shapes + the original regression + rate-limit/auth-error + cross-model + strict-TTL scenarios, all asserting the captured trace, affinity pin, and breaker/health snapshots. go build ./..., go vet, and the loadbalance / typ / routing / server / harness packages are green.

Known gap (deferred)

G1 — horizontal tactics (random/token/…) are breaker-blind at the selection layer; documented in priority-routing.md and marked as an executable t.Skip in the harness.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MCtGUNwURzSk34PQ8gkjZC