P0-7: instant-api terminationGracePeriodSeconds=35 + preStop (MR-P0-7)

[mastermanas805]

Diagnosis

The api Deployment in k8s/app.yaml was missing two pieces needed for graceful shutdown:

1. terminationGracePeriodSeconds — defaulted to 30s, which collides with
   the api's drain budget: preStop sleep 5 + readinessDrainGrace 3 + ShutdownWithTimeout 25 = 33s of in-process work. Kubelet was SIGKILLing mid-drain.

2. preStop lifecycle hook — without it, the kubelet sends SIGTERM
   immediately on pod termination. The LB doesn't refresh Service endpoints
   until the readinessProbe fails on the next tick — so new traffic kept
   landing on a pod that was about to stop accepting connections.

Diff Summary

k8s/app.yaml:

• New terminationGracePeriodSeconds: 35 on the api pod spec
  (budget: preStop 5s + readinessDrainGrace 3s + shutdownTimeout 25s + safety 2s).
• New lifecycle.preStop.exec.command: ["/bin/sh", "-c", "sleep 5"]
  on the api container — gives the kubelet a window to observe the api's
  /readyz 503 flip (via hooks.Readyz.MarkDraining in the api repo's
  companion PR) and update Service endpoints before SIGTERM is delivered.

Required Companion PR

api repo — ship/p0-7-graceful-shutdown-readiness-2026-05-20
adds MarkDraining to /readyz + wires hooks.Readyz.MarkDraining() into the
SIGTERM handler. Must land together — this manifest change alone widens the
grace period but doesn't flip /readyz to 503, so the LB still routes new
traffic to a draining pod.

Live Verify Plan (post-merge)

1. kubectl apply -f k8s/app.yaml (or whichever path is canonical for infra).
2. kubectl rollout restart deploy/instant-api -n instant
3. kubectl describe pod mid-roll shows preStop running, then probe failing, then container exit.
4. kubectl get events -n instant --sort-by='.lastTimestamp' | tail -20 — no
   'FailedKillPod' / 'killed before terminationGracePeriod' events.

🤖 Generated with Claude Code