fix(metrics)+docs(ops): engine_runs query, Caddyfile capture, deploy runbook (post-incident)

.github/workflows/deploy.yml

@@ -2,10 +2,17 @@ name: Deploy to production
 
 on:
   schedule:
-    # Every 15 minutes (UTC). Deploys only when main has new commits since the
+    # Every 30 minutes (UTC). Deploys only when main has new commits since the
     # last deploy — see the `gate` job. Replaces deploy-on-every-push so a burst
-    # of merges ships as one deploy at the next tick (≤15 min merge→prod lag).
-    - cron: '*/15 * * * *'
+    # of merges ships as one deploy at the next tick (≤30 min merge→prod lag).
+    #
+    # Why 30, not 15: each deploy still incurs a ~40s reload blip (single-fork
+    # Next.js cold start; see docs/runbooks/prod-deploy-and-proxy.md), so fewer
+    # ticks = fewer blips. GitHub's scheduled runs are also best-effort and get
+    # delayed/dropped under load regardless of interval, so a tighter cadence
+    # mostly just piles up undeployed commits. Do NOT use '*/45' — cron fires it
+    # at :00 and :45 (an uneven 45/15 split), not every 45 min. Use */30 or hourly.
+    - cron: '*/30 * * * *'
   workflow_dispatch:
     inputs:
       ref:

docs/runbooks/prod-deploy-and-proxy.md

@@ -0,0 +1,119 @@
+# Runbook: Production deploy & reverse proxy
+
+How production (`build.interactor.com`) is deployed and fronted, why it's built this
+way, and how to operate it. Written after the 2026-07-02 502/503 incident (see
+§ "Incident 2026-07-02").
+
+## Topology
+
+- **Host:** single VM, `root@build.interactor.com` (4 vCPU / ~8 GB RAM / 157 GB disk).
+- **App:** Next.js run via a custom `server.js` (WebSocket terminal-relay upgrade handler),
+  managed by **PM2** — `exec_mode: fork`, `instances: 1` (the relay holds daemon↔browser
+  sockets in-process, so it cannot be clustered). See `ecosystem.config.cjs`.
+  - `max_memory_restart: 2G`, `node_args: --max-old-space-size=1536` (runaway guardrail;
+    the box was resized 2 GB → 8 GB on 2026-07-02, cap raised 1 GB → 2 GB then).
+  - Background jobs run as a second PM2 app `product-manager-worker` (tsx, BullMQ).
+- **Data:** Postgres + Redis as Docker containers (`product-manager-postgres-1`,
+  `product-manager-redis-1`).
+- **Proxy:** Caddy v2 terminates TLS and reverse-proxies to `localhost:4025`. Config:
+  [`ops/caddy/Caddyfile`](../../ops/caddy/Caddyfile) (source of truth; live copy at
+  `/etc/caddy/Caddyfile`).
+
+## Deploy model — cron, not push
+
+`.github/workflows/deploy.yml` runs on a **`*/30` cron** (not on every push). A `gate`
+job deploys only when `main` has new commits since the last deploy, so **a burst of merges
+ships as ONE deploy at the next 30-minute tick** (≤30 min merge→prod lag). `workflow_dispatch`
+allows a manual deploy of any ref — use it when the cron is lagging (GitHub's scheduled runs
+are best-effort and frequently delayed/dropped under load, so don't rely on punctuality).
+
+The 30-min (vs 15-min) cadence is deliberate: each deploy still incurs a **~40 s reload blip**
+(single-fork Next.js cold start — see "Reducing deploy downtime"), so fewer ticks = fewer blips
+until the zero-downtime handoff lands.
+
+Each deploy: build on the CI runner (`npm ci` → prisma generate → `next build` → compile
+`server.js` → `npm prune --omit=dev`) → **rsync** the artifacts to the box → `prisma migrate
+deploy` + drift check → `pm2 startOrReload`.
+
+### Artifact sync is rsync, not scp-overwrite
+
+The "Sync build artifacts" step uses `rsync -a --checksum --delete` (box-side rsync under
+`ionice -c3 nice -n 19`). **`--checksum` is mandatory** — the runner's fresh `npm ci` rewrites
+every file's mtime, so rsync's default size+mtime check would re-transfer all of `node_modules/`
+every deploy (the ~70 MB/s in-place write storm that caused the 2026-07-02 incident). `--delete`
+is scoped strictly inside `node_modules/` and `.next/` so it can never touch the co-located git
+tree or `.env.local`.
+
+### PM2 self-heal
+
+`pm2 startOrReload` cannot change an online app's `script`/`node_args`/`max_memory_restart`.
+The "Restart PM2" step therefore detects drift (wrong script, or missing the current
+`--max-old-space-size` flag) and does a one-time `pm2 delete` + recreate; steady-state deploys
+take the graceful zero-downtime reload path.
+
+## Reducing deploy downtime
+
+Deploys reload the single web fork. Measured on 2026-07-02: the reload causes a **~40 s outage**
+— not a few seconds. The old fork is killed and the new Next.js fork must cold-start (`app.prepare`)
+before it can serve; on one port there is no overlap, so `:4025` is dead for the whole cold start.
+Mitigations, in order of value/effort:
+
+1. **rsync `--checksum` (shipped):** removes the in-place write storm — the dominant cause of
+   deploy-time 5xx. Verified: 60 consecutive 200s through the build+sync phases of a live deploy.
+2. **Caddy `lb_try_duration 15s` (shipped):** the proxy holds+retries over a *short* reload gap
+   instead of 502/503. NOTE: 15 s does **not** cover the measured ~40 s cold-start gap, so it only
+   partially helps today. The single-upstream active health check was removed (it hard-5xx'd the
+   sole backend during reloads).
+3. **Fewer deploys (shipped):** `*/30` cron (was `*/15`) → the ~40 s blip happens at most every
+   30 min, only when there are new commits.
+4. **Zero-downtime handoff (NOT yet built — the real fix):** let the OLD fork keep serving during
+   the new fork's cold start, then swap. Candidates: `SO_REUSEPORT` (`reusePort: true` on
+   `server.listen`; Node ≥22.12 — the box runs 22.22.2) so old+new co-bind `:4025` and pm2's
+   `wait_ready` overlap actually works; **or** blue-green on a second port with a Caddy upstream
+   flip; **or** a container-image deploy (build image in CI → health-check new container →
+   blue-green behind Caddy → drain old → migrations as a discrete gated step), which subsumes the
+   others and cleans up migrations. Verify pm2 fork-reload actually starts-new-before-killing-old
+   before relying on the `reusePort` variant.
+
+## Common operations
+
+### Check health / stability
+```bash
+ssh root@build.interactor.com
+curl -s -o /dev/null -w "%{http_code}\n" localhost:4025/api/health          # direct
+pm2 jlist | jq -r '.[] | "\(.name) status=\(.pm2_env.status) restarts=\(.pm2_env.restart_time) mem_MB=\(.monit.memory/1048576|floor)"'
+```
+A climbing `restart_time` with low uptime = restarts. Cross-check `dmesg -T | grep -i oom` for
+OOM kills, and the deploy history (a restart at a `*/15` boundary is just a deploy reload).
+
+### Prod ops metrics
+`GET /api/admin/metrics` (Bearer `ADMIN_API_SECRET`, in the box's `.env.local`) returns
+latency / queue / memory / PM2-restart diagnostics. On the box:
+```bash
+set -a; source /opt/product-manager/app/.env.local; set +a
+curl -s -H "Authorization: Bearer $ADMIN_API_SECRET" localhost:4025/api/admin/metrics | jq .system
+```
+
+### Updating the Caddy config
+The live `/etc/caddy/Caddyfile` is not auto-synced from the repo. To change it, edit
+[`ops/caddy/Caddyfile`](../../ops/caddy/Caddyfile) here, then on the box:
+```bash
+sudo cp -a /etc/caddy/Caddyfile "/etc/caddy/Caddyfile.bak-$(date +%Y%m%d-%H%M%S)"   # backup
+sudo vi /etc/caddy/Caddyfile                                                          # apply the repo change
+sudo caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile \
+  && sudo systemctl reload caddy                                                      # validate THEN reload
+```
+Rollback: `sudo cp -a /etc/caddy/Caddyfile.bak-<ts> /etc/caddy/Caddyfile && sudo systemctl reload caddy`.
+
+### Manual deploy / rollback of the app
+- Deploy a specific ref: run the `deploy.yml` workflow via `workflow_dispatch` with `ref`.
+- App rollback: re-deploy the previous good commit (there is no image registry yet — see the
+  container-deploy future work above, which would make rollback an image swap).
+
+## Incident 2026-07-02
+
+Sustained 502/503 storm. Root cause: a cluster of `*/15` cron deploys, each doing a ~15-min
+in-place `node_modules` write storm on the single disk (loadavg spike + iowait) plus a reload
+bind-gap that Caddy (single upstream, no retry) surfaced as 5xx. The box itself was healthy
+(no OOM, no crash-loop). Fixes: rsync `--checksum` sync (#1702) + Caddy `lb_try_duration`.
+Durable follow-up: the container-image deploy above.

ops/caddy/Caddyfile

@@ -0,0 +1,46 @@
+# Production reverse-proxy config for build.interactor.com
+#
+# This is the SOURCE OF TRUTH for the prod Caddy config. The live file lives at
+# /etc/caddy/Caddyfile on the deploy host (root@build.interactor.com) and is NOT yet
+# auto-synced — if you change it here, apply it on the box (see
+# docs/runbooks/prod-deploy-and-proxy.md § "Updating the Caddy config"). Keep the two
+# in lockstep so the proxy stops being an untracked snowflake.
+#
+# Caddy v2.11.2.
+
+build.interactor.com {
+    reverse_proxy localhost:4025 {
+        # Hold-and-retry over a brief upstream blip (e.g. the pm2 reload bind-gap on the
+        # single-fork web process) instead of returning 502/503 to the client. With only
+        # one upstream, an ACTIVE health check (health_uri) would instead mark the sole
+        # backend down for up to its interval and hard-5xx during every reload — so it is
+        # deliberately NOT used here; retry (+ Caddy's default passive behavior) smooths
+        # reloads. Revisit when a container/blue-green swap adds a SECOND upstream: then
+        # re-add per-upstream active health checks so traffic drains off the old one.
+        lb_try_duration 15s
+        lb_try_interval 250ms
+
+        header_up X-Real-IP {remote_host}
+        header_up X-Forwarded-For {remote_host}
+        header_up X-Forwarded-Proto {scheme}
+    }
+
+    log {
+        output file /var/log/caddy/interactor-build.log {
+            roll_size 10mb
+            roll_keep 5
+            roll_keep_for 720h
+        }
+        format json
+    }
+
+    header {
+        X-Content-Type-Options nosniff
+        X-Frame-Options DENY
+        X-XSS-Protection "1; mode=block"
+        Referrer-Policy strict-origin-when-cross-origin
+        -Server
+    }
+
+    encode gzip zstd
+}

src/app/api/admin/metrics/route.ts

@@ -111,6 +111,12 @@ async function collectAiTokenStats() {
 // instead of Prisma's `_sum`. Scoped to runs ended this month (matching ai_tokens) and to
 // rows that actually carry the key, so an old run without it contributes nothing. The
 // `::bigint` sums come back as JS BigInt — coerced to Number for JSON serialization.
+//
+// NOTE: a raw query must use the real DB identifiers, NOT the Prisma model name. The
+// EngineRun model is `@@map("engine_runs")`, so the table is `engine_runs` — querying
+// `FROM "EngineRun"` throws `relation "EngineRun" does not exist` (42P01) on every call
+// (the endpoint is polled, so this spammed prod logs and silently zeroed engine_usage).
+// The column names (`observations`, `endedAt`) are NOT @map'd, so they stay camelCase.
 async function collectEngineUsageStats() {
   const start = new Date();
   start.setDate(1);
@@ -134,7 +140,7 @@ async function collectEngineUsageStats() {
         COALESCE(SUM((observations -> 'claudeUsage' ->> 'cacheReadTokens')::bigint), 0)::bigint AS cache_read_tokens,
         COALESCE(SUM((observations -> 'claudeUsage' ->> 'cacheWriteTokens')::bigint), 0)::bigint AS cache_write_tokens,
         COALESCE(SUM((observations -> 'claudeUsage' ->> 'costUsd')::double precision), 0) AS cost_usd
-      FROM "EngineRun"
+      FROM engine_runs
       WHERE "endedAt" >= ${start}
         AND jsonb_exists(observations, 'claudeUsage')
     `);