From c35ef075a2b8e779360e34a64f19da98ffab0400 Mon Sep 17 00:00:00 2001 From: Jason Monroe Date: Thu, 2 Jul 2026 15:05:52 -0700 Subject: [PATCH 1/2] fix(metrics)+docs(ops): correct engine_runs raw query; capture Caddyfile + deploy runbook MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Post 2026-07-02-incident ops hardening (3 small, related items): 1. fix(metrics): the /api/admin/metrics engine_usage raw query used FROM "EngineRun", but the EngineRun model is @@map("engine_runs") — so it threw 42P01 (relation does not exist) on EVERY call, silently zeroing the metric AND spamming prod logs with prisma:error (the endpoint is polled). Fixed to FROM engine_runs (columns observations/ endedAt are not @map'd, so they stay camelCase — unchanged). Verified against live prod: the corrected query returns real data (318 runs this month) instead of erroring. 2. ops(caddy): capture the production Caddyfile in-repo (ops/caddy/Caddyfile) — it was an untracked box snowflake. Includes the lb_try_duration retry buffer applied live on 2026-07-02 (holds+retries over the pm2 reload bind-gap instead of 502/503; the single-upstream active health check was removed as counterproductive). 3. docs(runbook): docs/runbooks/prod-deploy-and-proxy.md — documents the cron-*/15 deploy model, rsync --checksum sync, PM2 self-heal, the downtime-reduction ladder (rsync → Caddy retry → future container blue-green), common ops, and the 07-02 incident. No agent-visible data/tools change → no PM Agent config update. Goal: 64b777fb-38cf-4101-a1b0-8df84be80427 --- docs/runbooks/prod-deploy-and-proxy.md | 105 +++++++++++++++++++++++++ ops/caddy/Caddyfile | 46 +++++++++++ src/app/api/admin/metrics/route.ts | 8 +- 3 files changed, 158 insertions(+), 1 deletion(-) create mode 100644 docs/runbooks/prod-deploy-and-proxy.md create mode 100644 ops/caddy/Caddyfile diff --git a/docs/runbooks/prod-deploy-and-proxy.md b/docs/runbooks/prod-deploy-and-proxy.md new file mode 100644 index 000000000..82c247ffe --- /dev/null +++ b/docs/runbooks/prod-deploy-and-proxy.md @@ -0,0 +1,105 @@ +# Runbook: Production deploy & reverse proxy + +How production (`build.interactor.com`) is deployed and fronted, why it's built this +way, and how to operate it. Written after the 2026-07-02 502/503 incident (see +§ "Incident 2026-07-02"). + +## Topology + +- **Host:** single VM, `root@build.interactor.com` (4 vCPU / ~8 GB RAM / 157 GB disk). +- **App:** Next.js run via a custom `server.js` (WebSocket terminal-relay upgrade handler), + managed by **PM2** — `exec_mode: fork`, `instances: 1` (the relay holds daemon↔browser + sockets in-process, so it cannot be clustered). See `ecosystem.config.cjs`. + - `max_memory_restart: 2G`, `node_args: --max-old-space-size=1536` (runaway guardrail; + the box was resized 2 GB → 8 GB on 2026-07-02, cap raised 1 GB → 2 GB then). + - Background jobs run as a second PM2 app `product-manager-worker` (tsx, BullMQ). +- **Data:** Postgres + Redis as Docker containers (`product-manager-postgres-1`, + `product-manager-redis-1`). +- **Proxy:** Caddy v2 terminates TLS and reverse-proxies to `localhost:4025`. Config: + [`ops/caddy/Caddyfile`](../../ops/caddy/Caddyfile) (source of truth; live copy at + `/etc/caddy/Caddyfile`). + +## Deploy model — cron, not push + +`.github/workflows/deploy.yml` runs on a **`*/15` cron** (not on every push). A `gate` +job deploys only when `main` has new commits since the last deploy, so **a burst of merges +ships as ONE deploy at the next 15-minute tick** (≤15 min merge→prod lag). `workflow_dispatch` +allows a manual deploy of any ref. + +Each deploy: build on the CI runner (`npm ci` → prisma generate → `next build` → compile +`server.js` → `npm prune --omit=dev`) → **rsync** the artifacts to the box → `prisma migrate +deploy` + drift check → `pm2 startOrReload`. + +### Artifact sync is rsync, not scp-overwrite + +The "Sync build artifacts" step uses `rsync -a --checksum --delete` (box-side rsync under +`ionice -c3 nice -n 19`). **`--checksum` is mandatory** — the runner's fresh `npm ci` rewrites +every file's mtime, so rsync's default size+mtime check would re-transfer all of `node_modules/` +every deploy (the ~70 MB/s in-place write storm that caused the 2026-07-02 incident). `--delete` +is scoped strictly inside `node_modules/` and `.next/` so it can never touch the co-located git +tree or `.env.local`. + +### PM2 self-heal + +`pm2 startOrReload` cannot change an online app's `script`/`node_args`/`max_memory_restart`. +The "Restart PM2" step therefore detects drift (wrong script, or missing the current +`--max-old-space-size` flag) and does a one-time `pm2 delete` + recreate; steady-state deploys +take the graceful zero-downtime reload path. + +## Reducing deploy downtime + +Deploys reload the single web fork, which has an unavoidable few-second **bind-gap** (the old +fork must release `:4025` before the new one binds). Mitigations, in order of value/effort: + +1. **rsync `--checksum` (shipped):** removes the in-place write storm — the dominant cause of + deploy-time 5xx. +2. **Caddy `lb_try_duration` (shipped):** the proxy holds and retries a request for up to 15 s + over a reload bind-gap instead of returning 502/503, making the gap invisible to clients. The + single-upstream active health check was removed (it hard-5xx'd the sole backend during reloads). +3. **True zero-downtime (future):** the real fix is a **container-image deploy** — build an image + in CI, health-check the new container, blue-green swap behind Caddy (add a second upstream + + per-upstream health checks), drain the old, and run migrations as a discrete gated step. This + subsumes any bare-metal `SO_REUSEPORT` handoff and cleans up the migration story. Not yet built. + +## Common operations + +### Check health / stability +```bash +ssh root@build.interactor.com +curl -s -o /dev/null -w "%{http_code}\n" localhost:4025/api/health # direct +pm2 jlist | jq -r '.[] | "\(.name) status=\(.pm2_env.status) restarts=\(.pm2_env.restart_time) mem_MB=\(.monit.memory/1048576|floor)"' +``` +A climbing `restart_time` with low uptime = restarts. Cross-check `dmesg -T | grep -i oom` for +OOM kills, and the deploy history (a restart at a `*/15` boundary is just a deploy reload). + +### Prod ops metrics +`GET /api/admin/metrics` (Bearer `ADMIN_API_SECRET`, in the box's `.env.local`) returns +latency / queue / memory / PM2-restart diagnostics. On the box: +```bash +set -a; source /opt/product-manager/app/.env.local; set +a +curl -s -H "Authorization: Bearer $ADMIN_API_SECRET" localhost:4025/api/admin/metrics | jq .system +``` + +### Updating the Caddy config +The live `/etc/caddy/Caddyfile` is not auto-synced from the repo. To change it, edit +[`ops/caddy/Caddyfile`](../../ops/caddy/Caddyfile) here, then on the box: +```bash +sudo cp -a /etc/caddy/Caddyfile "/etc/caddy/Caddyfile.bak-$(date +%Y%m%d-%H%M%S)" # backup +sudo vi /etc/caddy/Caddyfile # apply the repo change +sudo caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile \ + && sudo systemctl reload caddy # validate THEN reload +``` +Rollback: `sudo cp -a /etc/caddy/Caddyfile.bak- /etc/caddy/Caddyfile && sudo systemctl reload caddy`. + +### Manual deploy / rollback of the app +- Deploy a specific ref: run the `deploy.yml` workflow via `workflow_dispatch` with `ref`. +- App rollback: re-deploy the previous good commit (there is no image registry yet — see the + container-deploy future work above, which would make rollback an image swap). + +## Incident 2026-07-02 + +Sustained 502/503 storm. Root cause: a cluster of `*/15` cron deploys, each doing a ~15-min +in-place `node_modules` write storm on the single disk (loadavg spike + iowait) plus a reload +bind-gap that Caddy (single upstream, no retry) surfaced as 5xx. The box itself was healthy +(no OOM, no crash-loop). Fixes: rsync `--checksum` sync (#1702) + Caddy `lb_try_duration`. +Durable follow-up: the container-image deploy above. diff --git a/ops/caddy/Caddyfile b/ops/caddy/Caddyfile new file mode 100644 index 000000000..b8bd5b2ee --- /dev/null +++ b/ops/caddy/Caddyfile @@ -0,0 +1,46 @@ +# Production reverse-proxy config for build.interactor.com +# +# This is the SOURCE OF TRUTH for the prod Caddy config. The live file lives at +# /etc/caddy/Caddyfile on the deploy host (root@build.interactor.com) and is NOT yet +# auto-synced — if you change it here, apply it on the box (see +# docs/runbooks/prod-deploy-and-proxy.md § "Updating the Caddy config"). Keep the two +# in lockstep so the proxy stops being an untracked snowflake. +# +# Caddy v2.11.2. + +build.interactor.com { + reverse_proxy localhost:4025 { + # Hold-and-retry over a brief upstream blip (e.g. the pm2 reload bind-gap on the + # single-fork web process) instead of returning 502/503 to the client. With only + # one upstream, an ACTIVE health check (health_uri) would instead mark the sole + # backend down for up to its interval and hard-5xx during every reload — so it is + # deliberately NOT used here; retry (+ Caddy's default passive behavior) smooths + # reloads. Revisit when a container/blue-green swap adds a SECOND upstream: then + # re-add per-upstream active health checks so traffic drains off the old one. + lb_try_duration 15s + lb_try_interval 250ms + + header_up X-Real-IP {remote_host} + header_up X-Forwarded-For {remote_host} + header_up X-Forwarded-Proto {scheme} + } + + log { + output file /var/log/caddy/interactor-build.log { + roll_size 10mb + roll_keep 5 + roll_keep_for 720h + } + format json + } + + header { + X-Content-Type-Options nosniff + X-Frame-Options DENY + X-XSS-Protection "1; mode=block" + Referrer-Policy strict-origin-when-cross-origin + -Server + } + + encode gzip zstd +} diff --git a/src/app/api/admin/metrics/route.ts b/src/app/api/admin/metrics/route.ts index d9aef5205..4d90c2a6f 100644 --- a/src/app/api/admin/metrics/route.ts +++ b/src/app/api/admin/metrics/route.ts @@ -111,6 +111,12 @@ async function collectAiTokenStats() { // instead of Prisma's `_sum`. Scoped to runs ended this month (matching ai_tokens) and to // rows that actually carry the key, so an old run without it contributes nothing. The // `::bigint` sums come back as JS BigInt — coerced to Number for JSON serialization. +// +// NOTE: a raw query must use the real DB identifiers, NOT the Prisma model name. The +// EngineRun model is `@@map("engine_runs")`, so the table is `engine_runs` — querying +// `FROM "EngineRun"` throws `relation "EngineRun" does not exist` (42P01) on every call +// (the endpoint is polled, so this spammed prod logs and silently zeroed engine_usage). +// The column names (`observations`, `endedAt`) are NOT @map'd, so they stay camelCase. async function collectEngineUsageStats() { const start = new Date(); start.setDate(1); @@ -134,7 +140,7 @@ async function collectEngineUsageStats() { COALESCE(SUM((observations -> 'claudeUsage' ->> 'cacheReadTokens')::bigint), 0)::bigint AS cache_read_tokens, COALESCE(SUM((observations -> 'claudeUsage' ->> 'cacheWriteTokens')::bigint), 0)::bigint AS cache_write_tokens, COALESCE(SUM((observations -> 'claudeUsage' ->> 'costUsd')::double precision), 0) AS cost_usd - FROM "EngineRun" + FROM engine_runs WHERE "endedAt" >= ${start} AND jsonb_exists(observations, 'claudeUsage') `); From 9c4788d1d8f893bd7ab53a7da4ac4929dad1e1ca Mon Sep 17 00:00:00 2001 From: Jason Monroe Date: Thu, 2 Jul 2026 15:40:02 -0700 Subject: [PATCH 2/2] =?UTF-8?q?deploy:=20slow=20cron=20*/15=E2=86=92*/30?= =?UTF-8?q?=20+=20document=20the=20measured=20~40s=20reload=20gap?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Live deploy on 2026-07-02 (dispatched to ship phase-1) measured the actual deploy impact via a 1s health probe through the reload window: - build + rsync phases: 60 consecutive 200s → phase-1 (rsync --checksum) works, no write-storm disruption. - pm2 reload: ~40s outage (9 failed probes) — the single-fork Next.js cold start, longer than the 15s Caddy lb_try_duration, so Caddy only partially covers it. Changes: - cron */15 → */30: each deploy still incurs the ~40s blip, so fewer ticks = fewer blips (≤30 min merge→prod lag; fine with no real users). GitHub scheduled runs are best-effort/delayed regardless of interval, so a tighter cadence just piles up undeployed commits. Note added that '*/45' fires unevenly (:00/:45), not every 45m. - runbook: correct the reload-gap description (few seconds → measured ~40s), note the 15s Caddy retry doesn't fully cover it, and record the zero-downtime handoff (SO_REUSEPORT / blue-green / container) as the real fix — NOT yet built. Goal: 64b777fb-38cf-4101-a1b0-8df84be80427 --- .github/workflows/deploy.yml | 13 +++++++-- docs/runbooks/prod-deploy-and-proxy.md | 40 +++++++++++++++++--------- 2 files changed, 37 insertions(+), 16 deletions(-) diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml index aa1e284a0..f830a7630 100644 --- a/.github/workflows/deploy.yml +++ b/.github/workflows/deploy.yml @@ -2,10 +2,17 @@ name: Deploy to production on: schedule: - # Every 15 minutes (UTC). Deploys only when main has new commits since the + # Every 30 minutes (UTC). Deploys only when main has new commits since the # last deploy — see the `gate` job. Replaces deploy-on-every-push so a burst - # of merges ships as one deploy at the next tick (≤15 min merge→prod lag). - - cron: '*/15 * * * *' + # of merges ships as one deploy at the next tick (≤30 min merge→prod lag). + # + # Why 30, not 15: each deploy still incurs a ~40s reload blip (single-fork + # Next.js cold start; see docs/runbooks/prod-deploy-and-proxy.md), so fewer + # ticks = fewer blips. GitHub's scheduled runs are also best-effort and get + # delayed/dropped under load regardless of interval, so a tighter cadence + # mostly just piles up undeployed commits. Do NOT use '*/45' — cron fires it + # at :00 and :45 (an uneven 45/15 split), not every 45 min. Use */30 or hourly. + - cron: '*/30 * * * *' workflow_dispatch: inputs: ref: diff --git a/docs/runbooks/prod-deploy-and-proxy.md b/docs/runbooks/prod-deploy-and-proxy.md index 82c247ffe..2b219f0e5 100644 --- a/docs/runbooks/prod-deploy-and-proxy.md +++ b/docs/runbooks/prod-deploy-and-proxy.md @@ -21,10 +21,15 @@ way, and how to operate it. Written after the 2026-07-02 502/503 incident (see ## Deploy model — cron, not push -`.github/workflows/deploy.yml` runs on a **`*/15` cron** (not on every push). A `gate` +`.github/workflows/deploy.yml` runs on a **`*/30` cron** (not on every push). A `gate` job deploys only when `main` has new commits since the last deploy, so **a burst of merges -ships as ONE deploy at the next 15-minute tick** (≤15 min merge→prod lag). `workflow_dispatch` -allows a manual deploy of any ref. +ships as ONE deploy at the next 30-minute tick** (≤30 min merge→prod lag). `workflow_dispatch` +allows a manual deploy of any ref — use it when the cron is lagging (GitHub's scheduled runs +are best-effort and frequently delayed/dropped under load, so don't rely on punctuality). + +The 30-min (vs 15-min) cadence is deliberate: each deploy still incurs a **~40 s reload blip** +(single-fork Next.js cold start — see "Reducing deploy downtime"), so fewer ticks = fewer blips +until the zero-downtime handoff lands. Each deploy: build on the CI runner (`npm ci` → prisma generate → `next build` → compile `server.js` → `npm prune --omit=dev`) → **rsync** the artifacts to the box → `prisma migrate @@ -48,18 +53,27 @@ take the graceful zero-downtime reload path. ## Reducing deploy downtime -Deploys reload the single web fork, which has an unavoidable few-second **bind-gap** (the old -fork must release `:4025` before the new one binds). Mitigations, in order of value/effort: +Deploys reload the single web fork. Measured on 2026-07-02: the reload causes a **~40 s outage** +— not a few seconds. The old fork is killed and the new Next.js fork must cold-start (`app.prepare`) +before it can serve; on one port there is no overlap, so `:4025` is dead for the whole cold start. +Mitigations, in order of value/effort: 1. **rsync `--checksum` (shipped):** removes the in-place write storm — the dominant cause of - deploy-time 5xx. -2. **Caddy `lb_try_duration` (shipped):** the proxy holds and retries a request for up to 15 s - over a reload bind-gap instead of returning 502/503, making the gap invisible to clients. The - single-upstream active health check was removed (it hard-5xx'd the sole backend during reloads). -3. **True zero-downtime (future):** the real fix is a **container-image deploy** — build an image - in CI, health-check the new container, blue-green swap behind Caddy (add a second upstream + - per-upstream health checks), drain the old, and run migrations as a discrete gated step. This - subsumes any bare-metal `SO_REUSEPORT` handoff and cleans up the migration story. Not yet built. + deploy-time 5xx. Verified: 60 consecutive 200s through the build+sync phases of a live deploy. +2. **Caddy `lb_try_duration 15s` (shipped):** the proxy holds+retries over a *short* reload gap + instead of 502/503. NOTE: 15 s does **not** cover the measured ~40 s cold-start gap, so it only + partially helps today. The single-upstream active health check was removed (it hard-5xx'd the + sole backend during reloads). +3. **Fewer deploys (shipped):** `*/30` cron (was `*/15`) → the ~40 s blip happens at most every + 30 min, only when there are new commits. +4. **Zero-downtime handoff (NOT yet built — the real fix):** let the OLD fork keep serving during + the new fork's cold start, then swap. Candidates: `SO_REUSEPORT` (`reusePort: true` on + `server.listen`; Node ≥22.12 — the box runs 22.22.2) so old+new co-bind `:4025` and pm2's + `wait_ready` overlap actually works; **or** blue-green on a second port with a Caddy upstream + flip; **or** a container-image deploy (build image in CI → health-check new container → + blue-green behind Caddy → drain old → migrations as a discrete gated step), which subsumes the + others and cleans up migrations. Verify pm2 fork-reload actually starts-new-before-killing-old + before relying on the `reusePort` variant. ## Common operations