fix(metrics)+docs(ops): engine_runs query, Caddyfile capture, deploy runbook (post-incident)#1704
Open
pulzzejason wants to merge 2 commits into
Open
fix(metrics)+docs(ops): engine_runs query, Caddyfile capture, deploy runbook (post-incident)#1704pulzzejason wants to merge 2 commits into
pulzzejason wants to merge 2 commits into
Conversation
…ile + deploy runbook Post 2026-07-02-incident ops hardening (3 small, related items): 1. fix(metrics): the /api/admin/metrics engine_usage raw query used FROM "EngineRun", but the EngineRun model is @@Map("engine_runs") — so it threw 42P01 (relation does not exist) on EVERY call, silently zeroing the metric AND spamming prod logs with prisma:error (the endpoint is polled). Fixed to FROM engine_runs (columns observations/ endedAt are not @Map'd, so they stay camelCase — unchanged). Verified against live prod: the corrected query returns real data (318 runs this month) instead of erroring. 2. ops(caddy): capture the production Caddyfile in-repo (ops/caddy/Caddyfile) — it was an untracked box snowflake. Includes the lb_try_duration retry buffer applied live on 2026-07-02 (holds+retries over the pm2 reload bind-gap instead of 502/503; the single-upstream active health check was removed as counterproductive). 3. docs(runbook): docs/runbooks/prod-deploy-and-proxy.md — documents the cron-*/15 deploy model, rsync --checksum sync, PM2 self-heal, the downtime-reduction ladder (rsync → Caddy retry → future container blue-green), common ops, and the 07-02 incident. No agent-visible data/tools change → no PM Agent config update. Goal: 64b777fb-38cf-4101-a1b0-8df84be80427
Live deploy on 2026-07-02 (dispatched to ship phase-1) measured the actual deploy impact via a 1s health probe through the reload window: - build + rsync phases: 60 consecutive 200s → phase-1 (rsync --checksum) works, no write-storm disruption. - pm2 reload: ~40s outage (9 failed probes) — the single-fork Next.js cold start, longer than the 15s Caddy lb_try_duration, so Caddy only partially covers it. Changes: - cron */15 → */30: each deploy still incurs the ~40s blip, so fewer ticks = fewer blips (≤30 min merge→prod lag; fine with no real users). GitHub scheduled runs are best-effort/delayed regardless of interval, so a tighter cadence just piles up undeployed commits. Note added that '*/45' fires unevenly (:00/:45), not every 45m. - runbook: correct the reload-gap description (few seconds → measured ~40s), note the 15s Caddy retry doesn't fully cover it, and record the zero-downtime handoff (SO_REUSEPORT / blue-green / container) as the real fix — NOT yet built. Goal: 64b777fb-38cf-4101-a1b0-8df84be80427
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three small, related items hardening prod ops after the 2026-07-02 502/503 incident.
1. fix(metrics):
engine_usageraw query hit a non-existent table/api/admin/metricsranFROM "EngineRun", but theEngineRunPrisma model is@@map("engine_runs"). So the raw query threw42P01 relation "EngineRun" does not existon every call — silently zeroing theengine_usagemetric and logging aprisma:erroreach time (the endpoint is polled, e.g. by a memory-capture loop, so it spammed prod logs).FROM "EngineRun"→FROM engine_runs. The columns (observations,endedAt) are not@map'd, so they stay camelCase — unchanged.318runs this month, 1.76M input tokens) instead of erroring to 0.2. ops(caddy): capture the prod Caddyfile in-repo
ops/caddy/Caddyfile— the live/etc/caddy/Caddyfilewas an untracked box snowflake. Captured as source-of-truth, including thelb_try_duration 15sretry buffer applied live on 2026-07-02 (holds+retries over the pm2 reload bind-gap instead of 502/503; the single-upstream active health check was removed as counterproductive).3. docs(runbook):
docs/runbooks/prod-deploy-and-proxy.mdDocuments the
*/15cron deploy model (not per-push), the rsync--checksumsync, PM2 self-heal, the downtime-reduction ladder (rsync → Caddy retry → future container blue-green), common ops commands, how to update the Caddy config safely, and the 07-02 incident.Validation
admin-metricsunit test passes (9); corrected query verified on live prod DB.Agent config
No agent-visible data/tools change → no PM Agent config update.
Goal:
64b777fb-38cf-4101-a1b0-8df84be80427