Skip to content

Runbook: resize prod to >=2 cores and plan Postgres/Redis off-box #1665

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
pulzzejason wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Runbook: resize prod to >=2 cores and plan Postgres/Redis off-box #1665

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 17 additions & 1 deletion docs/ops-runbook.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,12 @@

Day-to-day operating procedures for the product-manager service.

### Focused runbooks

| Runbook | Covers |
|---|---|
| [Prod scale-up + Postgres/Redis off-box](runbooks/prod-scale-and-datastore-offbox.md) | Current box sizing (RAM/swap/**CPU cores**); resizing the prod box to ≥ 2 cores; running Postgres **and** Redis off the app box (harden-if-managed / migrate-if-co-located) |

## Service overview

| Component | Runtime | Notes |
Expand Down Expand Up @@ -99,7 +105,8 @@ The deploy pipeline:

### Deploy memory headroom

The prod box runs on **2 GB RAM + 4 GB swap**.
The prod box runs on **2 GB RAM + 4 GB swap** (CPU cores documented in the
[scale-up runbook](runbooks/prod-scale-and-datastore-offbox.md#current-production-sizing)).

| Phase | Before this change | After |
|---|---|---|
Expand All @@ -113,6 +120,11 @@ The PM2 `max_memory_restart: "1G"` guardrail (in `ecosystem.config.cjs`) will gr
restart the app if it leaks past 1 GB before the kernel OOM-killer can act — watch for
unexpected PM2 restarts as a signal that the app is approaching the limit.

> This note is about **RAM headroom** only. The **CPU-core** dimension is orthogonal — a core
> resize does not change RAM headroom or the `1G` cap. For adding cores (and why the web app
> stays fork/1 and cannot use them), see the
> [Prod scale-up runbook](runbooks/prod-scale-and-datastore-offbox.md#part-a--resize-prod-to--2-cpu-cores).

### Manual deploy (emergency / rollback)

Normal deploys are automated via `deploy.yml`. For a manual deploy or rollback:
Expand Down Expand Up @@ -188,6 +200,10 @@ Backups are managed by the hosting provider. Verify:
- Retention ≥ 7 days
- Monthly restore drill (documented in launch checklist issues #15)

> Running Postgres/Redis off the app box (or hardening an already-managed instance —
> backups, connection limits, failover, monitoring): see
> [Part B of the datastore off-box runbook](runbooks/prod-scale-and-datastore-offbox.md#part-b--run-postgres-and-redis-off-the-app-box).

### Pruning jobs

Automated maintenance runs nightly via the `maintenance` queue:
Expand Down
331 changes: 331 additions & 0 deletions docs/runbooks/prod-scale-and-datastore-offbox.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,331 @@
# Runbook: Prod scale-up + Postgres/Redis off-box

Two connected operations for the single production box at `/opt/product-manager/app`:

- **Part A** — resize the prod box to **≥ 2 CPU cores** (records current sizing first).
- **Part B** — run Postgres **and** Redis off the app box (formalize if already managed;
migrate off if co-located).

Both are covered here so the CPU/RAM headroom of the box and the datastores that live on
it are reasoned about together. Linked from [`ops-runbook.md`](../ops-runbook.md).

> This runbook keeps the house style of `ops-runbook.md`: sizing tables, ordered steps,
> and **verification then rollback** for every procedure.

---

## Current production sizing

`ops-runbook.md` documents the box RAM and swap but never stated the CPU-core count. Recorded
here as the canonical sizing line for the prod box:

| Dimension | Value | Source |
|---|---|---|
| RAM | 2 GB | `ops-runbook.md` § Deploy memory headroom; `ecosystem.config.cjs` comment |
| Swap | 4 GB | `ops-runbook.md` § Deploy memory headroom |
| CPU cores | **1 vCPU (assumption)** | Not recorded in the repo — see note below |
| App path | `/opt/product-manager/app` | `deploy.yml`, `rotate-secrets.ts` docs |
| Steady-state RSS | ~486 MB | `ecosystem.config.cjs` comment |

> **CPU-core assumption.** The repo does not pin the core count anywhere (deploy config,
> ecosystem, or existing runbook), so **1 vCPU** is assumed as the smallest sizing consistent
> with the documented 2 GB / 4 GB-swap tier that most single-box providers ship. **Verify
> on the box before resizing** and correct this table:
> ```bash
> ssh root@<DEPLOY_HOST>
> nproc # logical cores available to the process
> lscpu | grep -E '^CPU\(s\)|Core|Socket'
> free -h # confirm 2 GB RAM + 4 GB swap still matches this table
> ```

The CPU-core dimension is **orthogonal** to the RAM guidance in `ops-runbook.md`
§ *Deploy memory headroom*: that note is about steady-state RSS vs. the 2 GB ceiling and the
`max_memory_restart: "1G"` guardrail, and says nothing about cores. A core resize does not
change RAM headroom, and a RAM upgrade does not add cores — size each independently. When you
resize, cross-check both this table and that note so RAM and cores stay in sync.

---

## Part A — Resize prod to ≥ 2 CPU cores

The web app is single-process by design and cannot use extra cores itself (see
[Constraints](#constraints)), so this resize is primarily headroom for the **worker**, for
Prisma/`prisma migrate deploy` bursts during deploys, and for reduced CPU steal during the
`npm ci`-free deploy path. Do not expect the web app's throughput to scale with cores.

### Pre-flight

Run all four checks before touching the provider console.

| # | Check | Command / action | Expected |
|---|---|---|---|
| 1 | Recent DB backup exists | Confirm the provider's daily snapshot ran (Railway/Render/Fly console), or take one now — see [Part B backup](#2-take-a-consistent-backup). | Snapshot < 24 h old, restorable |
| 2 | Both PM2 apps online | `pm2 list` | `product-manager` **and** `product-manager-worker` show `online` |
| 3 | Note in-flight WS tunnels | `pm2 describe product-manager` + check active Machines "Open Terminal" sessions | Record count; a reboot **will** drop these relay sockets (they live in process memory) |
| 4 | Confirm current cores | `nproc` on the box | Record the pre-resize value into the [sizing table](#current-production-sizing) |

> **WS tunnels do not survive a reboot.** `server.ts` holds the daemon↔browser relay sockets
> in process memory (the reason the web app is fork/1 — see Constraints). A provider resize
> that reboots the box drops every in-flight terminal tunnel. There is no graceful drain
> across a reboot; PM2 draining only helps within a live-box reload. Announce the window and
> expect open terminals to reconnect.

### Resize procedure

1. **Announce a maintenance window.** A CPU resize on a single-box provider reboots the VM.
Expected downtime: **~1–3 min** of hard downtime for the reboot (provider-dependent), plus
~10 s for PM2 to bring the web app back to `ready`. In-flight WS tunnels drop and must
reconnect.

2. **Resize at the provider** (whichever hosts this box — the providers `ops-runbook.md`
already names):

| Provider | How to resize cores | Reboot? |
|---|---|---|
| **Railway** | Service → Settings → Resources → raise the vCPU limit | Rolling restart of the service |
| **Render** | Service → Settings → Instance Type → larger plan → Save | Instance replaced/rebooted |
| **Fly.io** | `fly scale vm shared-cpu-2x` (or `performance-*`) `-a <app>`; `fly scale show` to confirm | Machine restarts |

Pick the target tier at **≥ 2 cores** while keeping ≥ 2 GB RAM + 4 GB swap (do not drop
RAM/swap below the [sizing table](#current-production-sizing)).

3. **Reboot behavior.** On reboot the box comes up without the app running (PM2 does not
auto-resurrect unless `pm2 startup` + `pm2 save` were configured — `deploy.yml` runs
`pm2 save` after each deploy, so a `pm2 resurrect` on boot restores the saved list). PM2
graceful reload / `server.ts` connection draining is a **within-live-box** mechanism only;
it does not carry connections across a reboot. Any in-flight HTTP/WS is severed at reboot.

4. **Bring web + worker back up** (if PM2 did not auto-resurrect):
```bash
ssh root@<DEPLOY_HOST>
cd /opt/product-manager/app
set -a && source .env.local && set +a
pm2 resurrect # restore the saved process list, OR:
pm2 startOrReload ecosystem.config.cjs --update-env
pm2 save
```
This re-applies `ecosystem.config.cjs`: web = `server.js`, fork / 1 instance,
`max_memory_restart: "1G"`; worker = `tsx … src/worker/index.ts`.

### Verification

```bash
# 1. Web app readiness (DB + Redis + JWKS) — expect HTTP 200 and "status":"ok"
curl -i https://<domain>/api/health

# 2. Both apps online under PM2
pm2 list # product-manager AND product-manager-worker → online

# 3. Confirm the new core count landed
nproc # should now be ≥ 2
```

Update the [sizing table](#current-production-sizing) CPU-core row (drop the "assumption"
qualifier) and cross-reference the RAM note per the orthogonality reminder above.

### Rollback

If the larger tier misbehaves (unexpected billing, provider instability, app not healthy):

1. Resize back down to the previous tier via the same provider path (Fly:
`fly scale vm shared-cpu-1x -a <app>`). This reboots again — same downtime + WS-drop
caveat.
2. Bring the apps back up (step 4 above) and re-run [Verification](#verification).
3. If the box itself is unrecoverable, restore from the pre-flight snapshot onto a fresh box
of the previous tier, then trigger a deploy (`gh workflow run deploy.yml`) so CI ships a
matching `node_modules/`.

---

## Part B — Run Postgres and Redis off the app box

### Current state (assumption)

The repo is **inconsistent** about where the datastores live, so state the assumption
explicitly and cover both cases:

| Source | Says |
|---|---|
| [`ops-runbook.md`](../ops-runbook.md) § Service overview | PostgreSQL 16 = **Managed** (Railway/Render/Fly); Redis 7 = **Managed** (Upstash/Railway) — i.e. already off-box |
| [`docs/self-hosting/install.md`](../self-hosting/install.md) | Postgres **co-located** in the `docker compose` stack (`postgres_data` volume, `docker compose exec postgres pg_dump`) |

- **Case 1 — already managed** (the `ops-runbook.md` prod posture): the datastores are off-box.
The work is to **formalize, verify, and harden** — confirm they are truly external, then
lock down backups, connection limits, failover, and monitoring (see
[Harden an already-managed datastore](#case-1-harden-an-already-managed-datastore)).
- **Case 2 — co-located today** (the self-hosting/compose posture): Postgres (and possibly
Redis) runs on the app box. The work is to **move it off** (see
[Migrate a co-located datastore off-box](#case-2-migrate-a-co-located-datastore-off-box)).

Determine which case you are in before proceeding:
```bash
ssh root@<DEPLOY_HOST>
cd /opt/product-manager/app
grep -E 'DATABASE_URL|REDIS_URL' .env.local # localhost/127.0.0.1/compose-service host = co-located
docker compose ps 2>/dev/null | grep -E 'postgres|redis' || echo "no local compose datastores"
```

### Case 1 — Harden an already-managed datastore

No data movement; formalize the off-box posture.

| Area | What to verify / set | Where |
|---|---|---|
| **Backups** | Daily snapshots enabled, retention ≥ 7 days, monthly restore drill | Provider console; mirrors `ops-runbook.md` § Backups |
| **Connection limits** | App+worker use ~20 conns at idle (Prisma pool 10/process × 2). Set `?connection_limit=&pool_timeout=` under the managed ceiling | `DATABASE_URL` in `.env.local` — see [Connection-limit + pooling](#connection-limit--pooling-guidance) |
| **Failover** | Confirm the plan offers HA/read-replica or documented failover RTO; note it in on-call runbook | Provider plan |
| **Monitoring** | `/api/health` (DB + Redis checks), `/api/health/queues` (depths); provider CPU/conn dashboards; alert on conn-pool exhaustion | `ops-runbook.md` § Healthchecks |
| **Redis** | Confirm managed (Upstash/Railway), TLS `rediss://`, eviction policy `noeviction` (BullMQ requires it), no `FLUSHALL`-capable shared creds | `REDIS_URL` |

Then run [Post-cutover verification](#post-cutover-verification) (no cutover happened, but the
same checks confirm the hardened config is healthy). No rollback needed — no data moved.

### Case 2 — Migrate a co-located datastore off-box

Ordered, low-downtime cutover. Postgres is the stateful piece; Redis is queue/SSE state and
can largely be **re-created** (drain first).

#### 1. Choose + provision the target managed provider

Pick a provider `ops-runbook.md` already names — **Postgres**: Railway / Render / Fly;
**Redis**: Upstash / Railway. Provision:

- Postgres 16 (match the co-located major version — `docker compose exec postgres postgres --version`).
- Redis 7 with `maxmemory-policy noeviction` (BullMQ jobs must not be evicted).
- Record the new `DATABASE_URL` and `REDIS_URL` (use TLS: `sslmode=require` / `rediss://`).

#### 2. Take a consistent backup

Postgres — dump per `install.md`:
```bash
# from the app box, against the co-located container
docker compose exec postgres pg_dump -U postgres interactor_build \
> backup-$(date +%F).sql
```
Redis — **do not** dump/restore as a source of truth. It holds BullMQ queues + SSE pub/sub
state, which is re-creatable. Instead **drain the queues first** so no in-flight jobs are lost
at cutover:
- Stop the worker so no new jobs are picked up: `pm2 stop product-manager-worker`.
- Let active queues finish, or drain via a one-off BullMQ `queue.drain()` script (preferred
over raw `redis-cli DEL` — see `ops-runbook.md` § Redis).
- The SSE connection counter (`pm:sse:connections`) is ephemeral and resets to 0 on the new
instance — no migration needed.

#### 3. Restore Postgres into the managed instance

```bash
psql "<new-managed-DATABASE_URL>" < backup-$(date +%F).sql
# sanity: row counts match the source for a few core tables
psql "<new-managed-DATABASE_URL>" -c "SELECT count(*) FROM \"Organization\";"
```

#### 4. Cutover `DATABASE_URL` / `REDIS_URL`

Use the same env mechanism the deploy pipeline uses — the `update_env` helper writes to the
host's `.env.local`, then `pm2 … --update-env` reloads with the new values (see
`ops-runbook.md` § UI lifecycle feature flags and `deploy.yml`'s PM2 step):

```bash
ssh root@<DEPLOY_HOST>
cd /opt/product-manager/app

# Back up the current env first
cp .env.local .env.local.bak.$(date +%F)

# Point at the managed instances (edit the two lines in place, keeping the pooling params
# from step 5). For a durable change, also update the deploy secret so the next deploy
# does not revert it: gh secret set DATABASE_URL / REDIS_URL.
# DATABASE_URL="postgresql://…@<managed-host>/…?sslmode=require&connection_limit=…&pool_timeout=…"
# REDIS_URL="rediss://…@<managed-host>:…"

set -a && source .env.local && set +a
pm2 startOrReload ecosystem.config.cjs --update-env
pm2 save
```

#### 5. Connection-limit + pooling guidance

Prisma opens a pool of **10 connections per process** by default. The web app **and** the
worker each run their own Prisma client, so expect **~20 connections at idle**, more under
load. Managed Postgres tiers cap total connections (often 20–100 on small plans), and
`prisma migrate deploy` during deploys opens its own short-lived connection on top.

Set the limit against the managed ceiling, leaving headroom for migrations + admin tools:
```
DATABASE_URL="postgresql://…?connection_limit=5&pool_timeout=10"
```
- `connection_limit` is **per process**, not global — with app + worker, `5` each ≈ 10 total
steady, ~14 with a migration/psql session. Keep the total under the managed ceiling.
- If the managed plan is very small, front it with the provider's pooler (PgBouncer /
transaction mode) and set `connection_limit=1` + `pgbouncer=true` in the URL.
- This mirrors `ops-runbook.md` § Database → Connection limits — keep both consistent.

#### 6. Low-downtime cutover sequence

Ordered to minimize the write-gap:

1. Pre-provision + restore (steps 1–3) **ahead of the window** — the long part happens with
the app still live on the old DB.
2. Announce a short window (a few minutes).
3. **Quiesce writes:** `pm2 stop product-manager-worker` (stops job-driven writes), then
take a **final incremental** `pg_dump` and load only the delta (or accept a brief full
re-dump if the DB is small). For a small single-team DB, a stop-dump-restore-switch cycle
is simplest and fastest.
4. Flip `DATABASE_URL` / `REDIS_URL` and reload (step 4).
5. **Restart the worker:** `pm2 start product-manager-worker` (or it comes back via the
`startOrReload`).
6. Verify (below). Total hard-downtime for a small DB is typically **< 2 min**.

#### Post-cutover verification

```bash
# 1. DB + Redis dependency checks — expect HTTP 200 and "status":"ok"
curl -i https://<domain>/api/health

# 2. Queue depths healthy after worker restart — expect "status":"ok"
curl -H "Authorization: Bearer $ADMIN_API_SECRET" https://<domain>/api/health/queues

# 3. Both apps online, no restart-looping
pm2 list
```
`/api/health` runs the `database` (`SELECT 1`) and `redis` (`PING`) checks live, so a 200 with
`"status":"ok"` confirms the app is talking to the **new** managed instances. Watch
`/api/health/queues` depths settle after the worker drains the backlog.

#### Rollback

The old instance is untouched until you delete it — keep it until verification is solid.

1. Revert env: `cp .env.local.bak.<date> .env.local` (and revert the `gh secret set` values
if you changed them), then `set -a && source .env.local && set +a` +
`pm2 startOrReload ecosystem.config.cjs --update-env`.
2. This points `DATABASE_URL` / `REDIS_URL` back at the old co-located instance. Restart the
worker; re-run [Post-cutover verification](#post-cutover-verification).
3. Only decommission the old Postgres container / volume **after** the managed instance has
run clean for at least one full backup cycle.

---

## Constraints

These invariants come from `ecosystem.config.cjs` and the 2026-06-10 incident fix — do not
violate them while scaling.

- **The web app stays `exec_mode: "fork"` / `instances: 1`.** `server.ts` holds the
daemon↔browser WebSocket relay sockets in **process memory**; a multi-worker cluster would
land the two ends of a tunnel on different workers and never bridge them (Machines "Open
Terminal" → Caddy 502). **Never** recommend PM2 cluster mode for the web app, even after
adding cores.
- **Using the extra cores** is therefore scoped to the **worker** (BullMQ concurrency, or a
second worker instance if queue depth justifies it) or left as an **optional follow-up** —
not the web app. The web app being single-process is a correctness requirement, not a
tuning choice.
- **`max_memory_restart: "1G"` semantics are unchanged.** It is a runaway guardrail (~2× the
~486 MB steady state, well under the 2 GB box), not a normal-footprint trip. Adding cores
does not change RAM footprint — leave the 1G cap as-is.

---

_Docs-only change: no PM-Agent config update needed — this runbook adds no agent-visible
tools, data, or workflows (nothing under `src/app/api/tools/` or `src/server/agents/configs/`
changes)._
Loading