Skip to content

Atomic prod deploy: release-dir + symlink swap to end in-place node_modules/.next churn #1703

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
pulzzejason wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

Atomic prod deploy: release-dir + symlink swap to end in-place node_modules/.next churn #1703

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
337 changes: 337 additions & 0 deletions scripts/deploy/prod-release.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,337 @@
#!/usr/bin/env bash
# Copyright (c) 2026 Interactor, Inc.
# SPDX-License-Identifier: AGPL-3.0-or-later
#
# prod-release.sh — on-box, Capistrano-style release construction for the
# self-hosted PM2 VPS (root@DEPLOY_HOST, /opt/product-manager).
#
# WHY THIS EXISTS
# ---------------
# Today's deploy (.github/workflows/deploy.yml) mutates the SINGLE live dir
# /opt/product-manager/app in place: `git reset --hard origin/main` there, then
# SCP `.next/,server.js,ecosystem.config.cjs,node_modules/` with overwrite:true.
# Because the PM2 worker runs `tsx` from that same tree, it holds
# node_modules/esbuild/bin/esbuild open, so overwriting node_modules/ fails with
# "Text file busy" unless the worker is stopped first. Every deploy therefore
# needs a `pm2 stop product-manager-worker` workaround, and there is no atomic
# swap or ready-made rollback target.
#
# This script replaces that model with a releases/ + shared/ layout:
#
# /opt/product-manager/
# ├── app/ legacy live dir (reused as the shared git source of truth)
# ├── shared/
# │ └── .env.local the ONE persisted env file; symlinked into each release
# ├── releases/
# │ ├── <id>/ a git worktree at the target SHA + SCP'd build artifacts
# │ └── ...
# └── current -> releases/<id> (the live symlink — flipped by the WIRING task,
# NOT here)
#
# Nothing runs from a freshly-built release dir, so SCPing node_modules/ into it
# never hits the busy-binary problem — that is the whole point (foundation for
# `no-worker-prestop`).
#
# SCOPE OF THIS SCRIPT (deliberately does NOT flip `current` or touch PM2)
# -----------------------------------------------------------------------
# The atomic `current` symlink swap and the PM2 reload live in the dedicated
# WIRING task so this script stays independently reviewable + shellcheck-clean.
# The wiring task calls these subcommands, SCPing artifacts in between:
#
# 1. ssh: prod-release.sh prepare <id> [ref] # bootstrap + build release dir
# └─ prints the release dir path on stdout (the SCP target)
# 2. scp: .next/,server.js,ecosystem.config.cjs,node_modules/ -> <release dir>
# 3. ssh: prod-release.sh migrate <id> # prisma migrate + drift check
# 4. (wiring task) flip `current` symlink + pm2 startOrReload
# 5. ssh: prod-release.sh prune # retain last N releases
#
# `all` runs prepare→migrate→prune in one shot for local/manual testing where the
# build artifacts are already present in the release dir.
#
# Every subcommand is idempotent: re-running `prepare` for an existing release id
# is a no-op, `bootstrap` never clobbers an existing shared env, and `prune`
# always protects the deploy target + the currently-live release.

set -euo pipefail

# --- Configuration (env-overridable so the script is testable off-box) --------
DEPLOY_ROOT="${DEPLOY_ROOT:-/opt/product-manager}"
APP_DIR="${APP_DIR:-${DEPLOY_ROOT}/app}"
RELEASES_DIR="${RELEASES_DIR:-${DEPLOY_ROOT}/releases}"
SHARED_DIR="${SHARED_DIR:-${DEPLOY_ROOT}/shared}"
SHARED_ENV="${SHARED_ENV:-${SHARED_DIR}/.env.local}"
# A dedicated shared clone is preferred as the git source of truth; if absent we
# fall back to reusing the legacy app/.git (both satisfy `git worktree add`).
SHARED_REPO="${SHARED_REPO:-${SHARED_DIR}/repo}"
CURRENT_LINK="${CURRENT_LINK:-${DEPLOY_ROOT}/current}"
KEEP_RELEASES="${KEEP_RELEASES:-5}"

# Resolved lazily by resolve_git_src().
GIT_SRC=""

log() { printf '[prod-release] %s\n' "$*" >&2; }
die() { printf '[prod-release] ERROR: %s\n' "$*" >&2; exit 1; }

usage() {
cat >&2 <<'EOF'
Usage: prod-release.sh <command> [args]

Commands:
bootstrap Create releases/ + shared/ and migrate app/.env.local
into shared/.env.local (idempotent no-op if present).
prepare <id> [ref] Bootstrap, then build releases/<id> as a git worktree
checked out at <ref> (default: <id> if it is a SHA,
else origin/main) and symlink shared/.env.local into
it. Prints the release dir path (the SCP target).
migrate <id> From releases/<id>, source shared/.env.local and run
`prisma migrate deploy` + the schema-drift check.
prune [keep] [id] Remove old releases, keeping the newest [keep]
(default: 5). Never removes the deploy target [id] or
the currently-live (`current`-linked) release.
all <id> [ref] prepare -> migrate -> prune (for local testing where
build artifacts are already staged in the release).

Environment overrides: DEPLOY_ROOT, APP_DIR, RELEASES_DIR, SHARED_DIR,
SHARED_ENV, SHARED_REPO, CURRENT_LINK, KEEP_RELEASES.
EOF
}

# Generate a release id when the caller passes none: a UTC timestamp. Generating
# it on the box (a bash `date`, the shell analog of Date.now) is fine — the id is
# just a unique, sortable directory name.
gen_id() { date -u +%Y%m%dT%H%M%SZ; }

# Resolve the git dir used for `git worktree` operations. Prefer a dedicated
# shared clone; otherwise reuse the legacy app working tree. Returns non-zero if
# neither is a git repo (prune tolerates this and falls back to plain rm -rf).
resolve_git_src() {
if git -C "$SHARED_REPO" rev-parse --git-dir >/dev/null 2>&1; then
GIT_SRC="$SHARED_REPO"
elif git -C "$APP_DIR" rev-parse --git-dir >/dev/null 2>&1; then
GIT_SRC="$APP_DIR"
else
return 1
fi
return 0
}

# --- bootstrap: idempotent layout + shared-state migration --------------------
bootstrap() {
mkdir -p "$RELEASES_DIR" "$SHARED_DIR"

# Migrate the existing live env into shared state exactly once. Copy (not move)
# so the legacy app/ keeps working until the wiring task flips `current`.
if [ -f "$SHARED_ENV" ]; then
log "shared env already present: $SHARED_ENV (no-op)"
elif [ -f "${APP_DIR}/.env.local" ]; then
cp "${APP_DIR}/.env.local" "$SHARED_ENV"
log "migrated ${APP_DIR}/.env.local -> $SHARED_ENV"
else
log "WARNING: no shared env and no ${APP_DIR}/.env.local to migrate from"
fi
}

# --- prepare: build a fresh release dir at the target SHA ---------------------
# Args: <id> [ref]
prepare() {
local id="${1:-}"
[ -n "$id" ] || die "prepare requires a release id"
local ref
if [ -n "${2:-}" ]; then
ref="$2"
elif [[ "$id" =~ ^[0-9a-fA-F]{7,40}$ ]]; then
# The id is itself a commit SHA — check that exact commit out.
ref="$id"
else
ref="origin/main"
fi

bootstrap
resolve_git_src || die "no git source of truth (checked $SHARED_REPO and $APP_DIR)"

local rel_dir="${RELEASES_DIR}/${id}"
if [ -d "$rel_dir" ]; then
log "release dir already exists (idempotent no-op): $rel_dir"
else
# Fetch so the target ref/SHA is present. Plain fetch (no --tags): --tags
# refuses to clobber the local prod-deployed marker tag and would abort under
# `set -e` — the same footgun deploy.yml documents.
# Redirect git's own stdout ("HEAD is now at ...") to stderr: this
# subcommand's stdout is a contract — it must carry ONLY the release path so
# CI can capture it as the SCP target.
log "fetching origin into $GIT_SRC"
git -C "$GIT_SRC" fetch --prune origin 1>&2
# Each release is a detached worktree at the target commit, giving it its own
# prisma/, migrations, and src/ (the worker runs `tsx src/worker/index.ts`).
# --detach avoids per-worktree branch collisions across releases.
log "creating worktree $rel_dir @ $ref"
git -C "$GIT_SRC" worktree add --force --detach "$rel_dir" "$ref" 1>&2
fi

# Symlink shared env INTO the release so `--env-file=.env.local` (server + tsx
# worker) and Prisma both resolve it from the release's own directory.
ln -sfn "$SHARED_ENV" "${rel_dir}/.env.local"
log "linked ${rel_dir}/.env.local -> $SHARED_ENV"

# ==========================================================================
# CI INSERTION POINT
# --------------------------------------------------------------------------
# After this subcommand returns, CI SCPs the prebuilt artifacts INTO the
# release dir printed below:
#
# source: ".next/,server.js,ecosystem.config.cjs,node_modules/"
# target: <this path>/
#
# Nothing runs from this dir yet, so overwriting node_modules/ here CANNOT hit
# the "Text file busy" problem — no worker prestop needed.
# ==========================================================================
printf '%s\n' "$rel_dir"
}

# --- migrate: run prisma migrate deploy + drift check FROM the release --------
# Args: <id>
migrate() {
local id="${1:-}"
[ -n "$id" ] || die "migrate requires a release id"
local rel_dir="${RELEASES_DIR}/${id}"
[ -d "$rel_dir" ] || die "release dir missing: $rel_dir (run prepare first)"
[ -e "${rel_dir}/.env.local" ] || die "missing ${rel_dir}/.env.local (run prepare first)"

cd "$rel_dir" || die "cannot cd into $rel_dir"

# Prisma CLI does not auto-load .env.local; source it (via the symlink) so
# DATABASE_URL et al. are in the environment. This runs BEFORE any `current`
# swap so a failed migration never leaves a half-live release.
# shellcheck disable=SC1091
set -a && source ./.env.local && set +a

log "running prisma migrate deploy from $rel_dir"
npx prisma migrate deploy

# Schema-drift check (Prisma 7), mirroring deploy.yml: diff the LIVE prod DB
# against prisma/schema.prisma. Catches "ledger says applied but the DDL never
# took effect" drift that `migrate status` can't see. --from-config-datasource
# reads process.env.DATABASE_URL (sourced above); --exit-code: 0 = no diff,
# 2 = drift, 1 = the check itself errored.
#
# MODE: WARN — surface drift loudly but do NOT fail yet (flip once prod is
# confirmed drift-free). A check ERROR (exit 1) NEVER blocks; only confirmed
# drift (exit 2) blocks when DRIFT_GATE_BLOCK=true, so a broken check can't
# wedge every deploy.
local drift_gate_block="${DRIFT_GATE_BLOCK:-false}"
local drift_out drift_code
drift_out=$(npx prisma migrate diff --from-config-datasource --to-schema prisma/schema.prisma --script --exit-code 2>&1) && drift_code=0 || drift_code=$?
if [ "$drift_code" = "2" ]; then
log "### SCHEMA DRIFT DETECTED ###"
log "Live prod schema differs from prisma/schema.prisma:"
log "----- BEGIN DRIFT (SQL to reconcile prod -> schema) -----"
printf '%s\n' "$drift_out" >&2
log "----- END DRIFT -----"
if [ "$drift_gate_block" = "true" ]; then
die "DRIFT_GATE_BLOCK=true — failing on detected drift."
fi
log "DRIFT_GATE_BLOCK=false — WARN mode, continuing."
elif [ "$drift_code" = "0" ]; then
log "No schema drift: live prod matches prisma/schema.prisma."
else
log "### DRIFT CHECK ERROR (migrate diff exit ${drift_code}) — NOT treated as drift ###"
printf '%s\n' "$drift_out" >&2
log "The drift check itself failed; continuing. Fix the check — it never blocks."
fi
}

# Remove one release dir, unregistering its git worktree metadata so
# .git/worktrees/<id> does not accumulate stale entries.
remove_release() {
local id="$1"
local dir="${RELEASES_DIR}/${id}"
[ -e "$dir" ] || return 0
log "pruning old release: $id"
if [ -n "$GIT_SRC" ] && git -C "$GIT_SRC" worktree remove --force "$dir" 2>/dev/null; then
:
else
rm -rf "$dir"
fi
if [ -n "$GIT_SRC" ]; then
git -C "$GIT_SRC" worktree prune 2>/dev/null || true
fi
}

# --- prune: keep the newest N, always protecting target + live ----------------
# Args: [keep] [protect-id]
prune() {
local keep="${1:-$KEEP_RELEASES}"
local protect_id="${2:-}"
[ -d "$RELEASES_DIR" ] || { log "no releases dir yet: $RELEASES_DIR"; return 0; }

# Best-effort git source; missing is fine (remove_release falls back to rm -rf).
resolve_git_src || GIT_SRC=""

# The currently-live release (what `current` points to) is the immediately-
# previous release once the wiring task swaps in the target — never prune it.
local current_target=""
if [ -L "$CURRENT_LINK" ]; then
current_target="$(basename "$(readlink -f "$CURRENT_LINK")")"
fi

# Release ids newest-first by mtime. find -printf avoids parsing `ls`.
local ids=()
local line
while IFS= read -r line; do
[ -n "$line" ] && ids+=("$line")
done < <(find "$RELEASES_DIR" -mindepth 1 -maxdepth 1 -type d -printf '%T@ %f\n' 2>/dev/null | sort -rn | awk '{print $2}')

local idx=0
local id
for id in "${ids[@]}"; do
idx=$((idx + 1))
if [ "$idx" -le "$keep" ]; then
continue
fi
if [ -n "$protect_id" ] && [ "$id" = "$protect_id" ]; then
log "keeping $id (deploy target, beyond retention window)"
continue
fi
if [ -n "$current_target" ] && [ "$id" = "$current_target" ]; then
log "keeping $id (currently-live release, beyond retention window)"
continue
fi
remove_release "$id"
done
}

main() {
local cmd="${1:-}"
shift || true
case "$cmd" in
bootstrap)
bootstrap
;;
prepare)
prepare "$@"
;;
migrate)
migrate "$@"
;;
prune)
prune "$@"
;;
all)
local id="${1:-$(gen_id)}"
local ref="${2:-}"
prepare "$id" "$ref" >/dev/null
migrate "$id"
prune "$KEEP_RELEASES" "$id"
log "release $id prepared, migrated, and pruned"
;;
-h|--help|help|"")
usage
[ -n "$cmd" ] # empty command -> non-zero exit
;;
*)
die "unknown command: $cmd (try --help)"
;;
esac
}

main "$@"
Loading