fix(hetzner): retry Claude native installer with backoff + clear error

docs/hertzner_backlog.md

@@ -180,6 +180,8 @@ node apps/cli/dist/index.js destroy hetzner-smoke2 -y
 Items deferred during implementation or surfaced during Phase 7 live smoke. Each has the workaround documented above.
 
 - **install-box.sh diagnostic mystery (Phase 7).** Across three independent prepare runs on Hetzner Ubuntu 24.04 cx23, `bash -x /tmp/agentbox-install.sh 2>&1 | sudo tee /var/log/agentbox/install.log` consistently truncates the trace exactly at the end of `sudo -u vscode -H bash -lc 'playwright install chromium'`, and no file system changes from script lines beyond that point materialize on the snapshot — yet the install exits 0 and the orchestrator sees success. Direct repro of `bash -x -c "echo A; sudo -u vscode -H bash -lc 'echo X'; echo B"` works correctly, so it's not a `sudo` FD-mangling issue per se. Worked around by reordering `install-box.sh` so all small file-install steps (baked helpers, baked configs, sshd hardening drop-in, login-shell shim, credential pivot) run BEFORE the Chromium download. The reorder makes the snapshot complete. Real diagnosis is open: candidate causes are (a) Playwright/Node closing an inherited FD that propagates to our outer bash, (b) some apparmor/seccomp interaction killing `tee` mid-write, (c) a tmpfs / journald race. Next steps: try `script -q -c '...' /var/log/agentbox/install.log` instead of pipe-tee, or run the install via systemd-run to isolate the session.
+- **RESOLVED (2026-06-27): Claude install silently skipped → box has no `claude`, attach loops on "no server running on /tmp/tmux-1000/default".** `claude.ai` + `downloads.claude.ai` sit behind Cloudflare, which **intermittently** returns **HTTP 403** to cloud-datacenter egress IPs (Hetzner among them) under load — re-testing minutes later returned 200. The bake's `curl -fsSL https://claude.ai/install.sh | bash -s stable` masked this: `curl -f` exits non-zero on the 403 but the pipeline's status is `bash`'s 0, so the step "succeeded" while baking a claude-less snapshot — and every box from it had no `claude`, so the in-box agent tmux session died instantly and `attach` crash-looped. Fixed in `install-box.sh` (and mirrored in vercel `provision.sh` / e2b `build-template.sh` / docker `Dockerfile.box`): a `retry_backoff` helper retries the native installer **3 times with 60s then 240s backoff** (~5 min budget), keeping `set -o pipefail` and folding `command -v claude` into the retried command so a "succeeded but absent" result also retries; if all 3 attempts fail the bake aborts (`exit 71`) — a failed `prepare` beats a claude-less snapshot. `prepareHetzner` special-cases `exit 71` with an actionable error ("native installer unreachable after retries — transient Cloudflare 403 on the datacenter IP, wait and re-run `prepare --force`") instead of the opaque generic "(empty stderr)" message (`bash -x … 2>&1 | tee` merges stderr into stdout, so the captured stderr is always empty). **No npm fallback** (an earlier attempt used one, but `npm install -g @anthropic-ai/claude-code` lacks native-only features the user relies on and lands the binary at `/usr/bin/claude`, which mismatches the host-seeded `installMethod=native` and trips Claude Code's startup "claude command at ~/.local/bin/claude missing or broken" doctor warning). **Action required: rebake the base snapshot (`agentbox prepare --provider hetzner --force`) — the fix only applies to new bakes.**
+  - **KNOWN GAP / deferred follow-up: the 403 can outlast the ~5-min retry window**, so `prepare` can still fail and need a manual re-run. The real reliability fix (validated by PoC, not yet built): **host-proxy the native binary.** `claude.ai/install.sh` just downloads `downloads.claude.ai/claude-code-releases/{version}/{platform}/claude` (a complete ~244 MB self-contained ELF) and runs `<binary> install` (which re-fetches from the same blocked CDN — `claude install` has NO offline mode). PoC confirmed: the downloaded binary, placed directly at `~/.local/bin/claude` (skipping `install`), runs fully offline and satisfies the `installMethod=native` doctor check. So the fix is: during `prepare`, the **host** (always gets 200) downloads the binary for the VPS arch (cx23=`linux-x64`; cax=`linux-arm64`), checksum-verifies against `{version}/manifest.json`, caches it under `~/.agentbox/`, scp's it to the VPS via the runtime-assets push, and `install-box.sh` drops it at `~/.local/bin/claude` (chmod+chown vscode) instead of curling. True native, 100% reliable, no npm. Deferred per user (kept retries + clear error for now); apply to vercel/e2b too if they start 403ing.
 - **`agentbox checkpoint create` says "daytona snapshot" for hetzner boxes (cosmetic).** The progress message lives in shared cloud-provider code that hardcodes "daytona". Cheap fix: parameterize via `backend.name`.
 - **`agentbox checkpoint ls` returns empty from /tmp project dirs.** The macOS `/private/tmp/...` vs `/tmp/...` resolution mismatches the project-hash anchor used at create time. Manifests are correctly written; only the lookup is sensitive.
 - **`/usr/local/bin/chromium` symlink missing on the baked snapshot.** It's the very last command of the Chromium step, AFTER the playwright download — gets eaten by the same install-script truncation mystery above. Workaround: agent-browser falls back to the cached playwright binary path. Real fix: move the `ln -sf` line to an EARLIER step (or after agent-browser install).

packages/sandbox-docker/Dockerfile.box

@@ -32,10 +32,12 @@ ENV DEBIAN_FRONTEND=noninteractive
 # outer terminal regardless; this just unlocks the in-pane side.
 #
 # DISABLE_AUTOUPDATER follows Anthropic's own devcontainer guidance: there's no
-# point updating Claude Code inside a sandbox that's torn down regularly, and
-# leaving it enabled triggers a "installMethod is native, but .local/bin
-# missing" warning when the host's .claude.json (synced in) reports a native
-# install method while the box uses npm-global.
+# point updating Claude Code inside a sandbox that's torn down regularly. The box
+# installs claude via Anthropic's native installer (below), so the binary lands
+# at /home/vscode/.local/bin/claude and installMethod=native (synced in from the
+# host's .claude.json) matches the box — no "installMethod is native, but
+# .local/bin missing" warning, and the autoupdater would otherwise churn the
+# pinned `stable` build.
 #
 # LANG/LC_ALL must be set via ENV (not /etc/environment, which is what the
 # devcontainers base seeds): `docker exec` doesn't source login files, so
@@ -265,7 +267,20 @@ RUN mkdir -p /home/vscode/.vscode-server/extensions /home/vscode/.cursor-server/
 # the host's `.claude.json` (with installMethod=native) expects, so the in-box
 # integrity check doesn't fire.
 USER vscode
-RUN curl -fsSL https://claude.ai/install.sh | bash -s stable
+# Fetch the installer to a file (not `curl | bash`) so a blocked download — e.g.
+# a Cloudflare 403 the CDN intermittently returns under load — fails the `&&`
+# chain instead of being hidden behind bash's exit 0, and retry it (3 attempts,
+# 60s then 240s backoff) before giving up, then verify `claude` is on PATH.
+# (RUN's default shell is /bin/sh, so a plain loop — no pipefail/bash arrays.)
+RUN i=1; while :; do \
+      curl -fsSL https://claude.ai/install.sh -o /tmp/claude-install.sh \
+        && bash /tmp/claude-install.sh stable \
+        && command -v claude >/dev/null && break; \
+      [ "$i" -ge 3 ] && { echo "claude native install failed after 3 attempts" >&2; exit 71; }; \
+      w=60; [ "$i" -ge 2 ] && w=240; \
+      echo "claude install attempt $i failed — backing off ${w}s" >&2; sleep "$w"; \
+      i=$((i+1)); \
+    done; rm -f /tmp/claude-install.sh
 USER root
 
 # OpenAI Codex CLI. The @openai/codex npm package ships platform-native

packages/sandbox-e2b/scripts/build-template.sh

@@ -40,6 +40,24 @@ set -euo pipefail
 step() { printf '\n>>> BEGIN %s\n' "$1"; }
 done_() { printf '<<< END %s\n' "$1"; }
 
+# Retry a command with exponential backoff. Usage: retry_backoff <max> cmd...
+# Waits 60s before attempt 2, 240s before attempt 3 (~5 min total budget). Used
+# for the Claude native installer, whose CDN (claude.ai / downloads.claude.ai,
+# behind Cloudflare) intermittently 403s cloud-datacenter egress IPs under load.
+retry_backoff() {
+  local max=$1; shift
+  local attempt=1
+  local -a waits=(60 240)
+  while true; do
+    if "$@"; then return 0; fi
+    if [ "$attempt" -ge "$max" ]; then return 1; fi
+    local w=${waits[$((attempt-1))]:-240}
+    echo "retry_backoff: attempt ${attempt}/${max} failed — backing off ${w}s" >&2
+    sleep "$w"
+    attempt=$((attempt+1))
+  done
+}
+
 if [ "$(id -u)" -ne 0 ]; then
   echo "build-template.sh: must run as root (got uid $(id -u))" >&2
   exit 64
@@ -258,8 +276,20 @@ npm install -g @openai/codex opencode-ai agent-browser 2>&1 | tail -3 || \
 done_ "agent CLIs (codex + opencode + agent-browser, global npm)"
 
 step "Claude Code (native installer, run as vscode)"
-# Anthropic's canonical installer drops `claude` at /home/vscode/.local/bin/.
-sudo -u vscode -H bash -lc 'curl -fsSL https://claude.ai/install.sh | bash -s stable'
+# Anthropic's native installer drops `claude` at /home/vscode/.local/bin/claude
+# with installMethod=native (matching the host-seeded .claude.json, so the
+# startup integrity check stays quiet) and ships native-only features the npm
+# package lacks. Its CDN (claude.ai / downloads.claude.ai) sits behind
+# Cloudflare, which intermittently 403s cloud-datacenter egress IPs under load —
+# so retry with backoff rather than falling back to npm. A bare `curl | bash`
+# would hide a 403 (curl -f exits non-zero but the pipe's status is bash's 0),
+# so keep pipefail and fold the PATH check in so a "succeeded but absent" result
+# also retries. A failed build is better than a claude-less template.
+if ! retry_backoff 3 sudo -u vscode -H bash -lc \
+     'set -o pipefail; curl -fsSL https://claude.ai/install.sh | bash -s stable && command -v claude >/dev/null'; then
+  echo "build-template.sh: Claude native installer failed after 3 attempts (Cloudflare 403?) — aborting build" >&2
+  exit 71
+fi
 done_ "Claude Code (native installer, run as vscode)"
 
 step "Chrome runtime libs (apt)"

packages/sandbox-hetzner/scripts/install-box.sh

@@ -31,6 +31,24 @@ set -euo pipefail
 step() { printf '\n>>> BEGIN %s\n' "$1"; }
 done_() { printf '<<< END %s\n' "$1"; }
 
+# Retry a command with exponential backoff. Usage: retry_backoff <max> cmd...
+# Waits 60s before attempt 2, 240s before attempt 3 (~5 min total budget). Used
+# for the Claude native installer, whose CDN (claude.ai / downloads.claude.ai,
+# behind Cloudflare) intermittently 403s cloud-datacenter egress IPs under load.
+retry_backoff() {
+  local max=$1; shift
+  local attempt=1
+  local -a waits=(60 240)
+  while true; do
+    if "$@"; then return 0; fi
+    if [ "$attempt" -ge "$max" ]; then return 1; fi
+    local w=${waits[$((attempt-1))]:-240}
+    echo "retry_backoff: attempt ${attempt}/${max} failed — backing off ${w}s" >&2
+    sleep "$w"
+    attempt=$((attempt+1))
+  done
+}
+
 if [ "$(id -u)" -ne 0 ]; then
   echo "install-box.sh: must run as root (got uid $(id -u))" >&2
   exit 64
@@ -351,11 +369,24 @@ npm install -g @openai/codex opencode-ai
 done_ "Codex CLI prereqs (bubblewrap) + agent installs"
 
 step "Claude Code (native installer, run as vscode)"
-# Anthropic's native installer drops `claude` at /home/vscode/.local/bin/.
-# Run as vscode so the binary lands in the right home and is owned by the
-# user that'll execute it. DISABLE_AUTOUPDATER is set globally via
+# Anthropic's native installer is the path the rest of AgentBox expects: it
+# drops `claude` at /home/vscode/.local/bin/claude with installMethod=native
+# (matching the host-seeded .claude.json, so the startup integrity check stays
+# quiet) and ships native-only features the npm package lacks. Run as vscode so
+# the binary lands in the right home; DISABLE_AUTOUPDATER is set globally via
 # /etc/profile.d/agentbox.sh below.
-sudo -u vscode -H bash -lc 'curl -fsSL https://claude.ai/install.sh | bash -s stable'
+#
+# Its CDN (claude.ai / downloads.claude.ai) sits behind Cloudflare, which
+# intermittently 403s cloud-datacenter egress IPs (Hetzner among them) under
+# load. Retry with backoff rather than falling back to npm. A bare `curl | bash`
+# would hide a 403 (curl -f exits non-zero but the pipe's status is bash's 0),
+# so keep pipefail and fold the PATH check in so a "succeeded but absent" result
+# also retries. A failed prepare is better than a claude-less snapshot.
+if ! retry_backoff 3 sudo -u vscode -H bash -lc \
+     'set -o pipefail; curl -fsSL https://claude.ai/install.sh | bash -s stable && command -v claude >/dev/null'; then
+  echo "install-box.sh: Claude native installer failed after 3 attempts (Cloudflare 403?) — aborting bake" >&2
+  exit 71
+fi
 done_ "Claude Code (native installer, run as vscode)"
 
 step "Chromium download via Playwright (as vscode)"