Outer builder container: agent-server + rootless podman image

.dockerignore

@@ -0,0 +1,8 @@
+node_modules
+dist
+docs
+test
+.git
+.gen
+*.md
+!docker/builder/AGENTS.builder.md

README.md

@@ -24,6 +24,27 @@ WORKSPACE_DIR=/abs/path/to/workspace npm start
 
 Dev with watch: `WORKSPACE_DIR=/abs/path/to/workspace npm run dev`.
 
+## Run it in Docker (outer builder container)
+
+The repo ships the outer builder container from
+[`docs/architecture/important/builder-container-architecture.md`](docs/architecture/important/builder-container-architecture.md):
+agent-server plus **rootless podman**, so builder agents can build and run app
+containers (`podman build` / `podman run`) isolated from the host.
+
+```bash
+./docker/builder/run.sh
+# → agent-server on :4001, inner-app ports 3000-3010 forwarded
+```
+
+`run.sh` is the canonical spawn contract (no `--privileged`; `--device
+/dev/fuse` + relaxed seccomp/apparmor for nested user namespaces; named
+volumes for `/workspace` and podman storage). Orchestrators should replicate
+its flags. Provider credentials are passed with `-e` and live only in the
+agent-server process — never in inner containers. The builder system prompt
+(`docker/builder/AGENTS.builder.md`) is baked at
+`/usr/local/share/appx-builder/AGENTS.md` and installed at
+`${WORKSPACE_DIR}/.pi-global/AGENTS.md` when the container starts.
+
 ## Configuration
 
 All via env vars (see `.env.example`):
@@ -71,7 +92,8 @@ WORKSPACE_DIR/
   are centralised under `.pi-global/sessions/{id}/`, so they never leak into a
   project's git history and survive independently on the volume.
 - A project with no `.pi/AGENTS.md` starts with no pinned prompt (silent skip);
-  Pi's normal context-file discovery then applies.
+  Pi's normal context-file discovery then applies, including any global
+  `.pi-global/AGENTS.md` installed by the deployment.
 - LLM credentials are injected from env into memory at startup and are **not** the
   job of the volume to persist (`auth.json` holds only non-secret/OAuth state).
 

docker/builder/AGENTS.builder.md

@@ -0,0 +1,16 @@
+# Appx Builder Environment
+
+You are a builder agent inside the Appx outer builder container.
+
+- Your project lives in a directory under `/workspace/<project-id>` — treat it
+  as the normal editable project root.
+- `podman` is available for building and running app containers:
+  `podman build -t <project>-app .`, `podman run -d --name <project>-app -p 3000:3000 <project>-app`.
+- Publish app ports in the **3000–3010** range only; those ports are forwarded
+  out of this container to the host.
+- Run long-lived apps detached (`-d`), then verify them with `curl` and check
+  `podman ps` / `podman logs <name>` instead of blocking the shell.
+- Rebuild + restart flow: `podman build ... && podman rm -f <name> && podman run ...`.
+- NEVER pass provider credentials (`ANTHROPIC_API_KEY`, `LITELLM_API_KEY`, or
+  any other secret from your environment) into containers, Dockerfiles, or
+  generated app code.

docker/builder/Dockerfile

@@ -0,0 +1,70 @@
+# Outer builder container: agent-server + rootless podman.
+# See docs/superpowers/specs/2026-06-10-outer-builder-container-design.md and
+# docs/architecture/important/builder-container-architecture.md.
+
+# --- Stage 1: build agent-server from this repo ----------------------------
+FROM node:22-bookworm AS build
+WORKDIR /src
+COPY package.json package-lock.json tsconfig.json ./
+ENV HUSKY=0
+RUN npm ci
+COPY src ./src
+RUN npm run build && npm prune --omit=dev
+
+# --- Stage 2: runtime -------------------------------------------------------
+FROM ubuntu:24.04
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+# Node 22 (NodeSource) + rootless podman stack.
+RUN apt-get update \
+  && apt-get install -y --no-install-recommends ca-certificates curl gnupg \
+  && curl -fsSL https://deb.nodesource.com/setup_22.x | bash - \
+  && apt-get install -y --no-install-recommends \
+    nodejs \
+    podman \
+    fuse-overlayfs \
+    uidmap \
+    slirp4netns \
+    passt \
+    netavark \
+    aardvark-dns \
+    catatonit \
+    git \
+  && apt-get clean && rm -rf /var/lib/apt/lists/*
+
+# Non-root builder user with subordinate id ranges for rootless podman.
+# ubuntu:24.04 ships a default "ubuntu" uid-1000 user; replace it.
+RUN userdel -r ubuntu 2>/dev/null || true \
+  && useradd -m -u 1000 -s /bin/bash builder \
+  && echo "builder:100000:65536" > /etc/subuid \
+  && echo "builder:100000:65536" > /etc/subgid
+
+# Rootless podman defaults tuned for running nested inside Docker.
+COPY docker/builder/containers.conf /home/builder/.config/containers/containers.conf
+COPY docker/builder/storage.conf /home/builder/.config/containers/storage.conf
+
+# Builder-agent system prompt template; the entrypoint installs it into
+# WORKSPACE_DIR/.pi-global/AGENTS.md, the shared agentDir used by runtimes.
+COPY docker/builder/AGENTS.builder.md /usr/local/share/appx-builder/AGENTS.md
+
+# agent-server runtime.
+COPY --from=build /src/dist /app/dist
+COPY --from=build /src/node_modules /app/node_modules
+COPY package.json /app/package.json
+COPY docker/builder/entrypoint.sh /usr/local/bin/builder-entrypoint
+
+RUN mkdir -p /workspace /home/builder/.local/share/containers \
+  && chown -R builder:builder /home/builder /workspace /app \
+  && chmod +x /usr/local/bin/builder-entrypoint
+
+USER builder
+ENV HOME=/home/builder \
+    WORKSPACE_DIR=/workspace \
+    AGENT_SERVER_HOST=0.0.0.0 \
+    AGENT_SERVER_PORT=4001
+
+WORKDIR /workspace
+EXPOSE 4001 3000-3010
+
+ENTRYPOINT ["/usr/local/bin/builder-entrypoint"]

docker/builder/containers.conf

@@ -0,0 +1,14 @@
+# Rootless podman defaults for running nested inside an unprivileged Docker
+# container. slirp4netns is the most reliable rootless network backend in
+# nested user namespaces; cgroupfs avoids requiring a systemd user session.
+[containers]
+netns = "slirp4netns"
+cgroup_manager = "cgroupfs"
+log_driver = "k8s-file"
+
+[engine]
+events_logger = "file"
+runtime = "crun"
+
+[network]
+default_rootless_network_cmd = "slirp4netns"

docker/builder/entrypoint.sh

@@ -0,0 +1,24 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# The workspace volume may mount empty; agent-server requires the dir to exist.
+workspace_dir="${WORKSPACE_DIR:-/workspace}"
+global_agent_dir="${workspace_dir}/.pi-global"
+builder_agents_template="/usr/local/share/appx-builder/AGENTS.md"
+
+mkdir -p "${global_agent_dir}"
+
+if [ ! -e "${global_agent_dir}/AGENTS.md" ]; then
+  cp "${builder_agents_template}" "${global_agent_dir}/AGENTS.md"
+fi
+
+# First-run rootless storage init is slow; warm it up. Non-fatal: the REST
+# surface must come up even if nesting is broken — the failure then surfaces
+# in agent tool calls instead of a crash loop.
+if podman info >/dev/null 2>&1; then
+  echo "[builder] rootless podman ready"
+else
+  echo "[builder] WARNING: podman info failed — inner containers will not work" >&2
+fi
+
+exec node /app/dist/server.js

docker/builder/run.sh

@@ -0,0 +1,53 @@
+#!/usr/bin/env bash
+# Build and run the outer builder container. This script is the canonical
+# spawn contract — an orchestrator (appx) should replicate exactly these
+# flags. See docs/superpowers/specs/2026-06-10-outer-builder-container-design.md.
+set -euo pipefail
+
+cd "$(dirname "$0")/../.."
+
+IMAGE="${BUILDER_IMAGE:-appx-builder}"
+NAME="${BUILDER_NAME:-appx-builder}"
+AGENT_PORT="${BUILDER_AGENT_PORT:-4001}"
+APP_PORTS="${BUILDER_APP_PORTS:-3000-3010}"
+WORKSPACE_VOLUME="${BUILDER_WORKSPACE_VOLUME:-appx-workspace}"
+PODMAN_VOLUME="${BUILDER_PODMAN_VOLUME:-appx-podman}"
+
+docker build -t "$IMAGE" -f docker/builder/Dockerfile .
+
+docker rm -f "$NAME" >/dev/null 2>&1 || true
+
+# Notes on flags (no --privileged):
+#  --device /dev/fuse                    fuse-overlayfs storage for nested podman
+#  --device /dev/net/tun                 slirp4netns/pasta tap device for
+#                                        rootless container networking
+#  seccomp/apparmor unconfined           required for nested user namespaces on
+#                                        stock Docker; a tailored seccomp profile
+#                                        is a follow-up hardening task
+#  --cap-add SYS_ADMIN                   required for newuidmap to write the
+#                                        rootless uid/gid maps (verified: without
+#                                        it nested podman fails with EPERM)
+#  systempaths=unconfined                unmask /proc so crun can set per-container
+#                                        sysctls (ping_group_range) for inner nets
+#  $PODMAN_VOLUME                        keeps inner images/containers across
+#                                        outer restarts
+docker run -d --name "$NAME" \
+  --restart unless-stopped \
+  --device /dev/fuse \
+  --device /dev/net/tun \
+  --security-opt seccomp=unconfined \
+  --security-opt apparmor=unconfined \
+  --security-opt systempaths=unconfined \
+  --cap-add SYS_ADMIN \
+  -v "$WORKSPACE_VOLUME":/workspace \
+  -v "$PODMAN_VOLUME":/home/builder/.local/share/containers \
+  -p "$AGENT_PORT":4001 \
+  -p "$APP_PORTS":"$APP_PORTS" \
+  -e ANTHROPIC_API_KEY \
+  -e AGENT_SERVER_TOKEN \
+  -e LITELLM_BASE_URL -e LITELLM_API_KEY \
+  -e LITELLM_MODELS -e LITELLM_MODELS_JSON \
+  -e LITELLM_DEFAULT_MODEL -e LITELLM_DEFAULT_THINKING \
+  "$IMAGE"
+
+echo "builder container '$NAME' is up — agent-server on :$AGENT_PORT, app ports $APP_PORTS"

docker/builder/storage.conf

@@ -0,0 +1,9 @@
+# Rootless storage for the builder user. overlay + fuse-overlayfs works in
+# nested user namespaces (requires --device /dev/fuse on the outer container).
+[storage]
+driver = "overlay"
+runroot = "/tmp/podman-run"
+graphroot = "/home/builder/.local/share/containers/storage"
+
+[storage.options.overlay]
+mount_program = "/usr/bin/fuse-overlayfs"

docker/builder/verify.sh

@@ -0,0 +1,64 @@
+#!/usr/bin/env bash
+# Acceptance checks for the outer builder container (run on the docker host).
+# Expects the container started via run.sh. See the design spec for context.
+set -uo pipefail
+
+NAME="${BUILDER_NAME:-appx-builder}"
+PORT="${BUILDER_AGENT_PORT:-4001}"
+BASE="http://127.0.0.1:$PORT"
+fail=0
+
+check() {
+  local label="$1"; shift
+  if "$@" >/dev/null 2>&1; then
+    echo "ok - $label"
+  else
+    echo "not ok - $label"
+    fail=1
+  fi
+}
+
+body() { curl -fsS --max-time 10 "$@"; }
+
+# 1. REST surface up
+check "healthz" body "$BASE/v1/healthz"
+
+# 2. Idempotent project creation
+P1=$(body -X POST "$BASE/v1/projects" -H 'content-type: application/json' -d '{"name":"demo"}')
+P2=$(body -X POST "$BASE/v1/projects" -H 'content-type: application/json' -d '{"name":"demo"}')
+if [ -n "$P1" ] && [ "$P1" = "$P2" ]; then echo "ok - project create idempotent"; else echo "not ok - project create idempotent"; fail=1; fi
+
+# 3. Session creation (runtime boots, builder prompt loads — see container logs)
+check "session create" body -X POST "$BASE/v1/projects/demo/sessions"
+
+# 4. Nested rootless podman works
+if docker exec -u builder "$NAME" podman run --rm docker.io/library/alpine echo nested-ok 2>/dev/null | grep -q nested-ok; then
+  echo "ok - nested podman run"
+else
+  echo "not ok - nested podman run"; fail=1
+fi
+
+# 5. Inner app port chain: inner :3000 → outer -p → host
+docker exec -u builder "$NAME" podman rm -f verify-web >/dev/null 2>&1
+if docker exec -u builder "$NAME" podman run -d --name verify-web -p 3000:80 docker.io/library/nginx:alpine >/dev/null 2>&1; then
+  sleep 3
+  check "inner app reachable from host (:3000)" body "http://127.0.0.1:3000"
+  docker exec -u builder "$NAME" podman rm -f verify-web >/dev/null 2>&1
+else
+  echo "not ok - inner app container start"; fail=1
+fi
+
+# 6. Registry survives an outer restart
+docker restart "$NAME" >/dev/null
+sleep 5
+if body "$BASE/v1/projects" | grep -q '"demo"'; then
+  echo "ok - project registry survives restart"
+else
+  echo "not ok - project registry survives restart"; fail=1
+fi
+
+# 7. No credential leak into inner containers
+LEAK=$(docker exec -u builder "$NAME" podman run --rm docker.io/library/alpine env 2>/dev/null | grep -cE "ANTHROPIC_API_KEY|LITELLM_API_KEY")
+if [ "${LEAK:-0}" = "0" ]; then echo "ok - no credentials in inner env"; else echo "not ok - credentials leaked into inner env"; fail=1; fi
+
+exit $fail

docs/architecture/important/builder-container-architecture.md

@@ -176,12 +176,14 @@ No host-level work happens for any of this beyond running the outer container. *
 
 ## What Needs to Be Built
 
-1. **The outer container's Dockerfile** — Ubuntu/Alpine + podman + nodejs + agent-server (~10 lines, draft in `rootless-podman-isolation.md`)
-2. **A run script / docker-compose** that launches the outer container with the right flags (`--device /dev/fuse`, port forwards, volume mount, env vars)
+1. ✅ **The outer container's Dockerfile** — built: `docker/builder/Dockerfile` (Ubuntu 24.04 + NodeSource 22 + rootless podman, fuse-overlayfs, builder user with subuid/subgid)
+2. ✅ **A run script** — built: `docker/builder/run.sh` is the canonical spawn contract (`--device /dev/fuse`, relaxed seccomp/apparmor for nested userns, workspace + podman-storage volumes, port forwards, env passthrough)
 3. **Project provisioning logic** — `POST /v1/projects { name }` already creates `WORKSPACE_DIR/<id>/` and registers the project; provisioning is just calling that endpoint (plus any product-specific scaffolding the caller layers on top)
-4. **System prompt for the builder agent** — telling it that `podman` is available, where projects live, how to expose ports
+4. ✅ **System prompt for the builder agent** — built: `docker/builder/AGENTS.builder.md`, installed at `WORKSPACE_DIR/.pi-global/AGENTS.md` on container startup so runtimes using the shared agent dir can load it; prompt-engineering iteration remains open
 5. **(Optional) An idle-eviction sweep** — if many projects exist and stopping unused `ProjectRuntime`s would free memory; not needed for one admin user
 
+See `docs/superpowers/specs/2026-06-10-outer-builder-container-design.md` for the design and acceptance criteria.
+
 That's it. Maybe 1-2 days of work for the outer container + provisioning, plus prompt engineering iteration on point 4.
 
 ## What This Architecture Buys You