diff --git a/.claude/skills/deploy-spur-cluster/SKILL.md b/.claude/skills/deploy-spur-cluster/SKILL.md new file mode 100644 index 00000000..83ff5134 --- /dev/null +++ b/.claude/skills/deploy-spur-cluster/SKILL.md @@ -0,0 +1,620 @@ +--- +name: deploy-spur-cluster +description: Use when the user asks to deploy/install a Spur cluster on one or more bare-metal hosts over SSH. Covers every topology the Ansible playbook does — single-node, multi-node, HA (multi-controller Raft), and HA with separate compute nodes — plus optional PostgreSQL/spurdbd accounting and WireGuard mesh. Installs daemons as systemd services and Slurm-compatible CLI symlinks. Drives everything with plain SSH + bash; no Ansible required. ALWAYS asks the user up-front for mode + topology + controller count + accounting before touching anything. +--- + +# Deploy Spur Cluster + +Spur is an AI-native job scheduler with these daemons: + +- **spurctld** — controller / scheduler / Raft consensus (1 instance, or ≥ 3 for HA) +- **spurd** — node agent, runs on every compute host +- **spurdbd** — optional accounting daemon (sacct/fairshare), backed by PostgreSQL. One instance for the whole cluster, on `ACCT_HOST` (default: first controller; may be a dedicated node). + +This skill stands a cluster up with only SSH + bash on the targets. It is the standalone equivalent of `deploy/ansible/` and reaches the same end state: daemons run as **systemd services** (survive reboot), Slurm-compatible CLI names are symlinked, and accounting is optional (default on). One flow covers all four topologies — only the host list, Raft topology, and which hosts run an agent differ. + +Defaults (override only if the user asks): + +| Var | Default | +|---|---| +| `SPUR_HOME` | `/root/spur` | +| `SPUR_INSTALL_DIR` | `/root/.local/bin` | +| `SPUR_VERSION` | `latest` (passed to `install.sh`; or `nightly` / `vX.Y.Z`) | +| `SPUR_BINARY_SRC` | *(empty)* — local dir with pre-built `spur/spurctld/spurd/spurdbd`; used when set, else `install.sh` | +| `SPUR_CONTROLLER_PORT` | `6817` | +| `SPUR_AGENT_PORT` | `6818` | +| `SPUR_RAFT_PORT` | `6821` (hardcoded inside spurctld; cannot be changed via CLI) | +| `SPUR_ACCT_PORT` | `6819` | +| `SPUR_CLUSTER_NAME` | `spur-cluster` | +| `SPUR_LOG_LEVEL` | `info` | +| `SPUR_WIPE_STATE` | `false` (preserve Raft state so re-runs/upgrades are non-destructive; set `true` for a fresh install or intentional Raft reinit) | +| `ACCOUNTING` | `true` (deploy PostgreSQL + spurdbd; set `false` to skip) | +| `ACCT_DB_NAME` / `ACCT_DB_USER` / `ACCT_DB_PASSWORD` | `spur` / `spur` / `spur` | +| `TRANSPORT` | `direct` (LAN) — or `wireguard` for an encrypted mesh | +| SSH user | `root` (unless the user specifies otherwise) | + +> These commands assume passwordless `sudo` (or SSH as root). If the SSH user is non-root, prefix privileged commands with `sudo` and confirm they have it. systemd unit installs, apt, and `/etc/systemd/system` writes all require root. +> +> **Non-root SSH + install dir under `/root`:** when `SPUR_INSTALL_DIR` is `/root/.local/bin` (the default) but you SSH as a non-root user, `/root` is mode 700 — the SSH user cannot even *execute* the binaries. In that case **every `spur`/`sbatch`/… CLI invocation must also be `sudo`-prefixed**, not just the file-writing steps. Alternatively set `SPUR_INSTALL_DIR` to a world-readable path (e.g. `/usr/local/bin`). Writing unit files and `spur.conf` needs `sudo tee` or scp-to-`/tmp`-then-`sudo install`, since heredoc redirects run as the SSH user. + +## Step 0: gather inputs (MANDATORY — do not skip) + +Before any SSH, **ask the user** (use `AskUserQuestion` for anything they didn't state; don't guess): + +1. **Deployment mode** — pick exactly one: + | Mode | Use when | + |---|---| + | `single-node` | one host runs controller **and** agent | + | `multi-node` | 1 controller, N compute agents (controller may also run an agent — hyperconverged) | + | `ha` | ≥ 3 controllers (Raft), N agents; controllers may be hyperconverged **or** dedicated (separate compute) | + +2. **Hosts** — for each role: + - `CONTROLLERS` — SSH targets running `spurctld`. Counts: single-node 1, multi-node 1, ha odd ≥ 3. + - `AGENTS` — SSH targets running `spurd`. Any number ≥ 1. A host may appear in **both** lists (hyperconverged) or **only** in `AGENTS` (dedicated compute / "separate compute" HA). + +3. **Accounting** — deploy PostgreSQL + spurdbd for `sacct`/fairshare? Default **yes**. If no, set `ACCOUNTING=false`; job submission still works, only `sacct` is unavailable. + +4. **Transport** — `direct` (LAN, default) or `wireguard` (encrypted mesh). WireGuard adds Step 2b; everything else is identical (config advertises WG IPs instead of LAN IPs). + +> **For HA, warn the user** if controller count is even or < 3: +> - `N=1` → not HA; suggest `multi-node`. +> - `N=2` → "zero fault tolerance" (quorum 2, tolerates 0 failures) — code-path testing only. +> - even `N ≥ 4` → suggest `N−1` (strictly better). +> +> **Topologies map to inventory shape** exactly like the playbook: +> - single-node → same host in CONTROLLERS and AGENTS +> - multi-node → 1 controller, N agents +> - HA hyperconverged → controllers also in AGENTS +> - HA + separate compute → controllers **not** in AGENTS; distinct agent hosts + +Once gathered, define the arrays the rest of the skill uses: +```bash +CONTROLLERS=( user@host1 user@host2 user@host3 ) # ordered — index = Raft node_id - 1; ORDER MUST BE STABLE +AGENTS=( user@host4 ) # may overlap CONTROLLERS (hyperconverged) or be disjoint +SSH_USER=root +TRANSPORT=direct # or wireguard +ACCOUNTING=true # or false +ACCT_HOST="${CONTROLLERS[0]}" # accounting host: default first controller; may be ANY host — a controller, an agent, or a dedicated node (add it to HOSTS_ALL if dedicated) + +SPUR_HOME=/root/spur +SPUR_INSTALL_DIR=/root/.local/bin +SPUR_VERSION=latest +SPUR_BINARY_SRC= # e.g. /tmp/spur-bin to push pre-built binaries +SPUR_CONTROLLER_PORT=6817; SPUR_AGENT_PORT=6818; SPUR_RAFT_PORT=6821; SPUR_ACCT_PORT=6819 +SPUR_CLUSTER_NAME=spur-cluster; SPUR_LOG_LEVEL=info; SPUR_WIPE_STATE=false +ACCT_DB_NAME=spur; ACCT_DB_USER=spur; ACCT_DB_PASSWORD=spur + +HOSTS_ALL=( $(printf '%s\n' "${CONTROLLERS[@]}" "${AGENTS[@]}" | sort -u) ) +ha_enabled=false; [ ${#CONTROLLERS[@]} -gt 1 ] && ha_enabled=true +``` + +## Step 1: preflight all hosts + +Run on every unique host. Abort the whole deploy on any failure. + +```bash +for tgt in "${HOSTS_ALL[@]}"; do + echo "############ $tgt ############" + ssh -o BatchMode=yes -o ConnectTimeout=10 "$tgt" ' + set +e + echo "host=$(hostname -s) fqdn=$(hostname -f)" + echo "kernel=$(uname -r) nproc=$(nproc)" + echo "--- spur ports (6817/6818/6819/6821) ---" + ss -tlnpH 2>/dev/null | grep -E ":(6817|6818|6819|6821)\b" || echo "spur ports free" + echo "--- existing spur pids ---" + pgrep -ax spurctld; pgrep -ax spurd; pgrep -ax spurdbd; echo "(end pids)" + echo "--- tools ---" + for t in curl tar bash ss pgrep pkill systemctl; do command -v $t >/dev/null || echo "MISSING:$t"; done + echo "--- sudo ---"; sudo -n true 2>/dev/null && echo "sudo:ok" || echo "sudo:NEEDS-PASSWORD" + echo "--- ip ---" + ip -4 -o addr show | awk "{print \$2, \$4}" | grep -v "127.0.0.1" + echo "--- os ---" + . /etc/os-release 2>/dev/null && echo "$PRETTY_NAME" + ' +done +``` + +Fail-fast rules: +- A spur port held by a process that is NOT `spurctld`/`spurd`/`spurdbd` → abort. +- `MISSING:systemctl` → abort (this skill installs systemd units; systemd is required). +- `MISSING:curl`/`tar` → abort unless `SPUR_BINARY_SRC` is set (installer needs them; the binary-copy path does not). +- SSH fails → abort that host; fix auth first. + +(Existing spur daemons are fine — Step 4 stops them.) + +## Step 2: install Spur binaries on all hosts (idempotent) + +Two sources, same as the playbook. `SPUR_BINARY_SRC` (a local dir holding pre-built `spur`, `spurctld`, `spurd`, `spurdbd`) takes precedence — use it when the upstream repo has no published release (`install.sh` returns 403) or for air-gapped installs. Otherwise curl `install.sh`. + +```bash +for tgt in "${HOSTS_ALL[@]}"; do + ssh "$tgt" " + set -euo pipefail + mkdir -p ${SPUR_HOME} ${SPUR_HOME}/state ${SPUR_HOME}/log ${SPUR_HOME}/etc ${SPUR_INSTALL_DIR} + " + if [ -n "$SPUR_BINARY_SRC" ]; then + # Push pre-built binaries from the operator box. + for b in spur spurctld spurd spurdbd; do + scp -q "${SPUR_BINARY_SRC}/${b}" "${tgt}:${SPUR_INSTALL_DIR}/${b}" + done + ssh "$tgt" "chmod 0755 ${SPUR_INSTALL_DIR}/spur ${SPUR_INSTALL_DIR}/spurctld ${SPUR_INSTALL_DIR}/spurd ${SPUR_INSTALL_DIR}/spurdbd" + else + ssh "$tgt" " + set -euo pipefail + if [ ! -x ${SPUR_INSTALL_DIR}/spur ]; then + curl -fsSL https://raw.githubusercontent.com/ROCm/spur/main/install.sh \ + | INSTALL_DIR=${SPUR_INSTALL_DIR} bash -s -- ${SPUR_VERSION} + fi + " + fi + # Verify + create Slurm-compatible symlinks (the single `spur` binary dispatches on argv[0]). + ssh "$tgt" " + set -euo pipefail + test -x ${SPUR_INSTALL_DIR}/spur || { echo 'spur binary missing after install' >&2; exit 1; } + for n in sbatch squeue sinfo scancel sacct scontrol salloc srun; do + ln -sf ${SPUR_INSTALL_DIR}/spur ${SPUR_INSTALL_DIR}/\$n + done + echo 'spur installed + symlinks created' + " +done +``` + +> Do NOT rely on `spur --version` — it is not a supported flag and errors. Check for the file with `test -x` instead. + +Optional: prepend `${SPUR_INSTALL_DIR}` to `/etc/environment` so non-interactive SSH gets `spur`/`sbatch`/etc. on PATH. + +### Step 2b: WireGuard mesh (only when `TRANSPORT=wireguard`) + +Skip entirely for `direct`. WireGuard uses the built-in `spur net` CLI and is **single-controller only** — `spur net init` auto-assigns the controller `.1` and there is no multi-controller mesh command, so HA must use `direct`. Steps, using the real CLI (all `spur net` commands log to stderr): + +1. `apt install wireguard-tools` on every host. +2. On the controller: `spur net init --cidr 10.44.0.0/16 --port 51820 --interface spur0` (auto-assigns `.1`). Read its pubkey with `wg show spur0 public-key` (there is **no** `spur net pubkey`). +3. On each agent (assign `.2`, `.3`, …): `spur net join --endpoint :51820 --server-key --address 10.44.0. --prefix-len 16 --interface spur0`. `--prefix-len` **must match the CIDR** (defaults to 16). Read the agent pubkey with `wg show spur0 public-key`. +4. On the controller, register each agent: `spur net add-peer --key --allowed-ip 10.44.0./32 --interface spur0`. + +Then set `WG_IP[$host]` per host and use those in place of `IP[...]` for `[controller].hosts`, `peers`, and spurd `--address`/`--controller`. There is no `spur net down` — tear down with `wg-quick down spur0` (or `ip link del spur0`) and remove `/etc/wireguard/spur0.conf`. If the user wants WG but you cannot verify mesh connectivity (all hosts on one `/24` makes it moot), tell them and offer `direct` instead. + +## Step 3: derive per-host facts (hostnames, IPs, node_ids) + +```bash +host_short() { ssh "$1" 'hostname -s'; } +host_addr() { local t="${1#*@}"; echo "$t"; } # SSH target IP/host, minus user@ + +declare -A SHORT IP NODE_ID +for h in "${HOSTS_ALL[@]}"; do + SHORT[$h]=$(host_short "$h") + IP[$h]=$(host_addr "$h") # for TRANSPORT=wireguard, set IP[$h]=${WG_IP[$h]} instead +done + +# 1-based Raft node_id = position in CONTROLLERS. ORDER MATTERS — reordering after a +# deploy breaks openraft membership. To re-order, wipe state on every controller and redeploy. +for i in "${!CONTROLLERS[@]}"; do NODE_ID[${CONTROLLERS[$i]}]=$((i+1)); done +``` + +`hostname -s` (not `-f`) is intentional — the controller's `[[nodes]]` names, `spurd --hostname`, and `spur show node ` must all use the same short form. + +## Step 4: stop existing daemons + wipe state + +Stop via systemd if a unit exists, and belt-and-suspenders `pkill -x` (exact name — `pkill -f spurd` also kills `spurctld`). + +```bash +for tgt in "${HOSTS_ALL[@]}"; do + ssh "$tgt" ' + for svc in spurd spurctld spurdbd; do + systemctl stop "$svc" 2>/dev/null || true + done + pkill -x spurd 2>/dev/null || true + pkill -x spurctld 2>/dev/null || true + pkill -x spurdbd 2>/dev/null || true + for i in $(seq 1 10); do + pgrep -x spurctld >/dev/null || pgrep -x spurd >/dev/null || pgrep -x spurdbd >/dev/null || exit 0 + sleep 0.5 + done + echo "daemons still running after 5s" >&2; exit 1 + ' +done + +# Wipe Raft state on controllers BEFORE start (so spurctld does not rewrite the log we delete). +if [ "$SPUR_WIPE_STATE" = true ]; then + for tgt in "${CONTROLLERS[@]}"; do + ssh "$tgt" "rm -rf ${SPUR_HOME}/state && mkdir -p ${SPUR_HOME}/state" + done +fi +``` + +Default is **no wipe** so re-runs and upgrades preserve the job queue and node registrations. Wipe only for a fresh install or an intentional Raft reinit. Because Spur 0.3.0 has no online Raft membership change, **changing the controller set (add/remove/reorder) requires a wipe** — if you're keeping state but the controller list differs from the running cluster, warn the user and require `SPUR_WIPE_STATE=true`. Compute agents are not Raft members and can be added/removed freely without a wipe. When demoting a host from controller to agent-only, `systemctl disable --now spurctld` on it first, or the stale daemon keeps the old membership and can block quorum. + +## Step 5: deploy accounting (only when `ACCOUNTING=true`) — on `ACCT_HOST` + +Accounting is a single service for the whole cluster; it lives on `ACCT_HOST` (default `CONTROLLERS[0]`, but may be any host — a controller, an agent, or a dedicated node). Deploy it **before** the controllers so the `[accounting]` block spurctld reads has a live spurdbd. Idempotent: existence-checked role/DB creation. If `ACCT_HOST` is a dedicated node, make sure Step 2 installed the `spurdbd` binary there too (add it to `HOSTS_ALL`). + +```bash +if [ "$ACCOUNTING" = true ]; then + ssh "$ACCT_HOST" " + set -euo pipefail + export DEBIAN_FRONTEND=noninteractive + # Install PostgreSQL (Debian/Ubuntu). For RHEL, swap in dnf + postgresql-server + initdb. + if ! command -v psql >/dev/null 2>&1; then + apt-get update -qq + apt-get install -y -qq postgresql postgresql-contrib + fi + systemctl enable --now postgresql + # Create role + DB idempotently via the postgres superuser. + sudo -u postgres psql -tAc \"SELECT 1 FROM pg_roles WHERE rolname='${ACCT_DB_USER}'\" | grep -q 1 \ + || sudo -u postgres psql -c \"CREATE ROLE ${ACCT_DB_USER} LOGIN PASSWORD '${ACCT_DB_PASSWORD}'\" + sudo -u postgres psql -tAc \"SELECT 1 FROM pg_database WHERE datname='${ACCT_DB_NAME}'\" | grep -q 1 \ + || sudo -u postgres psql -c \"CREATE DATABASE ${ACCT_DB_NAME} OWNER ${ACCT_DB_USER}\" + test -x ${SPUR_INSTALL_DIR}/spurdbd || { echo 'spurdbd binary missing — install it or set ACCOUNTING=false' >&2; exit 1; } + " + + # Install + start the spurdbd systemd unit. --migrate creates the accounting tables. + ssh "$ACCT_HOST" "cat > /etc/systemd/system/spurdbd.service <&2; exit 1; }; done + " +fi +``` + +## Step 6: render spur.conf + install spurctld systemd unit on every controller + +All controllers get the **same** `spur.conf` except `node_id` (HA only). The `peers` list order must be identical on every host (by `node_id`). + +```bash +# CSVs for the controller list and the raft peers list. +ctl_hosts_csv=""; peers_csv="" +for h in "${CONTROLLERS[@]}"; do + ctl_hosts_csv+="\"${IP[$h]}\", " + peers_csv+="\"${IP[$h]}:${SPUR_RAFT_PORT}\", " +done +ctl_hosts_csv=${ctl_hosts_csv%, }; peers_csv=${peers_csv%, } + +# Per-agent [[nodes]] blocks (90% of RAM) + partition node list. +nodes_blocks=""; part_csv="" +for h in "${AGENTS[@]}"; do + cpus=$(ssh "$h" 'nproc') + mem_kb=$(ssh "$h" "awk '/MemTotal/{print \$2}' /proc/meminfo") + mem_mb=$(( mem_kb / 1024 * 9 / 10 )) + nodes_blocks+=$'\n'"[[nodes]]"$'\n'"names = \"${SHORT[$h]}\""$'\n'"cpus = ${cpus}"$'\n'"memory_mb = ${mem_mb}"$'\n' + part_csv+="${SHORT[$h]}," +done +part_csv=${part_csv%,} + +# Accounting block (only when enabled) points at ACCT_HOST's advertised IP. +# ${IP[$ACCT_HOST]} requires ACCT_HOST to be in HOSTS_ALL (Step 3); falls back to +# stripping user@ from ACCT_HOST if it wasn't resolved into the IP map. +acct_block="" +if [ "$ACCOUNTING" = true ]; then + acct_ip="${IP[$ACCT_HOST]:-${ACCT_HOST#*@}}" + acct_block=$'\n'"[accounting]"$'\n'"host = \"${acct_ip}:${SPUR_ACCT_PORT}\""$'\n'"database_url = \"postgresql://${ACCT_DB_USER}:${ACCT_DB_PASSWORD}@localhost/${ACCT_DB_NAME}\""$'\n'"fairshare_refresh_secs = 30"$'\n' +fi + +wg_line="wg_enabled = false" +[ "$TRANSPORT" = wireguard ] && wg_line="wg_enabled = true"$'\n'"wg_interface = \"spur0\"" + +for ctl in "${CONTROLLERS[@]}"; do + raft_block="" + if $ha_enabled; then + raft_block="node_id = ${NODE_ID[$ctl]}"$'\n'"peers = [${peers_csv}]" + fi + tmp=$(mktemp) + cat > "$tmp" < /etc/systemd/system/spurctld.service <&2; exit 1" +done +``` + +**HA only:** spurctld binds 6817 immediately but returns `no leader elected yet` until quorum forms. Wait on the first controller: + +```bash +if $ha_enabled; then + ssh "${CONTROLLERS[0]}" " + for i in \$(seq 1 60); do + # NOTE: prefix with sudo if the install dir is under /root and you SSH non-root. + out=\$(${SPUR_INSTALL_DIR}/spur nodes 2>&1); rc=\$? + # Ready only when the command SUCCEEDS and the output has a real node table. + # Gate on rc==0 too — a Permission denied / crash must NOT be read as 'leader up'. + if [ \$rc -eq 0 ] && ! echo \"\$out\" | grep -qE 'no leader|not the Raft leader|cannot reach leader|transport error|Connection refused|Permission denied'; then + echo OK; exit 0 + fi + sleep 1 + done + echo \"timeout waiting for leader: \$out\" >&2; exit 1 + " +fi +``` + +Set the client env on each **controller** so `squeue`/`sinfo`/`scontrol` (use `SPUR_CONTROLLER_ADDR`) and `sacct`/`sacctmgr`/`sreport`/`sshare` (use `SPUR_ACCOUNTING_ADDR`) work with no per-command flags — otherwise they default to `localhost` and fail on any controller that isn't the accounting host. Controllers only (agents don't run the CLI for users here): + +```bash +ACCT_IP="${IP[$ACCT_HOST]:-${ACCT_HOST#*@}}" +for ctl in "${CONTROLLERS[@]}"; do + ctl_ip="${IP[$ctl]}" + ssh "$ctl" " + sudo sed -i '/^SPUR_CONTROLLER_ADDR=/d;/^SPUR_ACCOUNTING_ADDR=/d' /etc/environment + echo 'SPUR_CONTROLLER_ADDR=http://${ctl_ip}:${SPUR_CONTROLLER_PORT}' | sudo tee -a /etc/environment >/dev/null + $( [ \"$ACCOUNTING\" = true ] && echo "echo 'SPUR_ACCOUNTING_ADDR=http://${ACCT_IP}:${SPUR_ACCT_PORT}' | sudo tee -a /etc/environment >/dev/null" ) + " +done +``` + +(`/etc/environment` is read at login, so a fresh shell / `bash -lc` picks it up; for `sudo` also running the CLI, use `sudo -E` to pass the vars through.) + +## Step 7: install spurd systemd unit on every agent + +`spurd --controller` takes a single URL — point every agent at `CONTROLLERS[0]` (followers forward writes via Raft). `WorkingDirectory=${SPUR_HOME}` in the unit sets the *fallback* stdout dir for `spur-.out` (a job's own `WorkDir`/submit-CWD takes precedence). `--hostname`/`--address` are explicit (auto-detect picks `127.0.0.1`, breaking inter-node dispatch). + +```bash +CTL0="${IP[${CONTROLLERS[0]}]}" +for ag in "${AGENTS[@]}"; do + ssh "$ag" "cat > /etc/systemd/system/spurd.service <&2; exit 1 + " +done +``` + +## Step 8: wait for every agent to register + +`spur nodes` collapses by partition, so check per-agent with `spur show node`: + +```bash +for ag in "${AGENTS[@]}"; do + for i in $(seq 1 30); do + if ssh "${CONTROLLERS[0]}" "${SPUR_INSTALL_DIR}/spur show node ${SHORT[$ag]} >/dev/null 2>&1"; then + echo "${SHORT[$ag]} registered"; break + fi + [ $i -eq 30 ] && { echo "${SHORT[$ag]} never registered" >&2; exit 1; } + sleep 1 + done +done +``` + +## Step 9: smoke test + +### Single-node job (every deploy) + +```bash +ssh "${CONTROLLERS[0]}" "cat > /tmp/spur-test-single.sh <<'EOF' +#!/bin/bash +#SBATCH --job-name=spur-test-single +echo \"ran on \$(hostname) at \$(date)\" +EOF +chmod +x /tmp/spur-test-single.sh +cd /tmp # predictable, world-accessible WorkDir; job stdout -> /tmp/spur-.out. Do NOT cd into a 0700 dir like /root/spur when SSHing non-root. +jid=\$(${SPUR_INSTALL_DIR}/spur submit /tmp/spur-test-single.sh | grep -oE '[0-9]+') +echo \"JOBID=\$jid\" +for i in \$(seq 1 30); do + st=\$(${SPUR_INSTALL_DIR}/spur show job \$jid 2>/dev/null | grep -oE 'JobState=[A-Z]+' | head -1 | cut -d= -f2) + case \"\$st\" in COMPLETED) echo OK; break ;; FAILED|CANCELLED|TIMEOUT|NODE_FAIL) echo \"BAD: \$st\" >&2; exit 1 ;; esac + sleep 1 +done +echo \"final state: \$st\" +" +``` + +Output lands in `spur-.out` on whichever agent ran the job, in the job's `WorkDir` — i.e. the CWD at submit time (`/tmp` above). It is NOT necessarily `${SPUR_HOME}`. To locate it robustly, loop agents and search the likely dirs: `sudo find /tmp ${SPUR_HOME} /home /root -maxdepth 2 -name 'spur-.out'` (no shared-FS assumption; use `sudo` for non-root SSH). + +### Multi-node job (when |AGENTS| ≥ 2) + +`-N` must not exceed the agent count. Use `<<'EOF'` so `$SPUR_*` vars reach the agent verbatim and expand at run time; inject `N` via sed. + +```bash +N=${#AGENTS[@]} +ssh "${CONTROLLERS[0]}" "cat > /tmp/spur-test-multi.sh <<'EOF' +#!/bin/bash +#SBATCH --job-name=spur-test-multi +#SBATCH -N __N__ +#SBATCH --ntasks-per-node=1 +echo \"node \$SPUR_TASK_OFFSET of \$SPUR_NUM_NODES on \$(hostname); peers=\$SPUR_PEER_NODES\" +EOF +sed -i 's/__N__/${N}/' /tmp/spur-test-multi.sh +chmod +x /tmp/spur-test-multi.sh +cd /tmp # predictable WorkDir (see single-node note); output -> /tmp/spur-.out per node +jid=\$(${SPUR_INSTALL_DIR}/spur submit /tmp/spur-test-multi.sh | grep -oE '[0-9]+') +echo \"JOBID=\$jid\" +for i in \$(seq 1 60); do + st=\$(${SPUR_INSTALL_DIR}/spur show job \$jid 2>/dev/null | grep -oE 'JobState=[A-Z]+' | head -1 | cut -d= -f2) + case \"\$st\" in COMPLETED) echo OK; break ;; FAILED|CANCELLED|TIMEOUT|NODE_FAIL) echo \"BAD: \$st\" >&2; exit 1 ;; esac + sleep 1 +done +" +# Multi-node writes locally on each node — fetch from every agent. +for ag in "${AGENTS[@]}"; do + echo "=== ${SHORT[$ag]} ===" + ssh "$ag" "ls ${SPUR_HOME}/spur-*.out 2>/dev/null | xargs -r tail -n +1" +done +``` + +### Accounting check (when `ACCOUNTING=true`) + +```bash +ssh "${CONTROLLERS[0]}" "${SPUR_INSTALL_DIR}/sacct | head" # sacct works from any controller (gRPC to spurdbd) +ssh "$ACCT_HOST" "sudo -u postgres psql -d ${ACCT_DB_NAME} -tAc 'SELECT count(*) FROM jobs;'" # postgres lives on ACCT_HOST +``` +Expect a row per completed job. (With `ACCOUNTING=false`, `sacct` is expected to fail — that's fine; jobs still run.) + +## Step 10: verify + +```bash +ssh "${CONTROLLERS[0]}" "${SPUR_INSTALL_DIR}/spur nodes" +ssh "${CONTROLLERS[0]}" "${SPUR_INSTALL_DIR}/spur queue" + +# Every daemon should be systemd-active. +for ctl in "${CONTROLLERS[@]}"; do ssh "$ctl" "systemctl is-active spurctld"; done +for ag in "${AGENTS[@]}"; do ssh "$ag" "systemctl is-active spurd"; done +[ "$ACCOUNTING" = true ] && ssh "$ACCT_HOST" "systemctl is-active spurdbd postgresql" + +# HA: identify the CURRENT leader. Every node that was ever leader has a +# 'become leader' line, so grep -m1 (first match) is wrong after any +# re-election. The authoritative current leader is in each node's persisted +# vote — read node_id from vote.json (identical on all healthy peers). +if $ha_enabled; then + ssh "${CONTROLLERS[0]}" "sudo cat ${SPUR_HOME}/state/raft/vote.json 2>/dev/null" \ + | grep -oE '\"node_id\":[0-9]+' | tail -1 | sed 's/.*://' \ + | xargs -I{} echo "current Raft leader: node_id={}" +fi + +# Separate-compute HA sanity: controllers that are NOT agents must have spurd inactive. +for ctl in "${CONTROLLERS[@]}"; do + is_agent=false; for a in "${AGENTS[@]}"; do [ "$a" = "$ctl" ] && is_agent=true; done + $is_agent || ssh "$ctl" "systemctl is-active spurd 2>/dev/null | grep -q inactive && echo '$ctl: no agent (correct)' || echo '$ctl: unexpected spurd'" +done +``` + +## Step 11: teardown (only when asked) + +```bash +for tgt in "${HOSTS_ALL[@]}"; do + ssh "$tgt" " + for svc in spurd spurctld spurdbd; do systemctl disable --now \$svc 2>/dev/null || true; done + pkill -x spurd 2>/dev/null; pkill -x spurctld 2>/dev/null; pkill -x spurdbd 2>/dev/null + rm -f /etc/systemd/system/spurd.service /etc/systemd/system/spurctld.service /etc/systemd/system/spurdbd.service + systemctl daemon-reload 2>/dev/null || true + rm -rf ${SPUR_HOME} + rm -f /root/spur-*.out ${SPUR_INSTALL_DIR}/spur-*.out /tmp/spur-*.out + " +done +``` + +To also remove accounting data (destructive): `ssh "$ACCT_HOST" "sudo -u postgres dropdb ${ACCT_DB_NAME}; sudo -u postgres dropuser ${ACCT_DB_USER}"`. Only do this if the user explicitly asks — it deletes all job history. Leave PostgreSQL itself installed unless asked to purge it. + +## Gotchas (all hard-won — don't relearn them) + +### Install / systemd +- **This skill uses systemd**, not `nohup`. Units are `/etc/systemd/system/spur{ctld,d,dbd}.service`, `enabled` (survive reboot), `Restart=on-failure`. Always `systemctl daemon-reload` after writing a unit. +- **`spur --version` is NOT supported** — it errors. Check the binary with `test -x`, not by running `--version`. +- **`install.sh` returns 403 when the repo has no published release.** Set `SPUR_BINARY_SRC` to a local dir of pre-built binaries and the skill scp's them instead. +- **Slurm-compat symlinks** (`sbatch`/`squeue`/`sinfo`/`scancel`/`sacct`/`scontrol`/`salloc`/`srun` → `spur`) are created on every install path. The `spur` multi-call binary dispatches on argv[0]. +- **`pkill -f spurd` also kills `spurctld`** (substring match). Always `pkill -x` (exact name). + +### Accounting +- **Deploy accounting BEFORE the controller.** spurctld reads the `[accounting]` block at start and connects to spurdbd; if spurdbd isn't up yet, wire it first (Step 5 precedes Step 6). +- **spurdbd needs `--migrate`** to create the accounting tables on first run. +- **Accounting is one instance for the whole cluster**, on `CONTROLLERS[0]`. Don't run it per-node. +- **Default DB creds are `spur/spur/spur`** — fine for a lab, change `ACCT_DB_PASSWORD` for anything real. Flag this to the user. +- **`SPUR_WIPE_STATE` defaults to `false`** — re-runs and upgrades preserve the Raft job queue and node registrations. `SPUR_WIPE_STATE=true` resets the Raft job-id counter (job ids restart at 1, upserting onto the same accounting rows); use it only for a fresh install or intentional reinit. +- **Rebuild all four binaries together for an upgrade.** The daemons share a Raft WAL schema and a spurdbd DB schema. Pushing a `spurctld` built from a different tree than its peers can crash it on start (`unknown variant …` / `LogIndex(N) violates`) when it reads a log entry it can't parse. +- **Changing the controller set needs a wipe** (Spur 0.3.0 has no online membership change). Adding/removing/reordering a controller with state preserved leaves openraft with a mismatched on-disk membership. Agents are not Raft members — add/remove them freely. +- **A stale dpkg lock** (`Could not get lock /var/lib/dpkg/lock-frontend`) means another apt/unattended-upgrade is running. Wait for it, or clear a genuinely hung `apt-get` before retrying — don't `--force`. + +### Spur quirks +- **Output file `spur-.out` goes to the job's `WorkDir` = the CWD at submit time.** Submitting from `/tmp` writes `/tmp/spur-.out`; submitting from the SSH user's home writes it there. Do NOT `cd` into a 0700 dir (e.g. `/root/spur`) when SSHing as a non-root user — the `cd` fails and WorkDir silently becomes the user's home. Pin the submit CWD to `/tmp` for predictability, and search `/tmp /home /root ${SPUR_HOME}` when hunting for output. +- **`spur nodes` collapses by partition.** To verify per-host registration, loop `spur show node `. +- **`spur show node ` is a prefix match**, not exact. Colliding prefixes (`gpu-a` vs `gpu-ab`) return both; disambiguate with `awk -v n= '/^NodeName=/{p=($0=="NodeName="n)} p'`. +- **`spur show job` uses `JobState=COMPLETED` (uppercase).** Parse `JobState=[A-Z]+`. +- **Raft port 6821 is hardcoded** in spurctld (not a CLI flag). Preflight must include it. +- **Harmless log spam `invalid transition from Completed to Completed`** on followers after multi-node jobs — the job actually succeeded. + +### Multi-node / HA specifics +- **Agent `--hostname` must match the `[[nodes]]` name in `spur.conf`** and `spur show node `. Use `hostname -s` consistently. +- **Pass `--hostname` and `--address` explicitly to spurd.** Auto-detect picks `127.0.0.1`, breaking inter-node dispatch. +- **No shared-FS assumption.** Each node writes its own `spur-.out` locally; fetch from every agent. +- **HA needs a leader-elected wait**, not just port-listening. Loop on `no leader elected yet` until it clears. +- **HA `peers` list order must be stable across redeploys.** `node_id` is the 1-based position; reordering breaks openraft. To re-order, wipe state on every controller and redeploy. +- **`-N` in a multi-node job must not exceed the agent count**, or it stays PENDING. +- **Separate-compute HA:** controllers NOT in `AGENTS` must have no `spurd`. Verify with `systemctl is-active spurd` → `inactive`. +- **Client-side failover is NOT implemented in Spur 0.3.0.** `spurd --controller` is a single URL; if `CONTROLLERS[0]` dies, agents are stranded even with a live Raft leader. Production HA needs an L4 VIP / DNS in front of `:6817`. + +## Report back + +End the run with: +- Mode + counts (X controllers, Y agents), transport, accounting on/off +- Per-host: install source (binary-src vs installer), `systemctl is-active` for each daemon, log source (`journalctl -u spurctld`/`spurd`/`spurdbd`) +- `spur nodes` output +- HA only: which `node_id` became leader +- Accounting only: `sacct` output + DB job count +- Test job IDs + stdout (single, and multi if run) +- Any deviation from this skill — flag it so the skill can be patched diff --git a/deploy/ansible/.gitignore b/deploy/ansible/.gitignore new file mode 100644 index 00000000..5b99cfd3 --- /dev/null +++ b/deploy/ansible/.gitignore @@ -0,0 +1,7 @@ +# Real inventories contain host IPs and credentials — never commit them. +# Only the *.example.ini templates (placeholder addresses) are tracked. +inventory/*.ini +!inventory/*.example.ini + +# Ansible runtime artifacts +*.retry diff --git a/deploy/ansible/README.md b/deploy/ansible/README.md new file mode 100644 index 00000000..76a6f82f --- /dev/null +++ b/deploy/ansible/README.md @@ -0,0 +1,375 @@ +# Spur — Ansible Deployment + +One playbook (`deploy.yml`) stands up a Spur cluster in every supported shape, driven entirely by inventory. Daemons run as **systemd services** (survive reboot), the Slurm-compatible CLI names are symlinked, and optional **PostgreSQL + spurdbd accounting** is deployed by default. + +| Shape | Inventory pattern | Transport | +|---|---|---| +| Single-node | one host in **both** `spur_controllers` and `spur_agents` | local loopback | +| Multi-node — direct LAN | one host in `spur_controllers`, all compute in `spur_agents` | LAN IP, unencrypted | +| Multi-node — WireGuard mesh | as above + `spur_transport=wireguard` (single controller only) | encrypted mesh on `spur0` | +| HA — multi-controller Raft | **≥ 3 hosts** in `spur_controllers` (any number in `spur_agents`); auto-enabled | direct or wireguard | +| HA — separate compute | `spur_controllers` and `spur_agents` are **disjoint** host sets | direct or wireguard | + +The playbook is idempotent — re-running it on a healthy cluster re-applies config and restarts daemons. + +--- + +## End-to-end walkthrough + +The full journey from a clean machine to a running cluster is: **clone → build binaries → stage them → write inventory → run the playbook.** + +### 1. Clone the repo + +```bash +git clone https://github.com/ROCm/spur.git +cd spur +``` + +### 2. Build all four binaries + +Spur has no published GitHub release yet, so the upstream `install.sh` returns 403. Build the binaries yourself on a machine matching the targets' architecture/libc (the lab targets are Ubuntu 22.04 x86-64), then push them with the playbook (Step 3). + +```bash +# Prerequisites (Rust toolchain is pinned in rust-toolchain.toml): +sudo apt install -y protobuf-compiler + +# Build the four daemons/CLI in release mode: +cargo build --release -p spur-cli -p spurctld -p spurd -p spurdbd +``` + +This produces, under `target/release/`: + +| Binary | Role | +|---|---| +| `spur` | multi-call CLI (also symlinked to `sbatch`, `squeue`, `sinfo`, … by the playbook) | +| `spurctld` | controller / scheduler / Raft | +| `spurd` | node agent | +| `spurdbd` | accounting daemon (needs PostgreSQL) | + +Stage them into one directory the control node can read: + +```bash +mkdir -p /tmp/spur-bin +cp target/release/spur target/release/spurctld target/release/spurd target/release/spurdbd /tmp/spur-bin/ +``` + +You point the playbook at this directory with `spur_binary_src` (see Step 4). If you omit `spur_binary_src`, the playbook falls back to the upstream `install.sh` — which only works once ROCm/spur publishes releases. + +### 3. Install Ansible on the control node + +The control node is wherever you run `ansible-playbook` — your workstation is fine; it does not need to be part of the cluster. + +```bash +python3 -m pip install --user 'ansible-core>=2.14' +# WireGuard transport additionally needs the ansible.utils collection and the +# netaddr Python library on the control node: +ansible-galaxy collection install -r requirements.yml +python3 -m pip install --user netaddr +``` + +Target hosts need: SSH reachable, sudo or root, `systemd`, and (for the `install.sh` fallback path only) `curl` + `tar`. + +### 4. Write your inventory + +Copy an example and edit it. Real inventories are git-ignored (see `.gitignore`) so host addresses and credentials are never committed — only the `*.example.ini` placeholder templates are tracked. + +```bash +cd deploy/ansible +cp inventory/hosts.example.ini inventory/hosts.ini +$EDITOR inventory/hosts.ini +``` + +Point `spur_binary_src` at the directory from Step 2 (in `[all:vars]` or on the command line with `-e`). + +### 5. Run the playbook + +```bash +ansible-playbook deploy.yml -i inventory/hosts.ini -e spur_binary_src=/tmp/spur-bin +``` + +The play ends by running a single-node test job (and, when ≥ 2 agents, a multi-node `-N` job) and — when accounting is enabled — recording them in PostgreSQL. Look for the `spur nodes` output and job stdout near the end of the run. + +--- + +## Example commands per scenario + +Each command assumes binaries are staged at `/tmp/spur-bin` (Step 2). Drop `-e spur_binary_src=…` once upstream publishes releases. + +```bash +# Single-node (controller + agent on one host) +ansible-playbook deploy.yml -i inventory/single.ini -e spur_binary_src=/tmp/spur-bin + +# Multi-node, direct LAN (1 controller + N agents) +ansible-playbook deploy.yml -i inventory/multi.ini -e spur_binary_src=/tmp/spur-bin + +# HA, hyperconverged (3 controllers that also run agents) +ansible-playbook deploy.yml -i inventory/ha.ini -e spur_binary_src=/tmp/spur-bin + +# HA, separate compute (3 dedicated controllers + distinct agent hosts) +ansible-playbook deploy.yml -i inventory/ha-separate.ini -e spur_binary_src=/tmp/spur-bin + +# Any scenario WITHOUT accounting (no PostgreSQL/spurdbd; sacct unavailable, jobs still run) +ansible-playbook deploy.yml -i inventory/multi.ini -e spur_binary_src=/tmp/spur-bin -e spur_accounting_enabled=false + +# WireGuard transport (encrypted mesh instead of LAN) +ansible-playbook deploy.yml -i inventory/multi.ini -e spur_binary_src=/tmp/spur-bin -e spur_transport=wireguard + +# Preserve Raft state on re-deploy (production; don't wipe job history) +ansible-playbook deploy.yml -i inventory/ha.ini -e spur_binary_src=/tmp/spur-bin -e spur_wipe_state=false + +# Limit to one host (e.g. re-push config to a single agent) +ansible-playbook deploy.yml -i inventory/multi.ini -e spur_binary_src=/tmp/spur-bin --limit gpu-1 + +# Dry-run the change set without applying +ansible-playbook deploy.yml -i inventory/multi.ini -e spur_binary_src=/tmp/spur-bin --check --diff +``` + +--- + +## Inventory examples + +### Single-node + +```ini +[spur_controllers] +node1 ansible_host=10.0.0.10 ansible_user=root + +[spur_agents] +node1 ansible_host=10.0.0.10 ansible_user=root +``` + +### Multi-node, direct LAN + +```ini +[spur_controllers] +ctl ansible_host=10.0.0.10 ansible_user=root + +[spur_agents] +ctl ansible_host=10.0.0.10 ansible_user=root ; controller also runs an agent (hyperconverged) +gpu-1 ansible_host=10.0.0.11 ansible_user=root +gpu-2 ansible_host=10.0.0.12 ansible_user=root + +[all:vars] +spur_transport=direct +``` + +### Multi-node, WireGuard mesh + +```ini +[spur_controllers] +ctl ansible_host=ctl.example.com ansible_user=root + +[spur_agents] +gpu-1 ansible_host=gpu1.example.com ansible_user=root +gpu-2 ansible_host=gpu2.example.com ansible_user=root + +[all:vars] +spur_transport=wireguard +spur_wg_cidr=10.44.0.0/16 +spur_wg_port=51820 +``` + +> WireGuard is **single-controller only** — `spur net init` auto-assigns the controller `.1` and there is no multi-controller mesh command, so HA over WireGuard is not supported (use `direct` for HA). Agents are auto-assigned `.2`, `.3`, … by inventory position; override any host with `spur_wg_address=…`. Requires the `ansible.utils` collection on the control node (`ansible-galaxy collection install -r requirements.yml`). + +### HA — multi-controller Raft (hyperconverged) + +```ini +[spur_controllers] +ctl-0 ansible_host=10.0.0.10 ansible_user=root +ctl-1 ansible_host=10.0.0.11 ansible_user=root +ctl-2 ansible_host=10.0.0.12 ansible_user=root ; 3 → tolerates 1 failure + +[spur_agents] +ctl-0 ansible_host=10.0.0.10 ansible_user=root ; controllers also run agents +ctl-1 ansible_host=10.0.0.11 ansible_user=root +ctl-2 ansible_host=10.0.0.12 ansible_user=root +``` + +### HA — separate compute (control plane ≠ compute plane) + +```ini +[spur_controllers] +ctl-0 ansible_host=10.0.0.10 ansible_user=root +ctl-1 ansible_host=10.0.0.11 ansible_user=root +ctl-2 ansible_host=10.0.0.12 ansible_user=root + +[spur_agents] +gpu-1 ansible_host=10.0.0.21 ansible_user=root ; dedicated compute — no spurctld here +gpu-2 ansible_host=10.0.0.22 ansible_user=root +``` + +What the playbook does in HA mode: + +- Writes the same `peers = [...]` list to every controller's `spur.conf` (order matters — don't reorder controllers between deploys without wiping state). +- Assigns `node_id` = the controller's 1-based position in `groups['spur_controllers']`. +- Waits for a leader to be elected before proceeding to agent registration. +- Non-leader controllers forward client RPCs to the leader internally — clients can talk to any controller. + +**Always use an odd `N` ≥ 3 in production.** Even N gives the same fault tolerance as `N-1` and is strictly worse. `N=2` has zero fault tolerance — only useful for exercising the HA code path. + +**Client-side failover is NOT automatic** in Spur 0.3.0. `spurd --controller` accepts a single URL — the playbook points it at `groups['spur_controllers'][0]`. If that host dies, agents lose their connection even though Raft still has a leader on surviving controllers. Production HA needs an L4 VIP / DNS round-robin in front of `:6817` across all controllers, and setting `ansible_host` on the first controller (or overriding `spur_controller_addr`) to that VIP/DNS name. + +A full HA inventory template lives at `inventory/hosts.ha.example.ini`. + +--- + +## Accounting (PostgreSQL + spurdbd) + +Enabled by default (`spur_accounting_enabled: true`). Accounting is one service for the whole cluster; it runs on **`spur_accounting_host`** (default: the first controller). Before the controllers start, it: + +1. Installs `postgresql` + `postgresql-contrib`. +2. Creates the `spur` role and `spur` database idempotently. +3. Installs and starts the `spurdbd` systemd unit (`--migrate` creates the accounting tables). +4. Every controller's `spur.conf` gets an `[accounting]` block pointing at that host. + +**Placing accounting on a dedicated node:** set `spur_accounting_host` to any managed host — a controller, an agent, or a standalone node listed in its own group (e.g. `[spur_accounting_node]`). Pass it via `-e` (a play's `hosts:` field does not reliably read inventory `[all:vars]`): + +```bash +ansible-playbook deploy.yml -i inventory/hosts.ini -e spur_accounting_host=acct-0 +``` + +Then `sacct`/fairshare work. The playbook writes `SPUR_CONTROLLER_ADDR` and `SPUR_ACCOUNTING_ADDR` to `/etc/environment` on **controller nodes**, so the CLI (`squeue`/`sinfo`/`scontrol` and `sacct`/`sacctmgr`/`sreport`/`sshare`) works there with no per-command flags — including `sacct` from a controller that isn't the accounting host. On non-controller nodes (or to override), set the flag/env yourself: `sacct --accounting http://:6819` or `export SPUR_ACCOUNTING_ADDR=...`. To skip the entire stack: + +```bash +ansible-playbook deploy.yml -i inventory/hosts.ini -e spur_accounting_enabled=false +``` + +Job submission still works without accounting — only `sacct`/fairshare are unavailable. + +> The default DB credentials are `spur` / `spur` / `spur` — fine for a lab, **change `spur_accounting_db_password` for anything real.** + +--- + +## Variables (defaults in `group_vars/all.yml`) + +| Variable | Default | What it does | +|---|---|---| +| `spur_cluster_name` | `spur-cluster` | `cluster_name` in `spur.conf` | +| `spur_binary_src` | *(unset)* | Local dir of pre-built binaries to push. Unset → use upstream `install.sh`. | +| `spur_version` | `latest` | Install channel for `install.sh`: `latest` / `nightly` / `vX.Y.Z` | +| `spur_install_dir` | `/root/.local/bin` | Where binaries + Slurm symlinks land (added to `/etc/environment`) | +| `spur_home` | `/root/spur` | Per-host state/log/etc root | +| `spur_transport` | `direct` | `direct` or `wireguard` | +| `spur_accounting_enabled` | `true` | Deploy PostgreSQL + spurdbd. `false` to skip. | +| `spur_accounting_host` | first controller | Host that runs postgres + spurdbd (any managed host). Pass via `-e`. | +| `spur_accounting_db_name` / `_user` / `_password` | `spur` / `spur` / `spur` | Accounting DB credentials | +| `spur_accounting_port` | `6819` | spurdbd listen port | +| `spur_wg_cidr` / `spur_wg_port` / `spur_wg_interface` | `10.44.0.0/16` / `51820` / `spur0` | WireGuard mesh settings | +| `spur_controller_port` / `spur_agent_port` / `spur_raft_port` | `6817` / `6818` / `6821` | Listen ports | +| `spur_log_level` | `info` | Daemon log verbosity | +| `spur_wipe_state` | `false` | Wipe `~/spur/state` (Raft job queue/registrations) on (re)deploy. Default `false` so re-runs and upgrades are non-destructive; set `true` for a fresh install or an intentional Raft reinit. | + +Override per-run with `-e key=value` (repeatable): + +```bash +ansible-playbook deploy.yml -i inventory/hosts.ini -e spur_binary_src=/tmp/spur-bin -e spur_wipe_state=true +``` + +--- + +## Upgrading + +The playbook rolls out newer binaries idempotently. Re-running is non-destructive by default (`spur_wipe_state=false`) — job state and node registrations survive. + +```bash +# rebuild all four so they share the same WAL/DB schema (see caveat below) +cargo build --release -p spur-cli -p spurctld -p spurd -p spurdbd +cp target/release/{spur,spurctld,spurd,spurdbd} /tmp/spur-bin/ + +# re-run — the binary copy is checksum-based, so only changed binaries are +# pushed; daemons restart to pick them up; Raft state is preserved +ansible-playbook deploy.yml -i inventory/hosts.ini -e spur_binary_src=/tmp/spur-bin +``` + +- **Binaries are rolled out by content, not version string** — Ansible compares checksums, so a newer local build is pushed and a re-run with unchanged binaries is a near no-op. +- **Rebuild all four together.** The daemons share a Raft WAL schema and a spurdbd DB schema. Mixing binaries from different builds (e.g. rebuilding only `spurctld`) can leave a controller unable to parse a log written by a differently-versioned peer. +- **HA topology changes are guarded.** Because Spur 0.3.0 has no online Raft membership change, adding/removing/reordering a **controller** requires a Raft reinit. If you change the controller set with `spur_wipe_state=false`, the controller role fails early with an actionable message telling you to re-run with `-e spur_wipe_state=true`. **Compute agents are not Raft members** — add or remove them freely, no wipe needed. +- **Demoting a controller to agent-only leaves a stale `spurctld`.** If you move a host from `[spur_controllers]` to agent-only, stop and disable its controller first: `systemctl disable --now spurctld` on that host — otherwise the leftover daemon keeps the old membership and can block quorum. + +--- + +## Managing the cluster after deploy + +Daemons are systemd services, so use the normal tools on any host: + +```bash +systemctl status spurctld # on a controller +systemctl status spurd # on an agent +systemctl status spurdbd postgresql # on the first controller (accounting) +journalctl -u spurctld -f # follow logs +``` + +Basic cluster commands (binaries are on `PATH` via `/etc/environment`): + +```bash +spur nodes # partition/node summary +spur queue # job queue +spur submit job.sh +sacct # accounting (when enabled) — Slurm-compatible symlink to spur +``` + +--- + +## Tear down + +```bash +ansible-playbook teardown.yml -i inventory/hosts.ini # stop + disable daemons, remove units +ansible-playbook teardown.yml -i inventory/hosts.ini -e wipe=true # also rm -rf ~/spur +``` + +Teardown stops and disables the systemd services and reaps any stray daemons. It leaves PostgreSQL installed and the accounting database intact — to drop the DB, do it manually on the first controller (`sudo -u postgres dropdb spur; sudo -u postgres dropuser spur`). + +--- + +## Hard-won gotchas baked into these roles + +These are real bugs we hit during validation — listed so anyone reading the playbook understands why the roles look the way they do. + +### Daemon / process management +- **Daemons run as systemd units**, not `nohup` — `/etc/systemd/system/spur{ctld,d,dbd}.service`, `enabled` (survive reboot), `Restart=on-failure`. The roles `daemon-reload` after templating a unit and use `state: restarted` so a redeploy always picks up new binaries/config. +- **`-D` is `--foreground`, not "daemonize"** (`crates/spurctld/src/main.rs`). The systemd units run the binary in the foreground under `Type=simple`, which is correct — do not add `-D`. +- **`pkill -f spurd` also kills `spurctld`** (substring match). Teardown uses `pkill -x` (exact name) only. +- **Stop-before-wipe ordering.** The controller role stops spurctld *before* wiping `~/spur/state`, so the daemon can't rewrite the Raft log mid-delete. + +### Install +- **Upstream `install.sh` returns 403** because ROCm/spur has no published release yet. Use `spur_binary_src` to push locally-built binaries. The symlink + verify tasks run on both paths. +- **`spur --version` is not a supported flag** — it errors. The roles check for the binary with `stat`, not by running `--version`. +- **Slurm-compat symlinks** (`sbatch`/`squeue`/`sinfo`/`scancel`/`sacct`/`scontrol`/`salloc`/`srun` → `spur`) are created regardless of install source; the `spur` multi-call binary dispatches on argv[0]. +- **A stale dpkg lock** during the accounting apt install means another apt/unattended-upgrade is running — wait for it (or clear a genuinely hung `apt-get`); don't `--force`. + +### Accounting +- **Accounting is deployed before the controller** so spurdbd is listening when spurctld first reads its `[accounting]` block. +- **spurdbd needs `--migrate`** to create the accounting tables on first start. +- **With `spur_wipe_state=true`, the Raft job-id counter resets each deploy**, so re-deploys reuse job ids 1, 2, … which upsert onto the same accounting rows. Set `spur_wipe_state=false` to preserve job history. + +### Spur quirks +- **Job stdout file lands in spurd's working directory at startup.** The `spurd` systemd unit sets `WorkingDirectory={{ spur_home }}` so output is predictably at `{{ spur_home }}/spur-.out`, even though `spur show job` reports the submitter's `WorkDir=`. +- **The single-node job's host is unpredictable.** The backfill scheduler picks any idle node, so `spur_verify` searches every agent rather than assuming the controller. +- **No shared-FS assumption in multi-node verify.** The play fetches `spur-.out` from each agent, not just the controller. +- **`spur nodes` collapses by partition** — the "NODES" column is a count, not one row per host. To confirm each expected host registered, loop `spur show node `. +- **`spur show job` uses `JobState=COMPLETED` (uppercase)**, not `State: Completed`. The wait-loop greps `JobState=[A-Z]+`. +- **Harmless log spam `invalid transition from Completed to Completed`** on followers after a multi-node job (`crates/spurctld/src/cluster.rs`). The leader already reported terminal state; the follower's redundant report is rejected. The job succeeded. +- **Raft port 6821 is hardcoded** in spurctld and isn't a flag in 0.3.0. Preflight checks it alongside 6817/6818. + +### Multi-node / HA specifics +- **Agent `--hostname` must match the `[[nodes]]` name in `spur.conf`.** The templates use `ansible_hostname` (short) on both sides. +- **HA needs a leader-elected wait**, not just a port-listening wait. `spurctld` binds `:6817` instantly but returns `Status::unavailable("no leader elected yet")` until quorum forms. The controller role loops on that message in `spur nodes` until it clears. +- **`peers` list order matters across controllers.** `node_id` is the 1-based position; openraft refuses to start if a node's `node_id` doesn't match its position. The role derives both from `groups['spur_controllers']` — don't reorder that group between deploys without wiping state. +- **`delegate_facts: true` evaluates `set_fact` on the controller, not the delegated target.** Use `hostvars[item].ansible_hostname` when setting a per-host fact via `delegate_to`. +- **`ansible.builtin.command` runs without a shell** — `command -v X` fails (bash builtin). Use `ansible.builtin.shell` with `executable: /bin/bash`. + +If you hit a new gotcha, please add it here and (where applicable) encode the fix in the roles. + +--- + +## Verified + +`spur 0.3.0`, Ubuntu 22.04, validated across all four topologies on a 4-node cluster (fresh install — PostgreSQL purged beforehand), Ansible run from an operator workstation. Every scenario ran basic commands, drove a sample job to `COMPLETED`, confirmed all daemons `active` under systemd, and recorded jobs in PostgreSQL via `sacct`: + +- **Single-node** — controller + agent on one host; job COMPLETED, accounted. +- **Multi-node** — 1 controller + 2 agents; `-N 2` job fanned across both nodes, accounted. +- **HA hyperconverged** — 3 controllers (Raft) + 3 agents; leader elected, `-N 3` job across all three, accounted. +- **HA + separate compute** — 3 dedicated controllers (no agent) + 1 dedicated agent; leader elected, controllers ran no `spurd`, job ran on the separate compute node, accounted. +- **Accounting disabled** (`-e spur_accounting_enabled=false`) — PostgreSQL/spurdbd skipped, job still COMPLETED. + +WireGuard transport is supported via `roles/spur_wireguard`. Enable it with `-e spur_transport=wireguard`. It is **single-controller only** (`spur net init` auto-assigns the controller `.1`; there is no multi-controller mesh command), so HA runs over `direct`. Agents auto-assign `.2`, `.3`, … Requires the `ansible.utils` collection (`ansible-galaxy collection install -r requirements.yml`). diff --git a/deploy/ansible/ansible.cfg b/deploy/ansible/ansible.cfg new file mode 100644 index 00000000..a741a07e --- /dev/null +++ b/deploy/ansible/ansible.cfg @@ -0,0 +1,14 @@ +[defaults] +inventory = inventory/hosts.ini +# On by default for safety. For first-time provisioning of fresh hosts whose +# keys aren't yet in known_hosts, override per-run with +# ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook ... +host_key_checking = True +forks = 20 +retry_files_enabled = False +deprecation_warnings = False +interpreter_python = auto_silent + +[ssh_connection] +pipelining = True +ssh_args = -o ControlMaster=auto -o ControlPersist=60s -o BatchMode=yes -o ConnectTimeout=15 diff --git a/deploy/ansible/deploy.yml b/deploy/ansible/deploy.yml new file mode 100644 index 00000000..8cb07b12 --- /dev/null +++ b/deploy/ansible/deploy.yml @@ -0,0 +1,62 @@ +--- +# Spur deployment — works for single-node, multi-node-direct, multi-node-wireguard. +# +# Run: +# cd deploy/ansible +# ansible-playbook deploy.yml -i inventory/hosts.ini +# +# Subset to specific hosts: +# ansible-playbook deploy.yml -i inventory/hosts.ini --limit gpu-1 +# +# Tear down (keeps installed binaries): +# ansible-playbook teardown.yml -i inventory/hosts.ini + +- name: Preflight + install Spur on all hosts + # Include the accounting host so a dedicated (non-controller/agent) accounting + # node still gets the spurdbd binary. + hosts: "spur_controllers:spur_agents:{{ spur_accounting_host | default(groups['spur_controllers'][0]) }}" + gather_facts: yes + any_errors_fatal: true + roles: + - role: spur_install + +- name: Configure WireGuard mesh (when spur_transport == wireguard) + hosts: spur_controllers:spur_agents + gather_facts: no + any_errors_fatal: true + roles: + - role: spur_wireguard + when: spur_transport == 'wireguard' + +- name: Deploy accounting stack (postgres + spurdbd) on the accounting host + # Runs BEFORE the controller so spurdbd is listening when spurctld starts and + # first tries to reach accounting. Accounting is a single service for the whole + # cluster; it runs on spur_accounting_host (default: first controller, but may be + # any managed host incl. a dedicated node). Set spur_accounting_enabled=false to skip. + hosts: "{{ spur_accounting_host | default(groups['spur_controllers'][0]) }}" + gather_facts: yes + any_errors_fatal: true + roles: + - role: spur_accounting + when: spur_accounting_enabled | bool + +- name: Start controller + hosts: spur_controllers + gather_facts: no + any_errors_fatal: true + roles: + - role: spur_controller + +- name: Start agents + hosts: spur_agents + gather_facts: no + any_errors_fatal: true + serial: 0 # parallel + roles: + - role: spur_agent + +- name: Verify with test jobs + hosts: spur_controllers[0] + gather_facts: no + roles: + - role: spur_verify diff --git a/deploy/ansible/group_vars/all.yml b/deploy/ansible/group_vars/all.yml new file mode 100644 index 00000000..d5bb1af5 --- /dev/null +++ b/deploy/ansible/group_vars/all.yml @@ -0,0 +1,54 @@ +--- +# Cluster identity +spur_cluster_name: spur-cluster + +# Install +spur_version: latest # latest | nightly | vX.Y.Z +spur_install_dir: /root/.local/bin +spur_home: /root/spur + +# Transport: direct (LAN, no encryption) | wireguard (encrypted mesh) +spur_transport: direct + +# WireGuard mesh settings (only used when spur_transport == wireguard) +spur_wg_cidr: 10.44.0.0/16 +spur_wg_port: 51820 +spur_wg_interface: spur0 + +# Ports +spur_controller_port: 6817 +spur_agent_port: 6818 +spur_raft_port: 6821 + +# Accounting (PostgreSQL + spurdbd). OPTIONAL — set spur_accounting_enabled: false to skip. +# Single service for the whole cluster. Runs on spur_accounting_host (default: the +# first controller), which may be any managed host — a controller, an agent, or a +# dedicated node. Controllers report usage to it over gRPC; it owns the local postgres. +spur_accounting_enabled: true +# spur_accounting_host: set in inventory to pin the accounting node (any managed +# host). Left unset here so an inventory value wins; defaults to the first +# controller via `| default(groups['spur_controllers'][0])` where referenced. +spur_accounting_db_name: spur +spur_accounting_db_user: spur +spur_accounting_db_password: spur +spur_accounting_port: 6819 + +# Logging +spur_log_level: info + +# Wipe Raft state on (re)deploy. Default false so re-runs/upgrades are non-destructive; set true for a fresh install. +spur_wipe_state: false + +# HA mode is enabled automatically when groups['spur_controllers'] has more than one host. +# In HA mode: +# - every controller is given a node_id (its 1-based position in groups['spur_controllers']) +# - spur.conf gets `peers = ["ctl1:6821", "ctl2:6821", ...]` (same order on every host) +# - agents are pointed at the FIRST controller; openraft forwards writes to the leader, +# but agents do not yet fail over to a different controller if the first one dies +# (Spur 0.3.0 limitation — `spurd --controller` takes a single URL). Put an L4 VIP / +# DNS round-robin in front of port 6817 for real failover. See README. +spur_ha_enabled: "{{ (groups.get('spur_controllers', []) | length) > 1 }}" + +# Whether the controller node also runs spurd (lets controller be a compute target). +# Auto: true if a host is in BOTH controller and agent groups. +spur_controller_runs_agent: "{{ inventory_hostname in groups.get('spur_agents', []) and inventory_hostname in groups.get('spur_controllers', []) }}" diff --git a/deploy/ansible/inventory/hosts.example.ini b/deploy/ansible/inventory/hosts.example.ini new file mode 100644 index 00000000..a78c0abf --- /dev/null +++ b/deploy/ansible/inventory/hosts.example.ini @@ -0,0 +1,36 @@ +; ============================================================================ +; Spur Ansible inventory — examples for three deployment shapes. +; Copy this file to hosts.ini and edit. Pick ONE shape per inventory file. +; ============================================================================ + +; --- Single-node (controller + agent on the same host) ---------------------- +; [spur_controllers] +; node1 ansible_host=10.0.0.10 ansible_user=root +; +; [spur_agents] +; node1 ansible_host=10.0.0.10 ansible_user=root + +; --- Multi-node, direct LAN transport --------------------------------------- +; [spur_controllers] +; ctl ansible_host=10.0.0.10 ansible_user=root +; +; [spur_agents] +; ctl ansible_host=10.0.0.10 ansible_user=root ; controller also runs an agent (optional) +; gpu-1 ansible_host=10.0.0.11 ansible_user=root +; gpu-2 ansible_host=10.0.0.12 ansible_user=root +; +; [all:vars] +; spur_transport=direct + +; --- Multi-node, WireGuard mesh transport ----------------------------------- +; [spur_controllers] +; ctl ansible_host=ctl.example.com ansible_user=root spur_wg_address=10.44.0.1 +; +; [spur_agents] +; gpu-1 ansible_host=gpu1.example.com ansible_user=root spur_wg_address=10.44.0.2 +; gpu-2 ansible_host=gpu2.example.com ansible_user=root spur_wg_address=10.44.0.3 +; +; [all:vars] +; spur_transport=wireguard +; spur_wg_cidr=10.44.0.0/16 +; spur_wg_port=51820 diff --git a/deploy/ansible/inventory/hosts.ha.example.ini b/deploy/ansible/inventory/hosts.ha.example.ini new file mode 100644 index 00000000..3da311ac --- /dev/null +++ b/deploy/ansible/inventory/hosts.ha.example.ini @@ -0,0 +1,36 @@ +; ============================================================================ +; HA Spur cluster — 3 controllers + N agents. +; +; Quorum math: tolerate floor((N-1)/2) controller failures. +; 3 controllers → tolerates 1 failure +; 5 controllers → tolerates 2 failures +; +; HA mode is auto-enabled when len(spur_controllers) > 1. The playbook: +; - assigns node_id = position-in-group (1-based) +; - writes `peers = ["ctl1:6821", "ctl2:6821", "ctl3:6821"]` to every controller's +; spur.conf in the SAME order (openraft requires this) +; +; CAVEAT: spurd --controller is single-target in Spur 0.3.0. Agents are pointed at +; the first controller in the inventory. If THAT controller dies, agents lose +; their connection even though the Raft cluster still has a leader. Put an +; L4 VIP / DNS round-robin in front of the controllers' :6817 to get real +; client-side failover, and set ansible_host to the VIP/DNS name on the first +; controller (or override spur_controller_addr globally). +; ============================================================================ + +[spur_controllers] +ctl-0 ansible_host=10.0.0.10 ansible_user=root +ctl-1 ansible_host=10.0.0.11 ansible_user=root +ctl-2 ansible_host=10.0.0.12 ansible_user=root + +[spur_agents] +; Controllers can also be compute targets (small clusters): +ctl-0 ansible_host=10.0.0.10 ansible_user=root +ctl-1 ansible_host=10.0.0.11 ansible_user=root +ctl-2 ansible_host=10.0.0.12 ansible_user=root +; Dedicated workers: +gpu-1 ansible_host=10.0.0.21 ansible_user=root +gpu-2 ansible_host=10.0.0.22 ansible_user=root + +[all:vars] +spur_transport=direct diff --git a/deploy/ansible/requirements.yml b/deploy/ansible/requirements.yml new file mode 100644 index 00000000..f77ed41e --- /dev/null +++ b/deploy/ansible/requirements.yml @@ -0,0 +1,9 @@ +--- +# Control-node collection dependencies. +# Install with: ansible-galaxy collection install -r requirements.yml +# +# The ansible.utils ipaddr/ipmath filters also require the Python `netaddr` +# library on the control node: python3 -m pip install netaddr +collections: + # Provides ansible.utils.ipaddr / ipmath used by the spur_wireguard role. + - name: ansible.utils diff --git a/deploy/ansible/roles/spur_accounting/defaults/main.yml b/deploy/ansible/roles/spur_accounting/defaults/main.yml new file mode 100644 index 00000000..7738436b --- /dev/null +++ b/deploy/ansible/roles/spur_accounting/defaults/main.yml @@ -0,0 +1,6 @@ +--- +spur_accounting_enabled: true +spur_accounting_db_name: spur +spur_accounting_db_user: spur +spur_accounting_db_password: spur +spur_accounting_port: 6819 diff --git a/deploy/ansible/roles/spur_accounting/tasks/main.yml b/deploy/ansible/roles/spur_accounting/tasks/main.yml new file mode 100644 index 00000000..3727ca05 --- /dev/null +++ b/deploy/ansible/roles/spur_accounting/tasks/main.yml @@ -0,0 +1,102 @@ +--- +# Optional accounting stack: PostgreSQL + spurdbd. Runs on the first controller only +# (accounting is a single service for the whole cluster). Everything is gated on +# spur_accounting_enabled so `-e spur_accounting_enabled=false` skips the stack entirely. + +- name: Install PostgreSQL + ansible.builtin.apt: + name: + - postgresql + - postgresql-contrib + state: present + update_cache: true + when: spur_accounting_enabled | bool + +- name: Ensure PostgreSQL is enabled and started + ansible.builtin.service: + name: postgresql + enabled: true + state: started + when: spur_accounting_enabled | bool + +# The community.postgresql collection is not guaranteed to be installed on the +# control node, so create the role/database with psql via `become_user: postgres`. +# Both operations are made idempotent by checking for existence first. +- name: Check whether the spur DB role exists + ansible.builtin.command: >- + psql -tAc "SELECT 1 FROM pg_roles WHERE rolname = '{{ spur_accounting_db_user }}'" + become: true + become_user: postgres + register: spur_db_role_check + changed_when: false + when: spur_accounting_enabled | bool + +- name: Create the spur DB role + ansible.builtin.command: >- + psql -c "CREATE ROLE {{ spur_accounting_db_user }} LOGIN PASSWORD '{{ spur_accounting_db_password }}'" + become: true + become_user: postgres + when: + - spur_accounting_enabled | bool + - spur_db_role_check.stdout | trim != "1" + +- name: Check whether the spur database exists + ansible.builtin.command: >- + psql -tAc "SELECT 1 FROM pg_database WHERE datname = '{{ spur_accounting_db_name }}'" + become: true + become_user: postgres + register: spur_db_check + changed_when: false + when: spur_accounting_enabled | bool + +- name: Create the spur database owned by the spur role + ansible.builtin.command: >- + psql -c "CREATE DATABASE {{ spur_accounting_db_name }} OWNER {{ spur_accounting_db_user }}" + become: true + become_user: postgres + when: + - spur_accounting_enabled | bool + - spur_db_check.stdout | trim != "1" + +- name: Verify spurdbd binary is present + ansible.builtin.stat: + path: "{{ spur_install_dir }}/spurdbd" + register: spurdbd_bin + when: spur_accounting_enabled | bool + +- name: Fail if spurdbd binary is missing + ansible.builtin.fail: + msg: >- + spurdbd binary not found at {{ spur_install_dir }}/spurdbd. The upstream + install.sh should install it alongside spur/spurctld/spurd — re-run the + installer or disable accounting with spur_accounting_enabled=false. + when: + - spur_accounting_enabled | bool + - not spurdbd_bin.stat.exists + +- name: Install spurdbd systemd unit + ansible.builtin.template: + src: spurdbd.service.j2 + dest: /etc/systemd/system/spurdbd.service + mode: "0644" + register: spurdbd_unit + when: spur_accounting_enabled | bool + +- name: Reload systemd so the unit is picked up + ansible.builtin.systemd: + daemon_reload: true + when: spur_accounting_enabled | bool + +- name: Enable and (re)start spurdbd + ansible.builtin.systemd: + name: spurdbd + enabled: true + state: restarted + when: spur_accounting_enabled | bool + +- name: Wait for spurdbd to listen on {{ spur_accounting_port }} + ansible.builtin.wait_for: + port: "{{ spur_accounting_port }}" + host: "127.0.0.1" + timeout: 30 + when: spur_accounting_enabled | bool diff --git a/deploy/ansible/roles/spur_accounting/templates/spurdbd.service.j2 b/deploy/ansible/roles/spur_accounting/templates/spurdbd.service.j2 new file mode 100644 index 00000000..c9dd35a1 --- /dev/null +++ b/deploy/ansible/roles/spur_accounting/templates/spurdbd.service.j2 @@ -0,0 +1,16 @@ +[Unit] +Description=Spur Accounting Daemon (spurdbd) +After=network-online.target postgresql.service +Wants=network-online.target +Requires=postgresql.service + +[Service] +Type=simple +ExecStart={{ spur_install_dir }}/spurdbd --database-url postgresql://{{ spur_accounting_db_user }}:{{ spur_accounting_db_password }}@localhost/{{ spur_accounting_db_name }} --listen [::]:{{ spur_accounting_port }} --migrate --log-level {{ spur_log_level }} +Restart=on-failure +RestartSec=3 +User=root +LimitNOFILE=65536 + +[Install] +WantedBy=multi-user.target diff --git a/deploy/ansible/roles/spur_agent/defaults/main.yml b/deploy/ansible/roles/spur_agent/defaults/main.yml new file mode 100644 index 00000000..ed97d539 --- /dev/null +++ b/deploy/ansible/roles/spur_agent/defaults/main.yml @@ -0,0 +1 @@ +--- diff --git a/deploy/ansible/roles/spur_agent/tasks/main.yml b/deploy/ansible/roles/spur_agent/tasks/main.yml new file mode 100644 index 00000000..3cae066f --- /dev/null +++ b/deploy/ansible/roles/spur_agent/tasks/main.yml @@ -0,0 +1,43 @@ +--- +- name: Resolve agent's advertised address + ansible.builtin.set_fact: + spur_agent_listen_ip: >- + {{ spur_wg_address if spur_transport == 'wireguard' + else (ansible_host | default(ansible_default_ipv4.address)) }} + spur_node_name: "{{ spur_node_name | default(ansible_hostname) }}" + +- name: Resolve controller address agents will talk to + # NOTE: spurd --controller takes a single URL in Spur 0.3.0. In HA mode we still + # point at the first controller — non-leader controllers forward writes via the + # Raft cluster. Real failover when ctl[0] dies needs a VIP / DNS in front of 6817. + ansible.builtin.set_fact: + spur_controller_addr: >- + {{ hostvars[groups['spur_controllers'][0]]['spur_wg_address'] + if spur_transport == 'wireguard' + else hostvars[groups['spur_controllers'][0]]['ansible_host'] + | default(hostvars[groups['spur_controllers'][0]]['ansible_default_ipv4']['address']) }} + +- name: Install spurd systemd unit + # WorkingDirectory in the unit is {{ spur_home }} so job stdout (spur-.out) + # always lands there regardless of where the daemon was launched from. + ansible.builtin.template: + src: spurd.service.j2 + dest: /etc/systemd/system/spurd.service + mode: "0644" + register: spurd_unit + +- name: Reload systemd so the unit is picked up + ansible.builtin.systemd: + daemon_reload: true + +- name: Enable and (re)start spurd + ansible.builtin.systemd: + name: spurd + enabled: true + state: restarted + +- name: Wait for spurd to listen on {{ spur_agent_port }} + ansible.builtin.wait_for: + port: "{{ spur_agent_port }}" + host: "127.0.0.1" + timeout: 30 diff --git a/deploy/ansible/roles/spur_agent/templates/spurd.service.j2 b/deploy/ansible/roles/spur_agent/templates/spurd.service.j2 new file mode 100644 index 00000000..e2ff17ee --- /dev/null +++ b/deploy/ansible/roles/spur_agent/templates/spurd.service.j2 @@ -0,0 +1,17 @@ +[Unit] +Description=Spur Node Agent (spurd) +After=network-online.target +Wants=network-online.target + +[Service] +Type=simple +Environment=HOME={{ spur_home }} +WorkingDirectory={{ spur_home }} +ExecStart={{ spur_install_dir }}/spurd --controller http://{{ spur_controller_addr }}:{{ spur_controller_port }} --hostname {{ spur_node_name }} --address {{ spur_agent_listen_ip }} --listen 0.0.0.0:{{ spur_agent_port }} --log-level {{ spur_log_level }} +Restart=on-failure +RestartSec=3 +User=root +LimitNOFILE=65536 + +[Install] +WantedBy=multi-user.target diff --git a/deploy/ansible/roles/spur_controller/defaults/main.yml b/deploy/ansible/roles/spur_controller/defaults/main.yml new file mode 100644 index 00000000..ed97d539 --- /dev/null +++ b/deploy/ansible/roles/spur_controller/defaults/main.yml @@ -0,0 +1 @@ +--- diff --git a/deploy/ansible/roles/spur_controller/tasks/main.yml b/deploy/ansible/roles/spur_controller/tasks/main.yml new file mode 100644 index 00000000..5d58555e --- /dev/null +++ b/deploy/ansible/roles/spur_controller/tasks/main.yml @@ -0,0 +1,156 @@ +--- +- name: Resolve every controller's listen IP (needed for the peers list on all hosts) + ansible.builtin.set_fact: + spur_controller_listen_ip: >- + {{ hostvars[item].spur_wg_address if spur_transport == 'wireguard' + else (hostvars[item].ansible_host | default(hostvars[item].ansible_default_ipv4.address)) }} + delegate_to: "{{ item }}" + delegate_facts: true + loop: "{{ groups['spur_controllers'] }}" + run_once: true + +- name: Assign 1-based node_id to each controller (position in inventory) + ansible.builtin.set_fact: + spur_controller_node_id: "{{ groups['spur_controllers'].index(item) + 1 }}" + delegate_to: "{{ item }}" + delegate_facts: true + loop: "{{ groups['spur_controllers'] }}" + run_once: true + +- name: Resolve the accounting host's advertised IP (for the [accounting] block) + vars: + acct_host: "{{ spur_accounting_host | default(groups['spur_controllers'][0]) }}" + ansible.builtin.set_fact: + spur_accounting_listen_ip: >- + {{ hostvars[acct_host].spur_wg_address if spur_transport == 'wireguard' + else (hostvars[acct_host].ansible_host + | default(hostvars[acct_host].ansible_default_ipv4.address)) }} + run_once: true + when: spur_accounting_enabled | bool + +- name: Resolve short node name for each agent (used by spur.conf and --hostname) + ansible.builtin.set_fact: + spur_node_name: "{{ hostvars[item].ansible_hostname }}" + delegate_to: "{{ item }}" + delegate_facts: true + loop: "{{ groups['spur_agents'] }}" + run_once: true + when: hostvars[item].spur_node_name is not defined + +# Client env so the CLI (squeue/sinfo/scontrol and sacct/sacctmgr/...) works on +# controllers without per-command flags. Controller-nodes only. Each controller +# points SPUR_CONTROLLER_ADDR at its own local spurctld. +- name: Set SPUR_CONTROLLER_ADDR in /etc/environment (controllers) + ansible.builtin.lineinfile: + path: /etc/environment + regexp: '^SPUR_CONTROLLER_ADDR=' + line: "SPUR_CONTROLLER_ADDR=http://{{ spur_controller_listen_ip }}:{{ spur_controller_port }}" + +- name: Set SPUR_ACCOUNTING_ADDR in /etc/environment (controllers) + ansible.builtin.lineinfile: + path: /etc/environment + regexp: '^SPUR_ACCOUNTING_ADDR=' + line: "SPUR_ACCOUNTING_ADDR=http://{{ hostvars[groups['spur_controllers'][0]]['spur_accounting_listen_ip'] }}:{{ spur_accounting_port }}" + when: spur_accounting_enabled | bool + +- name: Write spur.conf + ansible.builtin.template: + src: spur.conf.j2 + dest: "{{ spur_home }}/etc/spur.conf" + mode: "0644" + register: spur_conf + +- name: Install spurctld systemd unit + ansible.builtin.template: + src: spurctld.service.j2 + dest: /etc/systemd/system/spurctld.service + mode: "0644" + register: spurctld_unit + +- name: Reload systemd so the unit is picked up + ansible.builtin.systemd: + daemon_reload: true + +# openraft ignores config peers after first init and uses the persisted +# membership, so a changed controller set can't take effect on restart. +# Fail loudly rather than break silently. Agents are not Raft members and +# can be added/removed freely without a wipe. +- name: Read persisted Raft peer addresses (HA, no-wipe) + ansible.builtin.shell: | + set -o pipefail + grep -ho '"addr":"[^"]*"' {{ spur_home }}/state/raft/log/*.json 2>/dev/null \ + | sed -E 's/"addr":"([^"]*)"/\1/' | sort -u + args: + executable: /bin/bash + register: raft_peers + changed_when: false + failed_when: false + when: + - spur_ha_enabled | bool + - not (spur_wipe_state | bool) + +- name: Fail if the controller (Raft) set changed without a state wipe + ansible.builtin.fail: + msg: >- + Persisted Raft membership {{ raft_peers.stdout_lines | sort }} does not + match the inventory controllers + {{ groups['spur_controllers'] | map('extract', hostvars, 'spur_controller_listen_ip') | map('regex_replace', '$', ':' ~ spur_raft_port) | sort }}. + Spur 0.3.0 has no online membership change, so adding/removing/reordering a + controller needs a Raft reinit: re-run with -e spur_wipe_state=true. + Adding/removing compute agents does not require this. + vars: + desired_peers: "{{ groups['spur_controllers'] | map('extract', hostvars, 'spur_controller_listen_ip') | map('regex_replace', '$', ':' ~ spur_raft_port) | sort }}" + when: + - spur_ha_enabled | bool + - not (spur_wipe_state | bool) + - raft_peers.stdout | default('') | length > 0 + - (raft_peers.stdout_lines | sort) != desired_peers + +# Stop BEFORE wiping state so spurctld does not rewrite the Raft log we are about to delete. +- name: Stop spurctld before wiping state + ansible.builtin.systemd: + name: spurctld + state: stopped + failed_when: false + +- name: Wipe Raft state (when spur_wipe_state is true) + ansible.builtin.file: + path: "{{ spur_home }}/state" + state: absent + when: spur_wipe_state | bool + +- name: Recreate state dir after wipe + ansible.builtin.file: + path: "{{ spur_home }}/state" + state: directory + mode: "0755" + when: spur_wipe_state | bool + +- name: Enable and (re)start spurctld + ansible.builtin.systemd: + name: spurctld + enabled: true + state: restarted + +- name: Wait for spurctld to listen on {{ spur_controller_port }} + ansible.builtin.wait_for: + port: "{{ spur_controller_port }}" + host: "127.0.0.1" + timeout: 30 + +- name: Wait for Raft leader to be elected (HA mode needs all peers up before quorum) + # `spur nodes` returns the error `no leader elected yet` until election succeeds. + # In HA mode this can take a few seconds while peers find each other. + ansible.builtin.shell: | + for i in $(seq 1 60); do + out=$({{ spur_install_dir }}/spur nodes 2>&1) + echo "$out" | grep -qE 'no leader|not the Raft leader|cannot reach leader|transport error|Connection refused' || { echo OK; exit 0; } + sleep 1 + done + echo "timeout waiting for leader: $out" + exit 1 + args: + executable: /bin/bash + changed_when: false + when: spur_ha_enabled | bool + run_once: true diff --git a/deploy/ansible/roles/spur_controller/templates/spur.conf.j2 b/deploy/ansible/roles/spur_controller/templates/spur.conf.j2 new file mode 100644 index 00000000..b5a31fe1 --- /dev/null +++ b/deploy/ansible/roles/spur_controller/templates/spur.conf.j2 @@ -0,0 +1,42 @@ +cluster_name = "{{ spur_cluster_name }}" + +[controller] +listen_addr = "[::]:{{ spur_controller_port }}" +hosts = [{% for h in groups['spur_controllers'] %}"{{ hostvars[h]['spur_controller_listen_ip'] }}"{% if not loop.last %}, {% endif %}{% endfor %}] +state_dir = "{{ spur_home }}/state" +raft_listen_addr = "[::]:{{ spur_raft_port }}" +{% if spur_ha_enabled %} +node_id = {{ spur_controller_node_id }} +peers = [{% for h in groups['spur_controllers'] %}"{{ hostvars[h]['spur_controller_listen_ip'] }}:{{ spur_raft_port }}"{% if not loop.last %}, {% endif %}{% endfor %}] +{% endif %} + +[scheduler] +plugin = "backfill" +interval_secs = 1 + +{% if spur_accounting_enabled | default(true) %} +[accounting] +host = "{{ spur_accounting_listen_ip }}:{{ spur_accounting_port | default(6819) }}" +database_url = "postgresql://{{ spur_accounting_db_user | default('spur') }}:{{ spur_accounting_db_password | default('spur') }}@localhost/{{ spur_accounting_db_name | default('spur') }}" +fairshare_refresh_secs = 30 +{% endif %} + +[network] +wg_enabled = {{ (spur_transport == 'wireguard') | string | lower }} +{% if spur_transport == 'wireguard' %} +wg_interface = "{{ spur_wg_interface }}" +{% endif %} +agent_port = {{ spur_agent_port }} + +{% for h in groups['spur_agents'] %} +[[nodes]] +names = "{{ hostvars[h]['spur_node_name'] }}" +cpus = {{ hostvars[h]['ansible_processor_vcpus'] }} +memory_mb = {{ (hostvars[h]['ansible_memtotal_mb'] * 0.9) | int }} + +{% endfor %} +[[partitions]] +name = "default" +default = true +nodes = "{{ groups['spur_agents'] | map('extract', hostvars, 'spur_node_name') | join(',') }}" +max_time = "INFINITE" diff --git a/deploy/ansible/roles/spur_controller/templates/spurctld.service.j2 b/deploy/ansible/roles/spur_controller/templates/spurctld.service.j2 new file mode 100644 index 00000000..ce7816a0 --- /dev/null +++ b/deploy/ansible/roles/spur_controller/templates/spurctld.service.j2 @@ -0,0 +1,17 @@ +[Unit] +Description=Spur Controller Daemon (spurctld) +After=network-online.target +Wants=network-online.target + +[Service] +Type=simple +Environment=HOME={{ spur_home }} +WorkingDirectory={{ spur_home }} +ExecStart={{ spur_install_dir }}/spurctld -f {{ spur_home }}/etc/spur.conf --state-dir {{ spur_home }}/state --log-level {{ spur_log_level }} +Restart=on-failure +RestartSec=3 +User=root +LimitNOFILE=65536 + +[Install] +WantedBy=multi-user.target diff --git a/deploy/ansible/roles/spur_install/defaults/main.yml b/deploy/ansible/roles/spur_install/defaults/main.yml new file mode 100644 index 00000000..7c2a3171 --- /dev/null +++ b/deploy/ansible/roles/spur_install/defaults/main.yml @@ -0,0 +1,2 @@ +--- +# Inherits from group_vars/all.yml; placeholder for role-local overrides. diff --git a/deploy/ansible/roles/spur_install/tasks/main.yml b/deploy/ansible/roles/spur_install/tasks/main.yml new file mode 100644 index 00000000..ecdec0f6 --- /dev/null +++ b/deploy/ansible/roles/spur_install/tasks/main.yml @@ -0,0 +1,137 @@ +--- +- name: Preflight — check required commands + ansible.builtin.shell: "command -v {{ item }}" + args: + executable: /bin/bash + register: pf_cmd + changed_when: false + failed_when: pf_cmd.rc != 0 + loop: + - curl + - tar + - bash + +- name: Preflight — detect existing Spur daemons (informational) + ansible.builtin.shell: | + set +e + for p in spurctld spurd spurrestd; do + pid=$(pgrep -x "$p" 2>/dev/null | head -1) + [ -n "$pid" ] && echo "$p running (PID $pid)" + done + exit 0 + register: pf_running + changed_when: false + +- name: Preflight — note any running daemons + ansible.builtin.debug: + msg: "{{ pf_running.stdout_lines | default(['no spur daemons running']) }}" + +- name: Preflight — fail if Spur ports are held by a non-Spur process + ansible.builtin.shell: | + set +e + # ss -p prints "users:((\"\",pid=...))" as the last column when run as root. + # Any owner that is NOT spurctld/spurd/spurrestd on our ports is a conflict. + bad=$(ss -tlnpH "( sport = :{{ spur_controller_port }} or sport = :{{ spur_agent_port }} or sport = :{{ spur_raft_port }} )" 2>/dev/null \ + | grep -oE 'users:\(\("[^"]+"' \ + | sed -E 's|.*"([^"]+)".*|\1|' \ + | grep -v -E '^(spurctld|spurd|spurrestd)$') + if [ -n "$bad" ]; then + echo "non-spur process(es) on spur ports: $bad" + exit 1 + fi + exit 0 + args: + executable: /bin/bash + register: pf_ports + changed_when: false + failed_when: pf_ports.rc != 0 + +- name: Create Spur directory layout + ansible.builtin.file: + path: "{{ item }}" + state: directory + mode: "0755" + loop: + - "{{ spur_home }}" + - "{{ spur_home }}/state" + - "{{ spur_home }}/log" + - "{{ spur_home }}/etc" + - "{{ spur_install_dir }}" + +- name: Check for existing spur binary + ansible.builtin.stat: + path: "{{ spur_install_dir }}/spur" + register: spur_bin_stat + +# Two install sources: +# 1. spur_binary_src (a directory on the Ansible control node holding pre-built +# spur/spurctld/spurd/spurdbd) — used for air-gapped installs and when the +# upstream repo has no published GitHub release yet. Takes precedence. +# 2. upstream install.sh — downloads the latest published release. +- name: Copy pre-built Spur binaries from control node + ansible.builtin.copy: + src: "{{ spur_binary_src }}/{{ item }}" + dest: "{{ spur_install_dir }}/{{ item }}" + mode: "0755" + loop: + - spur + - spurctld + - spurd + - spurdbd + when: spur_binary_src | default('') | length > 0 + +- name: Install Spur via upstream installer + ansible.builtin.shell: | + set -euo pipefail + curl -fsSL https://raw.githubusercontent.com/ROCm/spur/main/install.sh \ + | INSTALL_DIR={{ spur_install_dir }} bash -s -- {{ spur_version }} + args: + executable: /bin/bash + when: + - spur_binary_src | default('') | length == 0 + - not spur_bin_stat.stat.exists + register: install_run + changed_when: install_run.rc == 0 + +- name: Verify spur binary is present + ansible.builtin.stat: + path: "{{ spur_install_dir }}/spur" + register: spur_ver + failed_when: not spur_ver.stat.exists + +# Slurm-compatible entrypoints. The single `spur` multi-call binary dispatches +# on argv[0], so these symlinks give users drop-in sbatch/squeue/etc. Created +# regardless of install source (pre-built or upstream installer). +- name: Symlink Slurm-compatible CLI names + ansible.builtin.file: + src: "{{ spur_install_dir }}/spur" + dest: "{{ spur_install_dir }}/{{ item }}" + state: link + force: true + loop: + - sbatch + - squeue + - sinfo + - scancel + - sacct + - scontrol + - salloc + - srun + +- name: Show install result + ansible.builtin.debug: + msg: "{{ inventory_hostname }}: spur binaries installed in {{ spur_install_dir }}" + +- name: Read remote /etc/environment + ansible.builtin.slurp: + src: /etc/environment + register: etc_environment + failed_when: false + +- name: Ensure ~/.local/bin is on PATH for non-interactive SSH sessions + ansible.builtin.lineinfile: + path: /etc/environment + backrefs: yes + regexp: '^PATH="(?!.*{{ spur_install_dir | regex_escape }})(.*)"$' + line: 'PATH="{{ spur_install_dir }}:\1"' + when: spur_install_dir not in (etc_environment.content | default('') | b64decode) diff --git a/deploy/ansible/roles/spur_verify/defaults/main.yml b/deploy/ansible/roles/spur_verify/defaults/main.yml new file mode 100644 index 00000000..ed97d539 --- /dev/null +++ b/deploy/ansible/roles/spur_verify/defaults/main.yml @@ -0,0 +1 @@ +--- diff --git a/deploy/ansible/roles/spur_verify/tasks/main.yml b/deploy/ansible/roles/spur_verify/tasks/main.yml new file mode 100644 index 00000000..057d54f7 --- /dev/null +++ b/deploy/ansible/roles/spur_verify/tasks/main.yml @@ -0,0 +1,146 @@ +--- +- name: Wait for agents to register with controller + # `spur nodes` collapses by partition; use `spur show node ` per host instead. + ansible.builtin.shell: | + set -e + for i in $(seq 1 30); do + missing=0 + for h in {{ groups['spur_agents'] | map('extract', hostvars, 'ansible_hostname') | join(' ') }}; do + {{ spur_install_dir }}/spur show node "$h" >/dev/null 2>&1 || missing=$((missing+1)) + done + [ "$missing" -eq 0 ] && exit 0 + sleep 1 + done + echo "timed out: $missing agents still missing" + exit 1 + args: + executable: /bin/bash + changed_when: false + +- name: Show spur nodes + ansible.builtin.command: "{{ spur_install_dir }}/spur nodes" + register: nodes_out + changed_when: false + +- name: Print spur nodes + ansible.builtin.debug: + var: nodes_out.stdout_lines + +- name: Drop test scripts under /tmp + ansible.builtin.copy: + dest: /tmp/spur-test-{{ item.name }}.sh + mode: "0755" + content: "{{ item.body }}" + loop: + - name: single + body: | + #!/bin/bash + #SBATCH --job-name=ansible-single + echo "ran on $(hostname) at $(date)" + - name: multi + body: | + #!/bin/bash + #SBATCH --job-name=ansible-multi + #SBATCH -N {{ groups['spur_agents'] | length }} + #SBATCH --ntasks-per-node=1 + echo "node $SPUR_TASK_OFFSET of $SPUR_NUM_NODES on $(hostname); peers=$SPUR_PEER_NODES" + +- name: Submit single-node test job + ansible.builtin.shell: | + cd {{ spur_home }} + {{ spur_install_dir }}/spur submit /tmp/spur-test-single.sh | grep -oE '[0-9]+' + args: + executable: /bin/bash + register: single_jobid + changed_when: true + +- name: Wait for single-node job {{ single_jobid.stdout }} to finish + ansible.builtin.shell: | + for i in $(seq 1 30); do + st=$({{ spur_install_dir }}/spur show job {{ single_jobid.stdout }} 2>/dev/null | grep -oE 'JobState=[A-Z]+' | head -1 | cut -d= -f2) + case "$st" in + COMPLETED|FAILED|CANCELLED|TIMEOUT|NODE_FAIL) echo "$st"; exit 0 ;; + esac + sleep 1 + done + echo "timeout" + exit 1 + args: + executable: /bin/bash + register: single_state + changed_when: false + failed_when: single_state.stdout != "COMPLETED" + +- name: Find single-node job output (looked up on every agent since assignment is dynamic) + ansible.builtin.shell: | + for d in {{ spur_home }} $HOME /root /tmp; do + f="$d/spur-{{ single_jobid.stdout }}.out" + [ -f "$f" ] && { cat "$f"; exit 0; } + done + echo "" + args: + executable: /bin/bash + delegate_to: "{{ item }}" + loop: "{{ groups['spur_agents'] }}" + register: single_outs + changed_when: false + +- name: Set single_out to first non-empty result + ansible.builtin.set_fact: + single_out_text: "{{ (single_outs.results | selectattr('stdout', 'ne', '') | list | first).stdout | default('(not found on any agent)') }}" + +- name: Show single-node output + ansible.builtin.debug: + msg: "Job {{ single_jobid.stdout }} ({{ single_state.stdout }}): {{ single_out_text }}" + +- name: Submit multi-node test job (only when more than one agent) + ansible.builtin.shell: | + cd {{ spur_home }} + {{ spur_install_dir }}/spur submit /tmp/spur-test-multi.sh | grep -oE '[0-9]+' + args: + executable: /bin/bash + register: multi_jobid + when: groups['spur_agents'] | length > 1 + changed_when: true + +- name: Wait for multi-node job to finish + ansible.builtin.shell: | + for i in $(seq 1 60); do + st=$({{ spur_install_dir }}/spur show job {{ multi_jobid.stdout }} 2>/dev/null | grep -oE 'JobState=[A-Z]+' | head -1 | cut -d= -f2) + case "$st" in + COMPLETED|FAILED|CANCELLED|TIMEOUT|NODE_FAIL) echo "$st"; exit 0 ;; + esac + sleep 1 + done + echo "timeout" + exit 1 + args: + executable: /bin/bash + register: multi_state + when: groups['spur_agents'] | length > 1 + changed_when: false + failed_when: multi_state.stdout != "COMPLETED" + +- name: Fetch multi-node outputs from every agent (no shared-FS assumption) + ansible.builtin.shell: | + for d in {{ spur_home }} $HOME /root /tmp; do + f="$d/spur-{{ multi_jobid.stdout }}.out" + [ -f "$f" ] && { cat "$f"; exit 0; } + done + echo "(no output on this host)" + args: + executable: /bin/bash + delegate_to: "{{ item }}" + loop: "{{ groups['spur_agents'] }}" + register: multi_outs + when: groups['spur_agents'] | length > 1 + changed_when: false + failed_when: false + +- name: Show multi-node outputs + ansible.builtin.debug: + msg: "{{ item.item }}: {{ item.stdout | default('(no output)') }}" + loop: "{{ multi_outs.results | default([]) }}" + when: groups['spur_agents'] | length > 1 + loop_control: + label: "{{ item.item }}" diff --git a/deploy/ansible/roles/spur_wireguard/defaults/main.yml b/deploy/ansible/roles/spur_wireguard/defaults/main.yml new file mode 100644 index 00000000..81245e62 --- /dev/null +++ b/deploy/ansible/roles/spur_wireguard/defaults/main.yml @@ -0,0 +1,6 @@ +--- +# WireGuard transport is single-controller only: `spur net init` auto-assigns +# the controller .1, and there is no multi-controller mesh command. The role +# assigns each agent a unique address (.2, .3, ...) by inventory position, or +# honors a per-host spur_wg_address set in inventory. Requires the ansible.utils +# collection on the control node (ansible-galaxy collection install -r requirements.yml). diff --git a/deploy/ansible/roles/spur_wireguard/tasks/main.yml b/deploy/ansible/roles/spur_wireguard/tasks/main.yml new file mode 100644 index 00000000..65c1302b --- /dev/null +++ b/deploy/ansible/roles/spur_wireguard/tasks/main.yml @@ -0,0 +1,158 @@ +--- +# WireGuard mesh setup. The `spur net` CLI supports a single controller (which +# auto-assigns itself .1) plus N agents that join and are registered as peers. +# Multi-controller (HA) WireGuard is not supported by the CLI, so this role +# asserts a single controller. +- name: Assert ansible.utils collection is available (provides ipaddr/ipmath) + ansible.builtin.assert: + that: + - "'10.0.0.0/24' | ansible.utils.ipaddr('network') == '10.0.0.0'" + fail_msg: >- + The ansible.utils collection is required for WireGuard transport. + Install it on the control node: + ansible-galaxy collection install -r requirements.yml + run_once: true + delegate_to: localhost + +- name: Assert a single controller (CLI WireGuard mesh is single-controller only) + ansible.builtin.assert: + that: + - groups['spur_controllers'] | length == 1 + fail_msg: >- + WireGuard transport supports exactly one controller (spur net init + auto-assigns the controller .1 and there is no multi-controller mesh + command). Use spur_transport=direct for HA, or a single controller. + run_once: true + +- name: Assert wireguard module is loadable + ansible.builtin.shell: "modinfo wireguard >/dev/null 2>&1" + changed_when: false + +- name: Install wireguard-tools (Debian/Ubuntu) + ansible.builtin.apt: + name: wireguard-tools + state: present + update_cache: yes + when: ansible_os_family == 'Debian' + +- name: Install wireguard-tools (RHEL family) + ansible.builtin.package: + name: wireguard-tools + state: present + when: ansible_os_family == 'RedHat' + +# The controller always gets .1 (assigned by `spur net init`). Agents get a +# unique address from their position in the mesh: .2, .3, ... A per-host +# spur_wg_address set in inventory always wins. +- name: Build ordered agent list (non-controller agents) + ansible.builtin.set_fact: + spur_wg_agents: "{{ groups['spur_agents'] | difference(groups['spur_controllers']) }}" + +- name: Set controller WG address to the mesh .1 + ansible.builtin.set_fact: + spur_wg_address: "{{ spur_wg_cidr | ansible.utils.ipaddr('network') | ansible.utils.ipmath(1) }}" + when: + - inventory_hostname in groups['spur_controllers'] + - spur_wg_address is not defined + +- name: Assign each agent a unique WG address (.2, .3, ... by position) + ansible.builtin.set_fact: + spur_wg_address: "{{ spur_wg_cidr | ansible.utils.ipaddr('network') | ansible.utils.ipmath(spur_wg_agents.index(inventory_hostname) + 2) }}" + when: + - inventory_hostname in spur_wg_agents + - spur_wg_address is not defined + +- name: Initialize WireGuard mesh on the controller + # `spur net init` auto-assigns .1 from --cidr and logs to stderr. It is not + # idempotent, so only run it when the interface is not already up. + ansible.builtin.shell: | + set -euo pipefail + if ip link show {{ spur_wg_interface }} >/dev/null 2>&1; then + echo "SPUR_WG_ALREADY_UP" + else + {{ spur_install_dir }}/spur net init \ + --cidr {{ spur_wg_cidr }} \ + --port {{ spur_wg_port }} \ + --interface {{ spur_wg_interface }} 1>&2 + fi + args: + executable: /bin/bash + register: wg_ctl_init + changed_when: "'SPUR_WG_ALREADY_UP' not in wg_ctl_init.stdout" + when: inventory_hostname in groups['spur_controllers'] + +- name: Read controller WG public key + ansible.builtin.command: "wg show {{ spur_wg_interface }} public-key" + register: wg_ctl_pubkey + changed_when: false + when: inventory_hostname in groups['spur_controllers'] + +- name: Broadcast controller pubkey + endpoint to all hosts + ansible.builtin.set_fact: + spur_wg_controller_pubkey: "{{ hostvars[groups['spur_controllers'][0]]['wg_ctl_pubkey'].stdout | trim }}" + spur_wg_controller_endpoint: "{{ hostvars[groups['spur_controllers'][0]]['ansible_host'] | default(groups['spur_controllers'][0]) }}:{{ spur_wg_port }}" + +- name: Assert each agent has a WG address before join + ansible.builtin.assert: + that: + - spur_wg_address is defined + fail_msg: "Agent {{ inventory_hostname }} has no spur_wg_address; set it in inventory or check spur_wg_cidr." + when: inventory_hostname in spur_wg_agents + +- name: Join WireGuard mesh on agents + # `spur net join` needs --prefix-len to match spur_wg_cidr (defaults to 16), + # else the interface address/AllowedIPs are wrong. Logs to stderr. + ansible.builtin.shell: | + set -euo pipefail + if ip link show {{ spur_wg_interface }} >/dev/null 2>&1; then + echo "SPUR_WG_ALREADY_UP" + else + {{ spur_install_dir }}/spur net join \ + --endpoint {{ spur_wg_controller_endpoint }} \ + --server-key {{ spur_wg_controller_pubkey }} \ + --address {{ spur_wg_address }} \ + --prefix-len {{ spur_wg_cidr | ansible.utils.ipaddr('prefix') }} \ + --interface {{ spur_wg_interface }} 1>&2 + fi + args: + executable: /bin/bash + register: wg_agent_join + changed_when: "'SPUR_WG_ALREADY_UP' not in wg_agent_join.stdout" + when: inventory_hostname in spur_wg_agents + +- name: Read agent WG public key + ansible.builtin.command: "wg show {{ spur_wg_interface }} public-key" + register: wg_agent_pubkey + changed_when: false + when: inventory_hostname in spur_wg_agents + +- name: Capture agent pubkey + ansible.builtin.set_fact: + spur_wg_pubkey: "{{ wg_agent_pubkey.stdout | trim }}" + when: inventory_hostname in spur_wg_agents + +- name: Register all agents as peers on the controller + ansible.builtin.command: >- + {{ spur_install_dir }}/spur net add-peer + --key {{ hostvars[item]['spur_wg_pubkey'] }} + --allowed-ip {{ hostvars[item]['spur_wg_address'] }}/32 + --interface {{ spur_wg_interface }} + loop: "{{ spur_wg_agents }}" + when: inventory_hostname == groups['spur_controllers'][0] + changed_when: true + +- name: Verify mesh — every host pings controller WG IP + ansible.builtin.command: "ping -c2 -W2 {{ hostvars[groups['spur_controllers'][0]]['spur_wg_address'] }}" + register: wg_ping + changed_when: false + failed_when: false + +- name: Warn if the WG mesh ping did not succeed + ansible.builtin.debug: + msg: >- + WARNING: could not ping the controller over WireGuard + ({{ hostvars[groups['spur_controllers'][0]]['spur_wg_address'] }}). + The tunnel is configured (interface + peers are up) but traffic did not + pass — expected when all hosts already share an L2/L3 network, or a real + routing/firewall issue otherwise. Verify with `wg show {{ spur_wg_interface }}`. + when: wg_ping.rc | default(1) != 0 diff --git a/deploy/ansible/teardown.yml b/deploy/ansible/teardown.yml new file mode 100644 index 00000000..9e94fd78 --- /dev/null +++ b/deploy/ansible/teardown.yml @@ -0,0 +1,55 @@ +--- +# Stop all Spur daemons. Leaves binaries and state in place. +# ansible-playbook teardown.yml -i inventory/hosts.ini +# +# Add -e wipe=true to also wipe ~/spur and /etc/wireguard/spur0.* (destructive). + +- name: Stop Spur daemons + # Include the accounting host too, in case it is a dedicated node not in either group. + hosts: "spur_controllers:spur_agents:{{ spur_accounting_host | default(groups['spur_controllers'][0]) }}" + gather_facts: no + tasks: + - name: Stop and disable Spur systemd services + ansible.builtin.systemd: + name: "{{ item }}" + state: stopped + enabled: false + loop: + - spurd + - spurctld + - spurdbd + failed_when: false + + - name: Reap any stray daemons started outside systemd (exact-name match) + ansible.builtin.shell: | + pkill -x spurd 2>/dev/null || true + pkill -x spurctld 2>/dev/null || true + pkill -x spurdbd 2>/dev/null || true + pkill -x spurrestd 2>/dev/null || true + sleep 1 + pgrep -x spurd || true + pgrep -x spurctld || true + pgrep -x spurdbd || true + pgrep -x spurrestd || true + args: + executable: /bin/bash + changed_when: false + + - name: Tear down WireGuard interface + ansible.builtin.shell: | + if ip link show {{ spur_wg_interface }} >/dev/null 2>&1; then + wg-quick down {{ spur_wg_interface }} || ip link del {{ spur_wg_interface }} + fi + args: + executable: /bin/bash + when: spur_transport == 'wireguard' + changed_when: false + + - name: Wipe state (only when -e wipe=true) + ansible.builtin.file: + path: "{{ item }}" + state: absent + loop: + - "{{ spur_home }}" + - "/etc/wireguard/{{ spur_wg_interface }}.conf" + when: wipe | default(false) | bool