feat(deploy): add Ansible playbook + standalone deploy-spur skill for bare-metal deployment

[yansun1996]

Closes #251

Summary

Two independent paths to deploy Spur on bare-metal, sharing the same gotcha knowledge:

1. deploy/ansible/ — self-contained Ansible playbook. Run ansible-playbook deploy.yml -i <inventory>. One inventory shape covers four modes:

   Mode	Inventory shape
   Single-node	one host in both [spur_controllers] and [spur_agents]
   Multi-node, single controller	1 host in [spur_controllers], N in [spur_agents]
   HA (multi-controller Raft)	≥ 2 hosts in [spur_controllers] — auto-enables Raft, assigns node_id from inventory position, writes peers to every controller's spur.conf
   WireGuard overlay	any of the above + spur_transport=wireguard and per-host spur_wg_address

   Roles: spur_install, spur_wireguard, spur_accounting, spur_controller, spur_agent, spur_verify.

2. .claude/skills/deploy-spur/SKILL.md — standalone operator skill. Drives the same deploy with only SSH + bash, no Ansible. Works on any workstation with SSH access to the targets. Covers single-node, multi-node, and HA from one flow. Includes install (curl install.sh), spur.conf rendering, daemon start with correct backgrounding, leader-election wait for HA, agent-registration wait, smoke tests, and teardown.

The two paths are decoupled — using the skill does not require the playbook tree to be present, and vice versa.

What's new in this revision

• systemd-managed daemons. spurctld, spurd, and spurdbd now run as systemd units (templated .service files, enabled: true, Restart=on-failure) instead of nohup. Survives reboots; systemctl status works as expected.
• Optional accounting stack (spur_accounting role). Installs PostgreSQL + wires spurdbd on the first controller and adds an [accounting] block to spur.conf. Gated by spur_accounting_enabled (default on); set false to skip entirely — job submission still works without it.
• Pre-built binary install source. spur_binary_src copies locally-built spur/spurctld/spurd/spurdbd from the control node (needed while the upstream repo has no published GitHub release — install.sh returns 403). Falls back to the upstream installer when unset.
• Slurm-compat symlinks on every install path. sbatch/squeue/sinfo/scancel/sacct/scontrol/salloc/srun → spur, created regardless of install source.

Test plan

Re-validated end-to-end on a 4-node Ubuntu 22.04 cluster (fresh install: PostgreSQL purged on all nodes beforehand). Ansible run from an operator workstation (not a control node on the cluster). Each scenario verified: basic commands (spur nodes/queue), a sample job driven to COMPLETED, all daemons active under systemd, and accounting rows in PostgreSQL via sacct.

Ansible playbook — 4 scenarios, all PASS:

• S1 single-node — controller+agent on one host; job COMPLETED, accounted
• S2 multi-node — 1 controller + 2 agents; -N 2 job fanned across both nodes, accounted
• S3 HA hyperconverged — 3 controllers (Raft) + 3 agents; leader elected, -N 3 job across all three, accounted
• S4 HA + separate compute — 3 dedicated controllers (no agent) + 1 dedicated agent; leader elected, controllers ran no spurd, job ran on the separate compute node, accounted
• Accounting disabled (-e spur_accounting_enabled=false) — deploy skips PostgreSQL/spurdbd; job submission still COMPLETED (sacct unavailable, as expected)
• systemd lifecycle — spurctld/spurd/spurdbd/postgresql all active; symlink task idempotent (ok, not changed) on re-run
• WireGuard transport — implemented (per-host unique WG address auto-assignment fixed for HA), but not validated end-to-end: the lab hosts share a /24, so a WG mesh would ride over already-reachable LAN and wouldn't exercise the cross-subnet case WG exists for. direct transport is validated across all 4 scenarios above; wireguard is code-complete and single-controller-sound, and needs validation on a representative multi-subnet topology. Full N-controller peer mesh is not yet automated (agents peer with the first controller).

Standalone skill (validated with three fresh subagents, one per mode, each reading SKILL.md directly to bypass any caching):

• single-node — PASS, 93s, smoke test COMPLETED
• multi-node (1 ctl + 2 ag) — PASS, 139s, single + -N 2 COMPLETED
• HA (2 ctl + 2 ag hyperconverged) — PASS, 141s, leader=node_id 2, single + -N 2 + follower-forwarding COMPLETED

Skill bugs surfaced during fresh-agent validation and patched in this PR:

• dead partition_nodes_csv first-attempt before the working fallback (Step 5)
• dead-end SSH heredoc in agent-registration loop before the working client-side version (Step 8)
• <<EOF vs <<'EOF' in multi-node smoke test — caused \$SPUR_TASK_OFFSET etc. to expand at submit-time instead of at job-run time on the agent (Step 9)

Notes worth knowing

• N=2 HA has zero fault tolerance (quorum = 2). Use odd N ≥ 3 in production. Both paths warn.
• spurd --controller takes a single URL in Spur 0.3.0 — agents don't fail over even if Raft does. Production HA needs an L4 VIP / DNS round-robin in front of :6817.
• HA peers list order must stay stable across deploys (openraft node_id is positional).
• Accounting runs a single spurdbd + PostgreSQL on the first controller for the whole cluster. Default DB creds are spur/spur — override spur_accounting_db_password for anything beyond a lab.
• With spur_wipe_state=true (default), the Raft job-id counter resets each deploy, so re-deploys reuse job ids 1, 2, … in the accounting DB (upsert on primary key). Set false to preserve state.

Gotchas baked into both paths

Real bugs hit during validation:

• -D means --foreground, not "daemonize". Never pass -D when backgrounding.
• pkill -f spurd also kills spurctld (substring match). Always pkill -x.
• SSH backgrounding needs < /dev/null + disown or the daemon dies on connection close (skill path; the playbook uses systemd instead).
• Job stdout file lands in spurd's CWD-at-startup, not the submitter's CWD as spur show job claims. The spurd systemd unit sets WorkingDirectory=$SPUR_HOME so output is predictable.
• HA needs a leader-elected wait, not just port-listening; spurctld binds :6817 immediately but returns Status::unavailable("no leader elected yet") until quorum forms.
• Raft port 6821 is hardcoded (not a CLI flag).
• spur show node <name> is a prefix match, not exact — collides on hostnames sharing a prefix.
• openraft info-level logs print a multi-line RaftState{...} per start that can clobber grep "become leader".

Files

• deploy/ansible/ — playbook, roles (incl. spur_accounting + systemd unit templates), inventory examples, README
• .claude/skills/deploy-spur/SKILL.md — standalone operator skill (no Ansible dependency)

[Copilot]

Pull request overview

Adds a self-contained bare-metal deployment workflow for Spur using Ansible, with inventory-driven topology (single-node, multi-node, HA Raft) and an accompanying operator “deploy-spur” skill to capture validated procedures and known gotchas.

Changes:

• Introduces deploy/ansible/deploy.yml plus roles to install Spur, configure controllers/agents, and verify the cluster by submitting test jobs.
• Adds optional WireGuard mesh setup and a teardown playbook for stopping/wiping deployments.
• Documents supported inventory shapes/variables and adds an operator skill under .claude/skills/.

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
deploy/ansible/deploy.yml	Main multi-play deployment entrypoint wiring install → (optional) wireguard → controller → agents → verify.
deploy/ansible/teardown.yml	Stops daemons and optionally wipes state on all hosts.
deploy/ansible/ansible.cfg	Local Ansible configuration for inventory defaults and SSH behavior.
deploy/ansible/group_vars/all.yml	Centralized default variables (ports, install dir, HA enablement, etc.).
deploy/ansible/README.md	Deployment modes, inventory examples, variables, teardown instructions, gotchas, and validation notes.
deploy/ansible/inventory/hosts.example.ini	Example inventory templates for common topologies/transports.
deploy/ansible/inventory/hosts.ha.example.ini	HA-focused inventory example and caveats for client failover.
deploy/ansible/roles/spur_install/tasks/main.yml	Preflight checks, directory creation, installer execution, and PATH adjustment.
deploy/ansible/roles/spur_install/defaults/main.yml	Role defaults placeholder.
deploy/ansible/roles/spur_wireguard/tasks/main.yml	WireGuard mesh init/join/peer registration/verification tasks.
deploy/ansible/roles/spur_wireguard/defaults/main.yml	WireGuard defaults documentation note.
deploy/ansible/roles/spur_controller/tasks/main.yml	Controller config generation, Raft state handling, daemon lifecycle, leader wait.
deploy/ansible/roles/spur_controller/templates/spur.conf.j2	Generates spur.conf including HA peers and node inventory.
deploy/ansible/roles/spur_controller/defaults/main.yml	Role defaults placeholder.
deploy/ansible/roles/spur_agent/tasks/main.yml	Agent addressing/controller targeting and daemon lifecycle.
deploy/ansible/roles/spur_agent/defaults/main.yml	Role defaults placeholder.
deploy/ansible/roles/spur_verify/tasks/main.yml	Cluster verification via registration wait + single/multi-node test job submission/output collection.
deploy/ansible/roles/spur_verify/defaults/main.yml	Role defaults placeholder.
.claude/skills/deploy-spur/SKILL.md	Operator skill documenting deployment decision flow, commands, and gotchas.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[shiv-tyagi]

Thanks for this work Yan.

I think the project is still in very early stages and things are changing very rapidly. This skill can keep drifting every now and then and start confusing AI. It might be better if we do this at some later stages.

Currently, I use AI to deploy spur, most of the times it is able to understand things on the fly. I have some user instructions, but those are uncommitted and local only.

We can follow the same model and commit when things stop changing shape very often.

[shiv-tyagi]

Same for ansible. We know it is important to have ansible scripts. Once things stabalize, we wont require changing ansible scripts much and the the project and ansible playbooks/roles versioning can (or perhaps should) be decoupled to avoid unnecessary maintenance problems in future.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #250      +/-   ##
==========================================
- Coverage   68.24%   68.22%   -0.02%     
==========================================
  Files         134      134              
  Lines       36098    36098              
==========================================
- Hits        24634    24625       -9     
- Misses      11464    11473       +9     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.