bug(spurctld): controller cannot restart after log compaction — last_purged not persisted → openraft "LogIndex(0) violates: try to get log at index 0 but got None"

[yansun1996]

Summary

A spurctld controller whose Raft log has been snapshot-compacted (purged) cannot restart. On startup it panics/exits with:

Error: 'LogIndex(0)' violates: 'try to get log at index 0 but got None'

and systemd crash-loops it. Because compaction happens automatically as the cluster runs, every controller becomes un-restartable over time. In an HA deployment this means a routine systemctl restart (or a host reboot) permanently bricks that controller's Raft state. The surviving controllers only stay up because they have never re-read state from disk since the purge.

This is the post-compaction counterpart to #273 (which is the panic when the log grows without compaction).

Impact (critical)

• HA controllers do not survive a restart → HA does not actually provide fault tolerance against the most basic operation (restarting/rebooting a controller).
• Observed on a live 3-controller cluster: restarting one controller to pick up a spur.conf change bricked it; the other two are now fragile (a restart/reboot of either would hit the same panic and break quorum).
• No data loss in the state machine (snapshot is intact), but the node is unrecoverable without a code fix.

Root cause (confirmed in source)

crates/spurctld/src/raft.rs, SpurStore:

• purge_logs_upto() (~line 320) sets inner.last_purged = Some(log_id) in memory only and deletes the on-disk log files. last_purged is never persisted to disk.
• SpurStore::new() (~line 94) reloads vote.json, the log entry files, and snapshot.json on startup — but does not restore last_purged (there is nothing on disk to restore it from). It defaults to None.
• get_log_state() (~line 264) then returns last_purged_log_id: None while last_log_id is the first surviving entry (e.g. index 10000, snapshot at 9999).
• openraft, seeing last_purged = None, walks the log back toward index 0, finds nothing there, and aborts with 'LogIndex(0)' violates: 'try to get log at index 0 but got None'.

Startup log from the affected node (snapshot at 9999, logs 10000–10246):

INFO spurctld::cluster: restored cluster state from Raft snapshot jobs=91 nodes=15
INFO spurctld::raft: raft store recovered from disk log_entries=1247 vote=Some(Vote { term: 1, node_id: 3, committed: true })
INFO openraft::storage::helper: get_initial_state vote=T1-N3:committed last_purged_log_id=None last_applied=T1-N3-9999 committed=None last_log_id=T1-N3-10246
INFO openraft::storage::helper: load membership from log: [10000..10247)
INFO openraft::storage::helper: load key log ids from (None,T1-N3-10246]
Error: 'LogIndex(0)' violates: 'try to get log at index 0 but got None'
spurctld.service: Main process exited, code=exited, status=1/FAILURE

Reproduce

1. Run a controller long enough for openraft to take a snapshot and purge the log (or force a snapshot).
2. systemctl restart spurctld (or reboot the host).
3. spurctld exits 1 on every start with the LogIndex(0) error; systemd crash-loops it.

Recovery paths that do NOT work (all reproduce the bug)

• Plain restart — hits the panic (log already purged).
• Wipe state-dir and let it rejoin — different but also fatal: with empty state the symmetric raft.initialize(members) in start_raft() (raft.rs ~line 663) actually bootstraps a new cluster and races the leader's InstallSnapshot, panicking in openraft raft_core.rs:779 with called Option::unwrap() on a None value. So a wiped node cannot cleanly rejoin an existing cluster either.
• Seed the node with the leader's state-dir (snapshot + logs) — reproduces the identical LogIndex(0) panic, because the copied on-disk state also has no persisted last_purged. This confirms the defect is in the recovery code, not the data.

Suggested fix direction (not prescriptive)

Persist last_purged durably (e.g. a purged.json / log_state.json written in purge_logs_upto, or derive it from snapshot meta) and restore it in SpurStore::new() so get_log_state() reports the correct last_purged_log_id after a compaction+restart. Separately, the empty-state rejoin path (unconditional initialize() in start_raft) should not bootstrap when the node is meant to join an existing cluster — that is a second, related crash.

Environment

• spurctld version 0.3.0, openraft 0.9.24
• 3-node HA (node_ids 1/2/3), systemd-managed, persistent --state-dir