feat(reservations): persist CRUD via raft WAL and enforce scheduling parity

[sgopinath1]

Summary

Adds durable node reservations with scheduling enforcement, and preemption interaction.

• Durability — Reservation create/update/delete go through WalOperation (Raft log replay), not snapshot-only state.
• Flags — MAINT, IGNORE_JOBS, NO_HOLD_JOBS, and OVERLAP with validation (unknown flags rejected at API boundary).
• Scheduling — Prospective backfill blocking for reserved nodes, reservation-targeting jobs prioritized in pending_jobs(), future-start window fit via now.max(res.start_time), and lifecycle hooks (purge_expired, end-time cancel with resv_overrun_minutes grace, hold pending jobs on delete/purge unless NO_HOLD_JOBS).
• Preemption — Honors partition PreemptMode; skips jobs in active reservations unless the preemptor has a higher priority_tier; only considers running jobs on nodes the pending job could use.
• Display / API — scontrol/sinfo/smd reservation state; PendingReason::ReservedMaintenance / ReservationDeleted; proto JobInfo.reservation, reservation_maint, ReservationInfo.state; squeue %v column.

Design notes

• Spur always runs Raft; reservations now follow the same propose → apply path as jobs and nodes.
• hold_jobs_for_deleted_reservation sets PendingReason::ReservationDeleted after hold_job (not WAL-persisted; same pattern as other hold reasons).
• IGNORE_JOBS bypasses running-job overlap checks at create time; OVERLAP / MAINT allow storage overlap between reservations.

Test plan

• cargo fmt --check
• cargo clippy --workspace --exclude spur-ffi --all-targets --locked (RUSTFLAGS=-D warnings)
• cargo test --locked (unit tests across spur-core, spur-sched, spurctld, spur-cli, spurdbd)
• Unit: overlap validation, priority ordering, resv_overrun grace, preemption immunity, node-overlap gating, purge_expired hold behavior, WAL apply for reservation CRUD
• E2E: tests/native_host/e2e/test_reservations.py on bare-metal lab (create/list/delete, unauthorized block, authorized schedule, busy-node rejection)

[codecov-commenter]

Codecov Report

❌ Patch coverage is 78.80974% with 235 lines in your changes missing coverage. Please review.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #385      +/-   ##
==========================================
+ Coverage   68.22%   68.67%   +0.44%     
==========================================
  Files         134      134              
  Lines       36087    37173    +1086     
==========================================
+ Hits        24619    25525     +906     
- Misses      11468    11648     +180     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

Adds durable, Raft-persisted node reservations to Spur and enforces reservation semantics across scheduling, backfill, display surfaces, and preemption behavior.

Changes:

• Persist reservation CRUD via Raft WAL operations and apply-path replay.
• Enforce reservation eligibility in scheduling/backfill (including future overlap fencing) and introduce reservation-driven pending reasons / formatting.
• Integrate reservations into UX/API surfaces (proto fields, CLI display/format codes) and refine preemption logic around active reservations.

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
tests/native_host/e2e/test_reservations.py	Adds native-host E2E coverage for reservation create/list/delete and scheduling enforcement.
proto/slurm.proto	Extends public proto with reservation fields (job reservation, node maint flag, reservation flags/state).
crates/spurdbd/src/server.rs	Populates new JobInfo.reservation field for accounting responses.
crates/spurctld/src/server.rs	Validates/accepts reservation flags at API boundary; lists reservation flags/state; annotates nodes with reservation info.
crates/spurctld/src/scheduler_loop.rs	Runs reservation lifecycle hooks each scheduler tick; expands preemption logic to account for reservations and partition policy.
crates/spurctld/src/cluster.rs	Moves reservation CRUD onto propose→apply Raft path; adds validation, purge/expiry handling, and reservation-driven pending ordering/reasons.
crates/spur-tests/src/t55_format.rs	Adds formatting parity mappings for new pending reasons.
crates/spur-tests/src/t50_core.rs	Updates core tests for new reservation flags field.
crates/spur-tests/src/t07_sched.rs	Updates scheduler tests for new reservation flags field.
crates/spur-sched/src/backfill.rs	Adds prospective reservation overlap checks and reservation window fit constraints to backfill scheduling.
crates/spur-core/src/wal.rs	Introduces WAL variants for reservation create/update/delete plus serialization round-trip tests.
crates/spur-core/src/reservation.rs	Adds reservation flags, overlap helpers, node-list normalization, and prospective overlap logic.
crates/spur-core/src/job.rs	Introduces new PendingReason values for reservation maintenance/deletion.
crates/spur-core/src/config.rs	Adds resv_overrun_minutes scheduler config for reservation end grace period.
crates/spur-cli/src/squeue.rs	Adds %v job format spec for reservation name display.
crates/spur-cli/src/smd.rs	Displays maint vs resv for nodes with active maintenance reservations.
crates/spur-cli/src/sinfo.rs	Displays maint vs resv for idle nodes with active maintenance reservations.
crates/spur-cli/src/scontrol.rs	Adds --flags to reservation creation and shows reservation/node reservation-related fields.
crates/spur-cli/src/format_engine.rs	Adds header label for %v reservation column.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.