You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Follow-up to #393. Slurm's native node metrics report slurm_nodes_resv (under an active reservation), slurm_nodes_maint (reservation-flagged maintenance), and slurm_nodes_noresp (missed a heartbeat but not yet marked down) as independent flags — a node can be IDLE and RESV and NORESP simultaneously. Spur's NodeState (crates/spur-core/src/node.rs) is a single flat enum: a node is in exactly one of Idle/Allocated/Mixed/Down/Drain/Draining/Error/Unknown/Suspended at a time, driven by a strict transition table (NodeState::transition). There's no independent flag bit for any of these three concepts.
What happened
Investigated closing the metrics gap for these three and found they split into two very different scopes:
resv/maint are cheap — the underlying data already exists from the reservations work (feat(reservations): persist CRUD via raft WAL and enforce scheduling parity #385-era code): Reservation::covers_node(&node_name) and Reservation::is_active(now) in crates/spur-core/src/reservation.rs, plus Reservation.flags.maint, already used together in reservation_fence_reason() (crates/spurctld/src/cluster.rs). These can be computed at metrics-collection time by joining cluster.get_nodes() against cluster.get_reservations(), the same pattern used for the /metrics/partitions join added in feat(spur-metrics): close OpenMetrics gaps vs Slurm 25.11 native metrics #394. No Node schema change needed.
noresp is architectural.evaluate_node_health() (crates/spurctld/src/cluster.rs) uses a single threshold (timeout_secs): once a node's last_heartbeat is stale past that threshold, NodeEvent::HeartbeatTimeout fires and the node transitions directly to NodeState::Down (with reason string "Not responding" baked in as text, not exposed as a queryable flag). Slurm's NORESP is a two-stage signal: a node can be flagged not-responding while still showing its last-known base state (e.g. IDLE+NORESP), only escalating to full DOWN after a longer separate timeout. Replicating this needs a second, shorter warning threshold in config, an independent noresp-style field on Node, and loosening NodeState::transition's single-value model to tolerate a state-plus-flag representation.
Expected behavior
/metrics/nodes and sinfo-equivalent output can represent a node as under-reservation, maintenance-flagged, and/or not-responding independently of its base operational state, matching what Slurm's native metrics (and scontrol show node) expose.
How to reproduce
Compare curl http://<spur-controller>:6822/metrics/nodes against a Slurm 25.11+ cluster's curl http://<slurmctld>:6817/metrics/nodes with an active reservation and/or a node that has just missed a heartbeat — Spur has no equivalent gauges for resv/maint/noresp.
Summary
Follow-up to #393. Slurm's native node metrics report
slurm_nodes_resv(under an active reservation),slurm_nodes_maint(reservation-flagged maintenance), andslurm_nodes_noresp(missed a heartbeat but not yet marked down) as independent flags — a node can beIDLEandRESVandNORESPsimultaneously. Spur'sNodeState(crates/spur-core/src/node.rs) is a single flat enum: a node is in exactly one ofIdle/Allocated/Mixed/Down/Drain/Draining/Error/Unknown/Suspendedat a time, driven by a strict transition table (NodeState::transition). There's no independent flag bit for any of these three concepts.What happened
Investigated closing the metrics gap for these three and found they split into two very different scopes:
resv/maintare cheap — the underlying data already exists from the reservations work (feat(reservations): persist CRUD via raft WAL and enforce scheduling parity #385-era code):Reservation::covers_node(&node_name)andReservation::is_active(now)incrates/spur-core/src/reservation.rs, plusReservation.flags.maint, already used together inreservation_fence_reason()(crates/spurctld/src/cluster.rs). These can be computed at metrics-collection time by joiningcluster.get_nodes()againstcluster.get_reservations(), the same pattern used for the/metrics/partitionsjoin added in feat(spur-metrics): close OpenMetrics gaps vs Slurm 25.11 native metrics #394. NoNodeschema change needed.norespis architectural.evaluate_node_health()(crates/spurctld/src/cluster.rs) uses a single threshold (timeout_secs): once a node'slast_heartbeatis stale past that threshold,NodeEvent::HeartbeatTimeoutfires and the node transitions directly toNodeState::Down(with reason string"Not responding"baked in as text, not exposed as a queryable flag). Slurm's NORESP is a two-stage signal: a node can be flagged not-responding while still showing its last-known base state (e.g.IDLE+NORESP), only escalating to fullDOWNafter a longer separate timeout. Replicating this needs a second, shorter warning threshold in config, an independentnoresp-style field onNode, and looseningNodeState::transition's single-value model to tolerate a state-plus-flag representation.Expected behavior
/metrics/nodesandsinfo-equivalent output can represent a node as under-reservation, maintenance-flagged, and/or not-responding independently of its base operational state, matching what Slurm's native metrics (andscontrol show node) expose.How to reproduce
Compare
curl http://<spur-controller>:6822/metrics/nodesagainst a Slurm 25.11+ cluster'scurl http://<slurmctld>:6817/metrics/nodeswith an active reservation and/or a node that has just missed a heartbeat — Spur has no equivalent gauges forresv/maint/noresp.