Skip to content

parity(spur-metrics): node reservation/maintenance flags and NORESP have no representation in NodeState #395

Description

@yansun1996

Summary

Follow-up to #393. Slurm's native node metrics report slurm_nodes_resv (under an active reservation), slurm_nodes_maint (reservation-flagged maintenance), and slurm_nodes_noresp (missed a heartbeat but not yet marked down) as independent flags — a node can be IDLE and RESV and NORESP simultaneously. Spur's NodeState (crates/spur-core/src/node.rs) is a single flat enum: a node is in exactly one of Idle/Allocated/Mixed/Down/Drain/Draining/Error/Unknown/Suspended at a time, driven by a strict transition table (NodeState::transition). There's no independent flag bit for any of these three concepts.

What happened

Investigated closing the metrics gap for these three and found they split into two very different scopes:

  • resv/maint are cheap — the underlying data already exists from the reservations work (feat(reservations): persist CRUD via raft WAL and enforce scheduling parity #385-era code): Reservation::covers_node(&node_name) and Reservation::is_active(now) in crates/spur-core/src/reservation.rs, plus Reservation.flags.maint, already used together in reservation_fence_reason() (crates/spurctld/src/cluster.rs). These can be computed at metrics-collection time by joining cluster.get_nodes() against cluster.get_reservations(), the same pattern used for the /metrics/partitions join added in feat(spur-metrics): close OpenMetrics gaps vs Slurm 25.11 native metrics #394. No Node schema change needed.
  • noresp is architectural. evaluate_node_health() (crates/spurctld/src/cluster.rs) uses a single threshold (timeout_secs): once a node's last_heartbeat is stale past that threshold, NodeEvent::HeartbeatTimeout fires and the node transitions directly to NodeState::Down (with reason string "Not responding" baked in as text, not exposed as a queryable flag). Slurm's NORESP is a two-stage signal: a node can be flagged not-responding while still showing its last-known base state (e.g. IDLE+NORESP), only escalating to full DOWN after a longer separate timeout. Replicating this needs a second, shorter warning threshold in config, an independent noresp-style field on Node, and loosening NodeState::transition's single-value model to tolerate a state-plus-flag representation.

Expected behavior

/metrics/nodes and sinfo-equivalent output can represent a node as under-reservation, maintenance-flagged, and/or not-responding independently of its base operational state, matching what Slurm's native metrics (and scontrol show node) expose.

How to reproduce

Compare curl http://<spur-controller>:6822/metrics/nodes against a Slurm 25.11+ cluster's curl http://<slurmctld>:6817/metrics/nodes with an active reservation and/or a node that has just missed a heartbeat — Spur has no equivalent gauges for resv/maint/noresp.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: metricsPrometheus/OpenMetrics endpoint

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions