scontrol: no runtime query or management of node feature labels (requires spur.conf edit + restart)

[yansun1996]

Problem

There is no runtime, admin-facing CLI to query or manage node feature labels. Today the only way to change a node's features is to hand-edit spur.conf on every controller and restart spurctld (which also forces all agents to re-register). For a multi-controller HA deployment this is disruptive and error-prone for what should be a lightweight operation.

Current state

Query side — cannot see features at all:

• scontrol show node / sinfo do not surface a node's configured features (the NodeInfo proto has no features field; the %f formatter prints ?). See related issue sinfo/spur nodes: node features (%f) not displayable — missing from NodeInfo proto and sinfo resolver #355.

Management side — scontrol update node only supports state/reason:
From crates/spur-cli/src/scontrol.rs (parse_and_update), the only recognized node update keys are:

nodename / node   -> node_name
state             -> node_state
reason            -> node_reason

There is no features= (or availablefeatures= / activefeatures=) key. Any attempt prints scontrol: unknown update key.

So changing a label means: edit spur.conf on controller-01/02/03 → systemctl restart spurctld on all three → wait for Raft leader → restart all spurd agents to re-register. That's a full control-plane bounce just to add a tag.

What's wanted (admin runtime CLI)

1. Query a node's feature labels at runtime, e.g.:

   scontrol show node <name>          # include AvailableFeatures / ActiveFeatures
   sinfo -N -o "%n %f"                # show features column (ties to #355)
   

2. Manage feature labels at runtime without a restart, Slurm-style, e.g.:

   scontrol update NodeName=<name> AvailableFeatures=mi350x,atl,rackA
   scontrol update NodeName=<name> ActiveFeatures=mi350x
   

   The controller should persist and apply the change to the live scheduler state so subsequent --constraint matching reflects it immediately.

Why it matters

Operators routinely re-tag nodes (maintenance pools, hardware tiers, per-rack/per-host targeting). Requiring an HA-wide controller+agent restart for a label change is a significant operational gap versus Slurm, where scontrol update NodeName=... AvailableFeatures=... is a live, non-disruptive operation.

Environment

• Spur 0.3.0
• 3-controller HA + 12 compute nodes
• Related: sinfo/spur nodes: node features (%f) not displayable — missing from NodeInfo proto and sinfo resolver #355 (features not displayable in sinfo/spur nodes)

[shiv-tyagi]

Refer to comment #355 (comment)

[shiv-tyagi]

Related to #289