Orchestrator: Many orphaned Firecracker instances and leftover network/iptables rules causing host resource leakage

[AdaAibaby]

We observed many Firecracker processes, API sockets, metrics FIFOs and per-slot iptables PREROUTING REDIRECT rules left on the host (≈44 firecracker processes detected). Some instances are adopted by PID 1 (orphans), while others are still managed by the orchestrator (PPID == orchestrator PID). There are no clear D-state (uninterruptible I/O) processes at the moment, but many leftover resources are not cleaned up. This inflates host file descriptors, network interfaces and NAT rules, reducing the ability to start new sandboxes and increasing operational burden.

[linear-code]

ENG-4129

[AdaAibaby]

I propose splitting the fix into short-term (quick PR) and long-term stages to reduce risk while quickly mitigating host resource leakage.

Short-term (I will prepare a PR):

Add a startup / periodic reconcile sweep in the orchestrator: scan /data0/tmp/fc-*.sock, uffd, metrics FIFOs and iptables, detect resources inconsistent with the internal management table, then report or clean them (new entrypoint/module for safe rollout and monitoring).
Add a “forced reclamation” branch in the cleanup path (suggested locations: network and cleanup.go): before deleting a netns, detect and orderly kill remaining processes inside it (SIGTERM → SIGKILL), and make veth/iptables deletions idempotent with robust logging.
Add cancellable contexts and timeouts for NBD/provider ReadAt/WriteAt (change dispatch.go and provider implementations) to prevent backend I/O blocking that can stop Firecracker from exiting.
Long-term (process & operational):

Regular read-only diagnostics on hosts and iptables backups (example backup: /tmp/iptables-nat-backup-*.rules) with reporting to ops.
Identify orphan candidates: PPID == 1 and the api-sock is not opened by orchestrator — mark candidates only (no automatic deletion).
Gracefully reclaim confirmed orphans one-by-one: kill -TERM wait 10–20s, then kill -KILL if needed; afterwards remove api-sock, metrics FIFO, and iptables rules (always backup iptables first).
For instances still managed by orchestrator, prefer reconciliation/recovery via orchestrator rather than direct kills.
Deliverables I can provide:

A short-lived PR implementing the short-term items (reconcile sweep, forced reclamation, cancellable NBD I/O), with code, testing notes and rollback instructions.
An interactive candidate-cleanup script (read-only listing + annotated candidates + interactive cleanup plan) for ops to review and run in controlled mode.

/cc @jakubno

[AdaAibaby]

PR Proposal: Regular job to remove orphaned Firecracker processes on orchestrator nodes.
Scans hosts and removes fc processes reparented to PID 1.
Seeking maintainers' feedback on this approach.
/cc @jakubno