Skip to content

fix(graceful-upgrade): fix uninterruptible grace period and stale FD …#6

Merged
gabioprisan merged 1 commit intogcorefrom
fix-graceful-upgrades
Apr 6, 2026
Merged

fix(graceful-upgrade): fix uninterruptible grace period and stale FD …#6
gabioprisan merged 1 commit intogcorefrom
fix-graceful-upgrades

Conversation

@gabioprisan
Copy link
Copy Markdown
Collaborator

…accumulation

Issue 1 — Uninterruptible grace period after SIGTERM/SIGQUIT: After main_loop consumed signal handlers via tokio::select!, a bare thread::sleep(grace_period) left the process unresponsive to further signals for up to 20 minutes. Replace with a 1-second polling loop that checks an AtomicBool set by a signal-watching task spawned on the still-running server_runtime. A second SIGTERM/SIGINT/SIGQUIT during the grace period now causes an early exit.

Issue 2 — Stale inherited FDs accumulate across upgrade generations: serialize() dumped the entire Fds map, including ports removed from the current config. These zombie FDs were forwarded to every subsequent process generation, holding ports open indefinitely.

Fix with a two-map design in Fds: inherited FDs from deserialize() live exclusively in inherited_pending until a listener claims them via add() or mark_used(). close_unused_inherited() drains inherited_pending, closing stale FDs before send_to_sock() on SIGQUIT. Drop provides a safety net.

Also fix the non-NoSteal listen() path in l4.rs which never called mark_used(), causing active listener sockets to be closed by close_unused_inherited() on third-generation upgrades.

Add integration tests:

  • server_phase_gracefulupgrade: SIGQUIT → full phase progression
  • server_phase_early_exit: second SIGTERM interrupts grace period

…accumulation

Issue 1 — Uninterruptible grace period after SIGTERM/SIGQUIT:
After main_loop consumed signal handlers via tokio::select!, a bare
thread::sleep(grace_period) left the process unresponsive to further
signals for up to 20 minutes. Replace with a 1-second polling loop
that checks an AtomicBool set by a signal-watching task spawned on the
still-running server_runtime. A second SIGTERM/SIGINT/SIGQUIT during
the grace period now causes an early exit.

Issue 2 — Stale inherited FDs accumulate across upgrade generations:
serialize() dumped the entire Fds map, including ports removed from
the current config. These zombie FDs were forwarded to every subsequent
process generation, holding ports open indefinitely.

Fix with a two-map design in Fds: inherited FDs from deserialize() live
exclusively in inherited_pending until a listener claims them via add()
or mark_used(). close_unused_inherited() drains inherited_pending,
closing stale FDs before send_to_sock() on SIGQUIT. Drop provides a
safety net.

Also fix the non-NoSteal listen() path in l4.rs which never called
mark_used(), causing active listener sockets to be closed by
close_unused_inherited() on third-generation upgrades.

Add integration tests:
- server_phase_gracefulupgrade: SIGQUIT → full phase progression
- server_phase_early_exit: second SIGTERM interrupts grace period
@gabioprisan gabioprisan merged commit 3d2d6a2 into gcore Apr 6, 2026
3 of 4 checks passed
@gabioprisan gabioprisan deleted the fix-graceful-upgrades branch April 6, 2026 07:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants