Skip to content

output.harbor: refactor listener management, add dedicated encoder mode#5003

Merged
toots merged 1 commit intomainfrom
harbor-output-changes
Apr 28, 2026
Merged

output.harbor: refactor listener management, add dedicated encoder mode#5003
toots merged 1 commit intomainfrom
harbor-output-changes

Conversation

@toots
Copy link
Copy Markdown
Member

@toots toots commented Mar 15, 2026

Reason

output.harbor was overdue for a cleanup. The old implementation used a Duppy monad-based async state machine with condition variables to manage connected clients, which was more complex than necessary and harder to reason about. This PR also addresses several scalability issues.

Changes

Architecture

Replaces the client state machine with a simpler synchronous listener record. Writes are moved off the streaming thread entirely into a single Duppy task that waits on listener socket write events (\Write fd`), fires only when at least one socket is ready, writes to all listeners non-blocking, and re-registers itself. This decouples the streaming cycle from network write latency.

New dedicated_encoder mode

A dedicated_encoder boolean parameter (default false) selects between two output classes:

  • shared_output (default): one encoder shared across all listeners, each seeded with the codec header and burst data on connect
  • dedicated_output: a fresh encoder created per listener at connect time, ensuring a clean stream from the first byte — particularly useful for %ffmpeg in copy mode

Scalability fixes

  • Write loop off streaming thread: writes now happen in a Duppy task, so slow listeners never stall the streaming cycle
  • No buffer copy in write path: per-listener buffers use Strings.Mutable (thread-safe); writes iterate chunks directly via Bytes.unsafe_of_string with no allocation
  • ICY metadata bug fix: metadata was previously consumed by the first listener in the list per frame, leaving all others stale; it is now captured once and shared across all listeners

Breaking API changes

  • user and password parameters removed; authentication is now exclusively via the auth function, which now receives a record {address, login, password} instead of individual arguments
  • burst parameter is now nullable (null disables burst)

Other improvements

  • ICY metadata constants renamed (max_titlemax_icy_title, etc.) and formatting extracted into standalone functions
  • Better variable naming throughout (buflenbuffer_limit, metaintmetadata_interval, etc.)
  • Shared logic moved into a virtual base class

Documentation

  • doc/content/harbor.md: new output.harbor section documenting dedicated_encoder, auth, and listener callback signatures
  • doc/content/migrating.md: breaking changes documented under "From 2.4.x to 2.5.x"

@toots toots force-pushed the harbor-output-changes branch 7 times, most recently from 2155cb8 to 6b8478c Compare March 16, 2026 14:35
@toots toots force-pushed the harbor-output-changes branch from 6b8478c to d5c36b8 Compare April 19, 2026 22:54
@toots toots force-pushed the harbor-output-changes branch 5 times, most recently from ac9aaed to 5f584fb Compare April 28, 2026 05:48
…ix write-task bugs

- Replace Duppy monad-based async client state machine with a simpler
  synchronous listener record
- Add dedicated_encoder mode: creates a fresh encoder per listener,
  useful for ffmpeg copy-mode and formats needing a clean start
- Remove user/password params; auth is now exclusively via the auth function
- Make burst parameter nullable
- Update harbor.md and migrating.md accordingly

Fix three bugs causing listeners to miss data under concurrent load:

1. Race condition in write_task_handler: the listener list was read,
   processed, then written back as a filtered snapshot — any listeners
   added concurrently were silently discarded. Fix: re-read the fresh
   list under lock and filter by the closed flag instead.

2. Busy-loop: write_task_next registered all non-closed listeners for
   write-readiness even with no pending data. Sockets are always writable
   when empty, causing thousands of no-op handler invocations. Fix: only
   watch fds that have pending_data.

3. POLLHUP not handled in poll(2) stub: when a client closes its TCP
   connection, poll() returns POLLHUP (not POLLOUT) on the write fd.
   The C stub ignored POLLHUP, so the Duppy task was never considered
   ready and sat in the queue permanently with write_task_active=true,
   blocking all subsequent listeners. Fix: treat POLLHUP on write-range
   fds as write-ready so the caller gets an EPIPE and can disconnect.

Also remove the stale listener.closed assignment in try_write_to_socket;
handle_disconnect is the single place responsible for that transition.

Add a scale test (replaces k6) using concurrent ffprobe probes with
per-listener timestamped logging and randomised connection stagger.
The startup sequence now uses a lightweight HTTP 200 check instead of
a full ffprobe probe, which is much faster.
@toots toots force-pushed the harbor-output-changes branch from 5f584fb to 7bf0dbf Compare April 28, 2026 06:05
@toots toots merged commit aae1139 into main Apr 28, 2026
59 of 61 checks passed
@toots toots deleted the harbor-output-changes branch April 28, 2026 14:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant