output.harbor: refactor listener management, add dedicated encoder mode#5003
Merged
output.harbor: refactor listener management, add dedicated encoder mode#5003
Conversation
2155cb8 to
6b8478c
Compare
6b8478c to
d5c36b8
Compare
ac9aaed to
5f584fb
Compare
…ix write-task bugs - Replace Duppy monad-based async client state machine with a simpler synchronous listener record - Add dedicated_encoder mode: creates a fresh encoder per listener, useful for ffmpeg copy-mode and formats needing a clean start - Remove user/password params; auth is now exclusively via the auth function - Make burst parameter nullable - Update harbor.md and migrating.md accordingly Fix three bugs causing listeners to miss data under concurrent load: 1. Race condition in write_task_handler: the listener list was read, processed, then written back as a filtered snapshot — any listeners added concurrently were silently discarded. Fix: re-read the fresh list under lock and filter by the closed flag instead. 2. Busy-loop: write_task_next registered all non-closed listeners for write-readiness even with no pending data. Sockets are always writable when empty, causing thousands of no-op handler invocations. Fix: only watch fds that have pending_data. 3. POLLHUP not handled in poll(2) stub: when a client closes its TCP connection, poll() returns POLLHUP (not POLLOUT) on the write fd. The C stub ignored POLLHUP, so the Duppy task was never considered ready and sat in the queue permanently with write_task_active=true, blocking all subsequent listeners. Fix: treat POLLHUP on write-range fds as write-ready so the caller gets an EPIPE and can disconnect. Also remove the stale listener.closed assignment in try_write_to_socket; handle_disconnect is the single place responsible for that transition. Add a scale test (replaces k6) using concurrent ffprobe probes with per-listener timestamped logging and randomised connection stagger. The startup sequence now uses a lightweight HTTP 200 check instead of a full ffprobe probe, which is much faster.
5f584fb to
7bf0dbf
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reason
output.harborwas overdue for a cleanup. The old implementation used a Duppy monad-based async state machine with condition variables to manage connected clients, which was more complex than necessary and harder to reason about. This PR also addresses several scalability issues.Changes
Architecture
Replaces the client state machine with a simpler synchronous
listenerrecord. Writes are moved off the streaming thread entirely into a single Duppy task that waits on listener socket write events (\Write fd`), fires only when at least one socket is ready, writes to all listeners non-blocking, and re-registers itself. This decouples the streaming cycle from network write latency.New
dedicated_encodermodeA
dedicated_encoderboolean parameter (defaultfalse) selects between two output classes:shared_output(default): one encoder shared across all listeners, each seeded with the codec header and burst data on connectdedicated_output: a fresh encoder created per listener at connect time, ensuring a clean stream from the first byte — particularly useful for%ffmpegin copy modeScalability fixes
Strings.Mutable(thread-safe); writes iterate chunks directly viaBytes.unsafe_of_stringwith no allocationBreaking API changes
userandpasswordparameters removed; authentication is now exclusively via theauthfunction, which now receives a record{address, login, password}instead of individual argumentsburstparameter is now nullable (nulldisables burst)Other improvements
max_title→max_icy_title, etc.) and formatting extracted into standalone functionsbuflen→buffer_limit,metaint→metadata_interval, etc.)virtual baseclassDocumentation
doc/content/harbor.md: newoutput.harborsection documentingdedicated_encoder, auth, and listener callback signaturesdoc/content/migrating.md: breaking changes documented under "From 2.4.x to 2.5.x"