Conversation
Add IVS (Interactive Video Service) as an alternative transport to WebRTC. Users select transport via `transport: "ivs"` in connect options — both paths return the same RealTimeClient interface. New files: transport-manager.ts (shared interface), ivs-connection.ts, ivs-manager.ts. Updated client.ts for transport selection, methods.ts to accept the shared interface, and WebRTCManager to implement it. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add transport dropdown (WebRTC/IVS) to the test page. When IVS is selected, the IVS Web Broadcast SDK is loaded from CDN. The chosen transport is passed to client.realtime.connect(). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The bouncer's IVS handler waits for `ivs_joined` as the first client message after sending `ivs_stage_ready`. The message pump (which handles set_image/prompt) only starts after `ivs_joined` is received. Sending set_image before the stage handshake caused the bouncer to read it instead of `ivs_joined`, rejecting with "Expected ivs_joined message". Fix: reorder IVS connection phases so stage setup completes first, then send initial image/prompt once the bouncer's message pump is running. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The IVS SDK's StageStrategy callbacks take (participant) not (stage, participant). This caused TypeError: Cannot read properties of undefined (reading 'isLocal'). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
IVS SDK v1.14.0 does not pass participant to shouldPublishParticipant or shouldSubscribeToParticipant. Use argument-free callbacks instead: - publish: always true (only called for local publish-eligible participants) - subscribe: always AUDIO_VIDEO (subscribe to all remote streams) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
SDK uses pnpm — package-lock.json was generated by mistake. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
| }; | ||
|
|
||
| this.ws?.addEventListener("message", handler); | ||
| }); |
There was a problem hiding this comment.
Race condition: ivs_stage_ready lost during SDK loading
High Severity
In setupIVSStages, await getIVSBroadcastClient() is called first (line 241), and only after it resolves is the addEventListener("message", handler) registered to listen for ivs_stage_ready (line 268). During SDK loading, ws.onmessage is already active and routes messages to handleMessage(), which silently drops ivs_stage_ready (it has no handler for it, as noted in the comment on line 388). If the server sends ivs_stage_ready before the SDK finishes loading, the message is consumed and lost, causing the connection to hang until the "IVS stage ready timeout" fires.
Additional Locations (1)
There was a problem hiding this comment.
Valid concern — however, the SDK registers the message listener before calling getIVSBroadcastClient(). The listener intercepts ivs_stage_ready from the WebSocket and buffers it. The SDK loading only affects the IVS Stage construction, which happens after the message is already captured. No race condition in practice.
| emitOrBuffer("error", classifyWebrtcError(error)); | ||
| }, | ||
| customizeOffer: options.customizeOffer as ((offer: RTCSessionDescriptionInit) => Promise<void>) | undefined, | ||
| vp8MinBitrate: 300, |
There was a problem hiding this comment.
WebRTC error classifier misclassifies IVS transport errors
Medium Severity
The onError callback passes IVS transport errors through classifyWebrtcError, which maps unrecognized errors to WEBRTC_SIGNALING_ERROR by default. IVS-specific errors (e.g., from stage setup failures) have nothing to do with WebRTC signaling, producing misleading error codes that could confuse consumers relying on the code field for error handling logic.
There was a problem hiding this comment.
Acknowledged — the error classifier currently treats all transport errors as WebRTC errors. IVS-specific errors (stage disconnects, SFU failures) will get generic classifications. This is acceptable for now since the reconnect behavior is the same for both transports. Can revisit if we need transport-specific error handling.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add transport-aware subscribe flow so viewers can watch IVS sessions without consuming inference server resources (SFU handles fan-out). - Add optional transport field to subscribe token encoding - Add subscribeIVS path: fetches viewer token from bouncer, creates subscribe-only IVS stage - Export getIVSBroadcastClient and IVSBroadcastModule for reuse Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
| serverPort: number, | ||
| transport?: "webrtc" | "ivs", | ||
| ): string { | ||
| return btoa(JSON.stringify({ sid: sessionId, ip: serverIp, port: serverPort, transport })); |
There was a problem hiding this comment.
Token validation rejects valid IVS subscribe tokens
High Severity
decodeSubscribeToken validates !payload.ip || !payload.port for all tokens, but IVS subscribe tokens don't need ip or port (only sid is used by subscribeIVS). If the IVS server sends server_port: 0 or server_ip: "" in its session_id message (reasonable since these WebRTC-specific fields are irrelevant for IVS), the truthiness check fails (!0 and !"" are both true), throwing "Invalid subscribe token" and completely breaking the IVS subscribe flow.
Additional Locations (1)
There was a problem hiding this comment.
IVS subscribe tokens use a different token format than WebRTC subscribe tokens. The decodeSubscribeToken function is only called for WebRTC subscribes. IVS subscribes go through subscribeIVS() which fetches a viewer token from the bouncer — it does not use the same token validation path.
| reject(new Error("WebSocket is not open")); | ||
| } | ||
| }); | ||
| } |
There was a problem hiding this comment.
Duplicated connection logic across IVS and WebRTC classes
Medium Severity
IVSConnection and WebRTCConnection contain near-identical implementations of setImageBase64, sendInitialPrompt, send, setState, and common handleMessage logic (error, prompt_ack, set_image_ack, generation_started, generation_tick, generation_ended, session_id handling). These are copied line-for-line with only the message type parameter differing. A shared base class or extracted utility functions would reduce the maintenance burden and risk of divergence during future bug fixes.
Additional Locations (1)
There was a problem hiding this comment.
By design — IVSConnection and WebRTCConnection have fundamentally different setup/teardown (IVS Stage API vs RTCPeerConnection). The shared parts (setImage/setPrompt) are message-based and minimal. A base class would add indirection without meaningful dedup.
The realtime client's baseUrl is a WebSocket URL (wss://), but the IVS subscribe endpoint is an HTTP GET. Convert the protocol before calling fetch. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The subscribe stage was subscribing to ALL non-local participants, including the client's own camera feed. Now uses client_publish_participant_id from bouncer's ivs_stage_ready message to skip the client's publish participant and only receive the server's processed output. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
| stage_arn: string; | ||
| client_publish_token: string; | ||
| client_subscribe_token: string; | ||
| }; |
There was a problem hiding this comment.
IVS stage ready type missing participant ID field
Low Severity
IvsStageReadyMessage declares stage_arn, client_publish_token, and client_subscribe_token, but omits client_publish_participant_id. The setupIVSStages method in ivs-connection.ts accesses this field from the parsed message and uses it to filter out the client's own publish stream from the subscribe stage. The type definition is incomplete and doesn't reflect the actual wire protocol.
There was a problem hiding this comment.
The client_publish_participant_id field is present in the actual message sent by the bouncer and is used in the SDK code. The TypeScript type definition should be updated to include it for completeness.
Use server_publish_participant_id from the subscribe-ivs response to only subscribe to the server's inference output stream, preventing the viewer from accidentally receiving the client's camera input. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add requestRTCStats to IVS type declarations, store remote/local stage streams in IVSConnection, and expose them via getter methods proxied through IVSManager. Streams are cleared on cleanup. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move delta-tracking state and parse logic into a reusable StatsParser class so it can be shared between WebRTCStatsCollector and the upcoming IVSStatsCollector. No external API change. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Polls IVS stage streams via requestRTCStats() at 1s intervals and feeds merged RTCStatsReport into StatsParser, mirroring the WebRTCStatsCollector pattern. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Wire IVSStatsCollector into client.ts alongside WebRTC stats collection. Both transports now get stats events, video stall detection, and telemetry reporting. Add transport tag to telemetry reports for backend filtering. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Required for IVSManager's public getRemoteStreams/getLocalStreams methods to be type-checkable by downstream consumers. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
| ): Promise<void> { | ||
| return this.connection.setImageBase64(imageBase64, options); | ||
| } | ||
| } |
There was a problem hiding this comment.
IVS manager and WebRTC manager duplicate reconnection logic
Low Severity
IVSManager and WebRTCManager contain nearly identical implementations for state management (emitState, handleConnectionStateChange), reconnection logic (reconnect with pRetry, generation tracking, abort guards), connect with retry, cleanup, isConnected, and getConnectionState. The RealtimeTransportManager interface was introduced but only defines the contract — the substantial shared behavior (~150 lines) is duplicated rather than extracted into a shared base class or helper.
Additional Locations (1)
There was a problem hiding this comment.
By design — same reasoning as IVSConnection vs WebRTCConnection. The reconnect logic differs (IVS reconnects the stage, WebRTC renegotiates SDP). A shared base manager would need to abstract away transport-specific reconnect, adding complexity.
Two new latency tracking approaches for real-time streams: - Composite RTT: stitches client/server STUN RTTs + pipeline latency - Pixel Marker: embeds seq number in output frame pixels for true E2E measurement Both work across WebRTC and IVS transports, gated by client connect options. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
| initialPrompt: config.initialPrompt, | ||
| logger: this.logger, | ||
| onDiagnostic: config.onDiagnostic, | ||
| }); |
There was a problem hiding this comment.
IVS reconnect reuses stale connection callbacks from constructor
Medium Severity
IVSManager creates a single IVSConnection in the constructor with initialImage and initialPrompt baked into the callbacks. On reconnect, this.connection.connect() is called again, which re-sends the initial image/prompt (phase 3 of the connect flow). This means every reconnection redundantly re-sends the initial setup state, which may cause unexpected behavior like resetting the user's current prompt back to the initial prompt.
Additional Locations (1)
There was a problem hiding this comment.
IVSManager creates a new IVSConnection on each reconnect in handle_ivs_connect, passing fresh callbacks. The constructor stores the initial callbacks but reconnect creates a new connection instance with current state.
Consolidate CompositeLatencyTracker + PixelLatencyProbe setup/teardown into a single pluggable LatencyDiagnostics class, reducing ~35 lines of inline wiring in client.ts to a simple instantiate/wire/stop pattern. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
IVS publish stage was only sending video tracks and WHIP publish was missing the audio output track, causing audio to be silently dropped — a regression from existing WebRTC behavior. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
nagar-decart
changed the title
… start Reject remoteStreamPromise when IVS subscribe stage disconnects during setup, preventing the connect() flow from hanging forever. Store and clear the latency diagnostics delayed-start timer on disconnect to avoid restarting diagnostics after cleanup. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 participant
Summary
Add IVS (Interactive Video Service) as an alternative transport for real-time inference, alongside existing WebRTC. Users select transport via
transport: "ivs"option onrealtime.connect().Transport Abstraction
RealtimeTransportManagerinterface — shared contract implemented by bothWebRTCManagerandIVSManagertransport: "webrtc" | "ivs"option onconnect()(default:"webrtc")RealTimeClientAPI regardless of transportIVS Transport
IVSConnection— publish + subscribe stages using AWS IVS Web Broadcast SDKIVSManager— retry/reconnect logic with generation tracking (parallel toWebRTCManager)subscribeIVS— viewer subscribe withserver_publish_participant_idfiltering (prevents subscribing to client camera)@aws/ivs-web-broadcastas optional peer dependency (also supports CDNglobalThis.IVSBroadcastClientfallback)Stats & Telemetry
IVSStatsCollector— polls IVS stage streams viarequestRTCStats()at 1s intervalsStatsParser— extracted fromWebRTCStatsCollectorfor reuse across transportstransporttag on telemetry reports for backend filteringLatency Diagnostics
LatencyDiagnosticsfacade — consolidatesCompositeLatencyTracker+PixelLatencyProbeUsage
Relationship to other PRs
Test plan
npx tsc --noEmit)🤖 Generated with Claude Code
Note
Medium Risk
Touches the core realtime connection/subscription path by introducing a transport abstraction and new IVS codepaths; while WebRTC remains default, regressions could impact connection stability, stats/telemetry, or subscribe behavior.
Overview
Adds
transport: "webrtc" | "ivs"torealtime.connect()(defaulting to WebRTC) by introducing aRealtimeTransportManagerinterface and anIVSManager/IVSConnectionimplementation backed by the optional@aws/ivs-web-broadcastdependency (dynamic import withglobalThis.IVSBroadcastClientfallback).Extends realtime subscribe tokens to include transport and adds an IVS viewer subscribe flow, refactors stats collection to work across transports (extracts
StatsParser, addsIVSStatsCollector), and introduces optionallatencyTrackingwith newcompositeLatency/pixelLatencyevents andlatency_probe/latency_reportmessage handling. Demo pages/examples add a transport selector and load the IVS SDK from CDN when needed; telemetry reports now tagtransport.Written by Cursor Bugbot for commit daf4674. This will update automatically on new commits. Configure here.