Skip to content

WebCodecs Support#2160

Open
seydx wants to merge 6 commits intoAlexxIT:masterfrom
seydx:webcodecs
Open

WebCodecs Support#2160
seydx wants to merge 6 commits intoAlexxIT:masterfrom
seydx:webcodecs

Conversation

@seydx
Copy link
Copy Markdown
Contributor

@seydx seydx commented Mar 17, 2026

This PR adds a new WebCodecs-based streaming mode to go2rtc, delivering raw encoded frames over WebSocket for browser-side decoding via the WebCodecs API.

Features

  • Raw encoded video/audio frames via WebSocket
  • H.264 and H.265 video support
  • AAC, Opus, PCMA, PCMU audio support (no server-side encoding needed)
  • Binary frame protocol with 9-byte header
  • AVCC to Annex-B conversion for browser compatibility
  • Native-style player controls (play/pause, volume, fullscreen)
  • GPU-accelerated rendering: WebGPU → WebGL2 → Canvas 2D (automatic fallback)
  • Configurable renderer cascade via renderer query parameter

Usage

# Default (full cascade)
stream.html?src=camera&mode=webcodecs

# Force specific renderer
stream.html?src=camera&mode=webcodecs&renderer=webgl

# Skip WebGPU
stream.html?src=camera&mode=webcodecs&renderer=webgl,2d

@seydx seydx changed the title Webcodecs Support WebCodecs Support Mar 17, 2026
Comment thread www/video-rtc.js Outdated
Comment thread www/links.html
<h2>H264/H265 source</h2>
<li><a href="stream.html?src=${src}&mode=webrtc">stream.html</a> WebRTC stream / browsers: all / codecs: H264, PCMU, PCMA, OPUS / +H265 in Safari</li>
<li><a href="stream.html?src=${src}&mode=mse">stream.html</a> MSE stream / browsers: Chrome, Firefox, Safari Mac/iPad / codecs: H264, H265*, AAC, PCMA*, PCMU*, PCM* / +OPUS in Chrome and Firefox</li>
<li><a href="stream.html?src=${src}&mode=webcodecs">stream.html</a> WebCodecs stream / browsers: Chrome, Edge, Firefox, Safari Mac/iPad / codecs: H264, H265, AAC, OPUS, PCMA, PCMU</li>
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Firefox

sure?

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WebCodecs is supported in all major browser, but hevc not:

Microsoft Edge on Windows

By default, Chromium browsers uses FFMpeg internally for WebCodecs API. However, Microsoft Edge, when running on Windows, uses Media Foundation decoders instead.

Decoding H.265 requires the HEVC Video Extensions ($0.99) or HEVC Video Extensions from Device Manufacturer (free but not available anymore) app from Microsoft Store.

Firefox

Firefox 133 supports playing H.265 videos, but does not support decoding H.265 streams using WebCodecs API yet.

Comment thread pkg/webcodecs/consumer.go Outdated
@AlexxIT
Copy link
Copy Markdown
Owner

AlexxIT commented Mar 17, 2026

  1. I like using new, modern technologies - it's cool.
  2. I don't like the large amount of JavaScript code in the main player. Ideally, move everything into a separate file and make it easily connectable.
  3. I'm not sure about the header for the binary format. It needs to be carefully designed. Otherwise, it will be difficult to change later. You're using uint32 for the time. That might not be enough. The header includes the packet size (maybe that's a good thing), but it isn't being used.

@seydx
Copy link
Copy Markdown
Contributor Author

seydx commented Mar 17, 2026

  1. I like using new, modern technologies - it's cool.
  2. I don't like the large amount of JavaScript code in the main player. Ideally, move everything into a separate file and make it easily connectable.
  3. I'm not sure about the header for the binary format. It needs to be carefully designed. Otherwise, it will be difficult to change later. You're using uint32 for the time. That might not be enough. The header includes the packet size (maybe that's a good thing), but it isn't being used.
  • Code is now split into three modules: video-webcodecs.js (player/decoder/audio/controls), video-renderer.js (WebGPU/WebGL2/Canvas2D cascade), and the existing video-rtc.js just wires them together
  • Header redesigned: dropped the unused payload length field, timestamp is now uint64 (microseconds). 9 bytes total: flags(1) + timestamp_us(8) + payload.

skrashevich added a commit that referenced this pull request Mar 21, 2026
@skrashevich
Copy link
Copy Markdown
Collaborator

merged into dev branch
@seydx please note the fixes in da908ca

@skrashevich skrashevich added dev Issues/PRs related only to dev branch and removed dev Issues/PRs related only to dev branch labels Mar 21, 2026
@skrashevich skrashevich added this to the dev milestone Mar 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants