Skip to content

subtitle burn: use ffmpeg-full for libass-dependent filter#357

Closed
sonichi wants to merge 2 commits intofix/subtitled-pending-false-positivefrom
fix/subtitle-burn-use-ffmpeg-full
Closed

subtitle burn: use ffmpeg-full for libass-dependent filter#357
sonichi wants to merge 2 commits intofix/subtitled-pending-false-positivefrom
fix/subtitle-burn-use-ffmpeg-full

Conversation

@sonichi
Copy link
Copy Markdown
Owner

@sonichi sonichi commented Apr 16, 2026

Summary

Fixes the root cause Susan called out early and I spent all afternoon chasing: the homebrew ffmpeg bottle for 8.1 on arm64_tahoe is built without libass, so the subtitles filter doesn't exist in the binary at all. Every burnLiveTranscriptSubtitles() call has been silently failing with:

[AVFilterGraph] No option name near '.../subtitle.srt'

because the parser was treating subtitles as an unknown option name rather than a filter. Four wrong theories chased before ffmpeg -filters | grep subtitles returned zero hits.

Fix

  • Use the keg-only ffmpeg-full binary at /opt/homebrew/opt/ffmpeg-full/bin/ffmpeg for the subtitle burn specifically. ffmpeg-full bundles libass + libfreetype + fontconfig + 43 other deps.
  • Fallback: if ffmpeg-full isn't installed, fall through to the narrow /opt/homebrew/bin/ffmpeg. Preserves behavior on MacBook / CI / any node without ffmpeg-full. The narrow call will fail loudly as before — strictly no worse than today's silent failure.
  • Scope: only burnLiveTranscriptSubtitles() — all other ffmpeg call sites (narration mux via videotoolbox, record.py mux, etc.) keep using the fast narrow bottle. They don't need libass.

Install required (per-node, one-time)

brew install ffmpeg-full

~500MB–1GB, 5–15 min, 46 deps. Already done on Mac Mini (verified: ffmpeg -filters | grep subtitles returns the filter definition).

End-to-end test (already run on Mac Mini)

Ran the EXACT ffmpeg command with the new /opt/homebrew/opt/ffmpeg-full/bin/ffmpeg path against /tmp/sutando-recording-1776298340-narrated.mov + /tmp/sutando-live-transcript-subtitle.srt (both from owner's earlier failed test call):

frame=  229 fps= 58 q=-0.0 Lsize=    2900KiB time=00:00:15.20 bitrate=1563.1kbits/s

Output: /tmp/TEST-subtitled.mov, 2.9 MB, valid mov. ✓

Log delta

Success log now reports which ffmpeg was used:

[ScreenRecord] live transcript subtitles burned: /tmp/...-subtitled.mov (ffmpeg=/opt/homebrew/opt/ffmpeg-full/bin/ffmpeg)

Stacks on #355

Base branch: fix/subtitled-pending-false-positive (MacBook's #355). Neither PR alone is sufficient:

Merge #355 first, then this.

History (for the retro)

Chased before landing here:

  1. Wrong process restart (voice-agent vs conversation-server)
  2. Stale local main (git fetch vs git pull)
  3. Filter rejection (all-entries-filtered on short recordings)
  4. Comma escaping in force_style (multiple backslash counts, none worked)

Susan raised "is it a dependency issue?" first. Owner also. I dismissed it twice. The trivial verification that would have landed this in 5 minutes: ffmpeg -filters | grep subtitles → zero hits → check ffmpeg -version → no --enable-libass.

🤖 Generated with Claude Code

The `subtitles` filter in ffmpeg requires libass at build time. The
homebrew `ffmpeg` bottle for 8.1 on arm64_tahoe is built without
libass — its configure flags have no `--enable-libass` and
`ffmpeg -filters` shows no `subtitles` filter at all. Every
`burnLiveTranscriptSubtitles()` call today hit:

    [AVFilterGraph] No option name near '.../subtitle.srt'

because the parser was treating `subtitles` as an unknown option name,
not as a filter.

### Fix

Use the keg-only `ffmpeg-full` binary at
`/opt/homebrew/opt/ffmpeg-full/bin/ffmpeg` for the subtitle burn
specifically. ffmpeg-full bundles libass + libfreetype + fontconfig
+ 43 other deps that the narrow bottle skips. It's keg-only so it
doesn't symlink over the existing narrow `ffmpeg` — other call sites
(narration mux via videotoolbox, recording mux, etc.) keep using the
fast narrow bottle. Only the subtitle burn reaches for the full one.

### Fallback

If ffmpeg-full isn't installed, fall through to the narrow `/opt/homebrew/bin/ffmpeg`
call as before. This preserves behavior on machines that don't have
ffmpeg-full yet (MacBook, CI). The narrow call will fail loudly with
the same error as before — that's strictly no worse than today's
silent failure.

### Install (one-time, per-node)

    brew install ffmpeg-full  # ~500MB–1GB, 5–15 min, 46 deps

### Log delta

The success log line now includes which ffmpeg binary was used:

    [ScreenRecord] live transcript subtitles burned: /tmp/...-subtitled.mov (ffmpeg=/opt/homebrew/opt/ffmpeg-full/bin/ffmpeg)

so operators can confirm the full binary was picked up post-install.

### Stacks on

#355 (subtitled_pending false-positive gate). Same branch tree. Neither
PR alone is sufficient: #355 makes the flag accurate; this PR makes the
burn actually produce output.

### History

Four wrong theories chased before landing on this one:
1. wrong process (voice-agent vs conversation-server)
2. stale local main (git fetch vs git pull)
3. filter rejection (entries.length === 0 after aggressive filter)
4. comma escaping in force_style (tried multiple backslash counts, all failed)

Susan raised "is it a dependency issue?" early in the thread. Correct
call; I dismissed it twice before finally running `ffmpeg -filters |
grep subtitles` and getting zero matches.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Copy link
Copy Markdown

@josuachavesolmos josuachavesolmos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed by Sutando proactive loop.

APPROVE — minimal, surgical change. Only burnLiveTranscriptSubtitles() is affected; all other ffmpeg call sites verified untouched. Graceful fallback via existsSync check. Good observability with (ffmpeg=${ffmpegBin}) in logs.

One follow-up note:

  • [MEDIUM] skills/screen-record/scripts/subtitle.py also uses the subtitles ffmpeg filter with bare ffmpeg — will hit the same libass error. Track as a separate issue.

Safe to merge after PR #355 lands (stacked dependency).

Replace the hardcoded ffmpeg-full path with a runtime probe that
automatically finds the right ffmpeg binary. No user configuration
needed.

findFfmpegWithSubtitles() probes candidates in order:
  1. $FFMPEG_SUBTITLE_BIN env var (explicit override, skips probe)
  2. System `ffmpeg` (whatever is on PATH)
  3. /opt/homebrew/bin/ffmpeg (homebrew narrow)
  4. /opt/homebrew/opt/ffmpeg-full/bin/ffmpeg (homebrew full, keg-only)

Each candidate gets `ffmpeg -filters 2>&1` and we grep for "subtitles".
First match wins. Result cached for the process lifetime (~50ms first
call, 0ms thereafter). If none have it, logs the failure with an
install hint and returns null — the subtitle burn is skipped gracefully
instead of crashing with the cryptic AVFilterGraph error.

On Mac Mini today: system ffmpeg and /opt/homebrew/bin/ffmpeg both
return subtitles=false. /opt/homebrew/opt/ffmpeg-full/bin/ffmpeg
returns subtitles=true → selected as winner. On MacBook (if ffmpeg
has libass baked in), the system ffmpeg would win on the first probe.

Also supports $FFMPEG_SUBTITLE_BIN env var as an explicit override for
any edge case (custom builds, non-homebrew installs, CI).

Owner asked "can we figure out the right ffmpeg without user effort?"
— this is the answer.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Copy link
Copy Markdown
Owner Author

@sonichi sonichi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MacBook review: LGTM. Auto-detects ffmpeg with subtitle filter support, cached after first probe, env var override available. Falls back to homebrew ffmpeg-full. Clean, no runtime cost after first call.

sonichi pushed a commit that referenced this pull request Apr 16, 2026
Per owner's design directive: "The recording tool should return the
files and the default file to use. The caller should process the result
and report abnormality if any. Then open_file tool should be called with
filename."

Changes:

1. open_file: stripped recording-specific logic.
   - Requires `path` arg (get it from recording tool result).
   - `find_latest_recording: true` flag as explicit fallback — only
     called when model lost the path, not as silent default.
   - Removed: subtitled_pending flag, version detection, polling
     logic, findRecording() as default behavior.
   - Kept: known file aliases (diagnostics), QuickTime activation,
     playback-path write, duration/size metadata.

2. screen_record stop: returns explicit file list.
   - New `files` object: `{raw, narrated, subtitled, recommended}`.
   - `instruction` string tells the model exactly which path to pass
     to open_file and how to offer alternatives to the user.
   - Subtitle burn still happens synchronously on stop.

3. No changes to record_screen_with_narration — its auto-stop timer
   fires in a setTimeout and can't return files to the model through
   a tool result. The model already has the raw path from the start
   call; open_file with that path works. Full file-list return for
   record_screen_with_narration is a follow-up.

Net: +39/-38 (near-zero). This is a separation of concerns, not new
features. findRecording() and findFfmpegWithSubtitles() are preserved
as helpers; they just stop being called by default in open_file.

Supersedes the approach in #353 (non-blocking return + subtitled_pending)
and makes #355 (subtitled_pending gate) + #357 (ffmpeg-full path)
unnecessary as open_file changes — though #357's findFfmpegWithSubtitles
is still needed in the recording tool's burn path.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@sonichi
Copy link
Copy Markdown
Owner Author

sonichi commented Apr 16, 2026

Superseded by #392 (refactor: decouple open_file from recording logic). findFfmpegWithSubtitles() is folded into #392 as part of the unified refactor.

@sonichi sonichi deleted the branch fix/subtitled-pending-false-positive April 16, 2026 13:43
@sonichi sonichi closed this Apr 16, 2026
@sonichi sonichi deleted the fix/subtitle-burn-use-ffmpeg-full branch April 16, 2026 13:43
sonichi added a commit that referenced this pull request Apr 16, 2026
Per owner's design directive: "The recording tool should return the
files and the default file to use. The caller should process the result
and report abnormality if any. Then open_file tool should be called with
filename."

Changes:

1. open_file: stripped recording-specific logic.
   - Requires `path` arg (get it from recording tool result).
   - `find_latest_recording: true` flag as explicit fallback — only
     called when model lost the path, not as silent default.
   - Removed: subtitled_pending flag, version detection, polling
     logic, findRecording() as default behavior.
   - Kept: known file aliases (diagnostics), QuickTime activation,
     playback-path write, duration/size metadata.

2. screen_record stop: returns explicit file list.
   - New `files` object: `{raw, narrated, subtitled, recommended}`.
   - `instruction` string tells the model exactly which path to pass
     to open_file and how to offer alternatives to the user.
   - Subtitle burn still happens synchronously on stop.

3. No changes to record_screen_with_narration — its auto-stop timer
   fires in a setTimeout and can't return files to the model through
   a tool result. The model already has the raw path from the start
   call; open_file with that path works. Full file-list return for
   record_screen_with_narration is a follow-up.

Net: +39/-38 (near-zero). This is a separation of concerns, not new
features. findRecording() and findFfmpegWithSubtitles() are preserved
as helpers; they just stop being called by default in open_file.

Supersedes the approach in #353 (non-blocking return + subtitled_pending)
and makes #355 (subtitled_pending gate) + #357 (ffmpeg-full path)
unnecessary as open_file changes — though #357's findFfmpegWithSubtitles
is still needed in the recording tool's burn path.

Co-authored-by: Chi <wangchi@Chis-Mac-mini.local>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants