Skip to content

add optional twelvelabs pegasus transcription backend#36

Open
mohit-twelvelabs wants to merge 1 commit into
dsymbol:mainfrom
mohit-twelvelabs:feat/twelvelabs-integration
Open

add optional twelvelabs pegasus transcription backend#36
mohit-twelvelabs wants to merge 1 commit into
dsymbol:mainfrom
mohit-twelvelabs:feat/twelvelabs-integration

Conversation

@mohit-twelvelabs

Copy link
Copy Markdown

Hi! I'm Mohit, I work at TwelveLabs (@mohit-twelvelabs).

This PR adds an optional TwelveLabs Pegasus transcription backend alongside the default Whisper pipeline.

What it adds

  • A new decipher/pegasus.py module with video_to_srt(...): uploads the video as a TwelveLabs asset, waits for it to be ready, and asks Pegasus to return SubRip (SRT) subtitles (output is fence-stripped and validated against an SRT cue regex).
  • A --backend {whisper,pegasus} flag on decipher transcribe (default whisper), wired through action.transcribe(...) via a new backend argument.
  • An optional dependency group: pip install "decipher[pegasus]" (twelvelabs>=1.2.8).
  • README docs and a tests/test_pegasus.py (no-network unit tests for parsing/validation + a key-gated wiring test that skips without TWELVELABS_API_KEY).

Why it helps

Pegasus is a video-understanding model that reads the video directly rather than just the extracted audio track, so it can use on-screen context (overlaid text, speaker cues) when transcribing. It also runs in the cloud, so users without a local GPU get an alternative to Whisper. Marengo/Pegasus and Whisper are complementary, not a replacement.

Opt-in / non-breaking

Default behaviour is unchanged: backend defaults to whisper, the twelvelabs package is only imported inside the pegasus branch, and it's an extras_require (not installed by default). Existing Whisper users see no difference.

How it was tested

  • tests/test_pegasus.py passes (5 passed): fence stripping, SRT cue regex, missing-key error, and SDK construction with a key.
  • End-to-end verified against the live TwelveLabs API: a short clip was uploaded, polled to ready, analysed with Pegasus 1.5, and returned a valid SRT cue (~8s). Marengo text embeddings also confirmed (512-dim) while validating the SDK contract.
  • Formatted with black.

You can grab a free API key at https://twelvelabs.io — there's a generous free tier.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant