Add MuScriptor audio-to-MIDI transcription (model + example)#3738
Open
sinkingsugar wants to merge 4 commits into
Open
Add MuScriptor audio-to-MIDI transcription (model + example)#3738sinkingsugar wants to merge 4 commits into
sinkingsugar wants to merge 4 commits into
Conversation
…chains float16x8_t needs the stdarch_neon_f16 library feature, which is still unstable (rust-lang/rust#136306), and target_feature=fp16 is on by default on aarch64-apple-darwin — so stable Apple Silicon builds failed with E0658. build.rs now detects a nightly compiler and sets cfg(candle_nightly); only then are the fp16-accumulating kernels compiled, with stable falling back to the FCVTL widening tier.
The dependabot bump to symphonia 0.6 broke every example behind the symphonia/mimi/encodec/snac features: 0.6 is a full API rework (formats::probe, audio::sample, ...) while the examples still use the 0.5 API. CI never builds those features, so it went unnoticed.
Port of the Kyutai/Mirelo MuScriptor inference (decoder-only prefix-LM, audiocraft lineage): log-mel conditioning prepended as prefix tokens, MIDI event tokens generated autoregressively over a single stream, then decoded to notes through the MT3 tokenizer's tie-prologue state machine. - candle-transformers/src/models/muscriptor: transformer + LM + mel/class conditioners (mel STFT in pure Rust, window and HTK filterbank loaded from the checkpoint), token vocabulary, streaming note-event decoder, note cleanup passes; unit tests for the pure-logic parts. - candle-examples/examples/muscriptor: CLI (symphonia decode, julius-style sinc resampler, chunked batched generation, mido-compatible SMF writer). Verified against the PyTorch reference (large checkpoint, greedy): MIDI output is byte-identical on CPU f32 and Metal f32, event-identical on Metal f16. Weights: hf.co/MuScriptor (CC BY-NC 4.0).
- Use the fused Metal SDPA (vector kernel for single-token decode, causal full kernel for prefill) instead of manual matmul/softmax/matmul; the kernels read the KV cache's narrowed views through their strides. - Use the fused layer-norm kernel on Metal instead of the decomposed mean/variance op chain (2 norms x 48 layers of small launches). - One batched argmax and a single device-to-host transfer per decode step instead of one sync per batch row. Large model, 30 s of music, batch 4, M-series Metal f16: 38 s -> 16.6 s generation (PyTorch MPS f16 reference: 11.9 s). Verification gates hold: MIDI byte-identical to the PyTorch reference on CPU f32 and Metal f32, event-identical on Metal f16.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds MuScriptor (Kyutai/Mirelo lineage), an audio→MIDI music transcription model, plus two small build fixes it surfaced.
Model
A decoder-only prefix-LM in the audiocraft/MusicGen style: audio is processed in 5-second chunks, each chunk's log-mel spectrogram is projected and prepended as prefix tokens (together with instrument-group/dataset class embeddings), then MIDI event tokens are generated autoregressively over a single stream and decoded into notes via an MT3-style tokenizer.
candle-transformers/src/models/muscriptor/: transformer + LM + conditioners; a pure-Rust STFT/mel front-end (the Hann window and HTK filterbank are loaded from checkpoint buffers, so they match the reference by construction); the token vocabulary and the streaming token→note-event decoder, with unit tests for the pure-logic parts.candle-examples/examples/muscriptor/: CLI — symphonia decode, a julius-style sinc resampler, chunked/batched generation with streaming note output, and a MIDI (SMF format 0) writer. No new dependencies.Verified against the PyTorch reference implementation (large checkpoint, greedy decoding): MIDI output is byte-identical on CPU f32 and Metal f32, and event-identical on Metal f16, on both synthetic and real-music inputs.
Note the published weights are CC BY-NC 4.0 (small/medium/large variants; gated repos).
Build fixes included
float16x8_trequires the unstablestdarch_neon_f16feature (Tracking Issue for NEON fp16 intrinsics rust-lang/rust#136306) andtarget_feature=fp16is enabled by default onaarch64-apple-darwin, socargo checkon stable currently fails with E0658 on Apple Silicon. A smallbuild.rssetscfg(candle_nightly)on nightly compilers; stable builds fall back to the existing FCVTL-widening kernels.symphonia/mimi/encodec/snacfeatures; CI doesn't build those features so it went unnoticed.Happy to split the two fixes into separate PRs if preferred.