Moonshine speech engine + Wayland overlay fixes (v0.3.0)#61
Merged
Conversation
The "moonshine" backend was selectable in Settings, config, and docs but had no implementation: choosing it silently substituted whisper-cpp (a warning-only log), and the startup/keypress "Whisper model not downloaded" guards were skipped for a Moonshine selection — so a user who picked Moonshine could end up with a permanently silent dictation pipeline and no indication why. Backend (crates/voxctrl-inference/src/moonshine.rs, feature-gated): - Runs the four upstream Moonshine ONNX graphs (preprocess, encode, uncached_decode, cached_decode) via ort, with greedy autoregressive decoding from the start-of-transcript token to the end-of-transcript token, capped by audio duration. - Decodes token ids to text with the model's tokenizer.json (tokenizers crate). - Binds tensors to graph inputs positionally and detects the optional seq_len input and KV-cache arity from the graph, so it tolerates export revisions. - Downloads the ONNX graphs + tokenizer on demand into ~/.local/share/voxctrl/models/moonshine/<size>/, or reads a manual copy. Wiring and correctness: - build_backend constructs the real backend when compiled; otherwise it keeps a clear whisper-cpp fallback. - MOONSHINE_COMPILED exposes whether the backend is built in; the two src-tauri model-download guards now only skip the Whisper-model check when Moonshine is actually compiled (a non-Moonshine build still needs the Whisper model). - New tauri commands (moonshine_available, check_moonshine_downloaded, download_moonshine_model) drive a Moonshine download button + status and a "not included in this build" notice in Settings → Engine. Moonshine is an opt-in compile feature (--features moonshine), like cuda/vulkan, since it links ONNX Runtime. Docs updated accordingly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GpXWVxBBJvtRgkdMWasSH6
Make the Moonshine engine ship in every distributed artifact so users can pick either speech engine, the same way whisper-cpp is always available: - build_appimage.sh: add `moonshine` to both the CUDA and CPU `tauri build` invocations. - release.yml: add `moonshine` to every artifact build — Linux CPU AppImage/deb, Linux Vulkan AppImage/deb, Linux CUDA deb, and Windows CPU/CUDA installers. ort's `download-binaries` (default) links ONNX Runtime *statically* into the executable, so these binaries stay self-contained — no extra .so/.dll to bundle, and the AppImage/NSIS packaging is unchanged. The trade-off is that these build steps now fetch the ONNX Runtime static lib at compile time (needs network) and produce a somewhat larger binary. The `moonshine` cargo feature is kept as the toggle (so a plain `cargo check --workspace` stays fast and network-free); it is simply enabled by every real build/release command. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GpXWVxBBJvtRgkdMWasSH6
The download failed because the backend targeted an outdated 4-file model layout (preprocess/encode/uncached_decode/cached_decode) that no longer exists upstream. The current Moonshine ONNX export (matching useful-moonshine-onnx) is two graphs: onnx/merged/<size>/float/encoder_model.onnx onnx/merged/<size>/float/decoder_model_merged.onnx Rewrite the backend to that architecture: - Encoder: raw 16 kHz audio -> hidden states (with optional attention_mask). - Merged KV-cached decoder driven by a `use_cache_branch` toggle and named `past_key_values.<layer>.<decoder|encoder>.<key|value>` inputs. Greedy decode from SOT (1) to EOT (2): the first step feeds empty caches; later steps reuse the decoder self-attention cache while the encoder cross-attention cache stays frozen. Decoder inputs are bound by name (they include optional masks and many cache tensors); empty rank-4 caches are built via the allocator since `from_array` rejects their 0-sized dimension. - Per-size geometry: tiny = 6 layers, base = 8 layers (8 kv-heads; head_dim 36/52), validated against the loaded graph. The tokenizer is bundled (embedded from assets/moonshine_tokenizer.json) rather than downloaded, matching upstream, so model download now fetches just the two ONNX graphs. Docs updated to describe the 2-graph layout. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GpXWVxBBJvtRgkdMWasSH6
Reduce duplication and speed up the decode hot path. Deduplication: - Extract the twice-copied `expand_tilde` and the models-base-dir construction into a shared `util` module; whisper-cpp and Moonshine now both use it. Move the tilde tests alongside the helper. Decode-loop optimization: - Resolve each decoder input to its source once at load time into a `decoder_plan`, then bind inputs positionally at run time. This drops the per-step HashMap rebuilds, per-input string matching, and name-string clones that ran on every one of up to 192 decode steps. - Store KV caches in a Vec (emit order) instead of a name-keyed HashMap, and skip re-copying the encoder cross-attention caches on cached steps (they are fixed after the first step) — only the decoder self-attention caches are read back and grown. - Allocate the all-ones attention mask only when a graph actually consumes it. No behavior change; 48 tests pass with the feature enabled, 34 without. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GpXWVxBBJvtRgkdMWasSH6
The overlay's always-on-top level was applied only once, when the window was first mapped, and never re-asserted. On X11 a window is free to lose its topmost position afterwards — another window taking `_NET_WM_STATE_ABOVE`, a fullscreen app, or the compositor re-stacking between dictations — and the overlay would stay behind, sometimes fully hidden, for the rest of the session. Re-assert the always-on-top level (and click-through) via a new `reassert_topmost` helper: - when the overlay activates for a new dictation (idle → visible), so it comes back to the front even if it fell behind while idle; - on a ~1s heartbeat while visible, so it can't linger behind a window that came forward mid-dictation. The helper toggles Normal → AlwaysOnTop rather than re-setting the same level, because re-adding an already-set state is a no-op that won't re-raise; the toggle forces the WM to re-evaluate stacking. Both calls happen within one render tick with no frame presented between them, so there is no visible flicker. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GpXWVxBBJvtRgkdMWasSH6
The always-on-top re-assertion toggled the window level (Normal → AlwaysOnTop) every time it was applied, including on the very first map. Toggling the state while the surface is still being mapped left some compositors with the window unpresented until the next state change — so at launch no overlay appeared until re-selecting the overlay style forced another update. Only toggle when re-raising an already-mapped window (activation after idle, and the visible heartbeat). On the first map, set AlwaysOnTop directly, as before. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GpXWVxBBJvtRgkdMWasSH6
Re-asserting the always-on-top level disturbs the window's position: winit's set_window_level re-syncs the window geometry from winit's own position tracking (and some WMs re-place a window on a state change), moving the overlay toward the WM's default spot. The render tick then failed to correct it because Slint's set_position is cached — re-sending the same (unchanged) value is a no-op, so no corrective request reached the compositor and the overlay stuck near center instead of the configured position. For a short window (~100 ms) after every level change — the first map, each activation, and each heartbeat — push the position through winit's set_outer_position directly, which is not cached and re-asserts the geometry. Steady-state frames keep using Slint's cheap cached set_position. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GpXWVxBBJvtRgkdMWasSH6
On a native Wayland session a regular client cannot control its own stacking or
position — the compositor decides both — so the overlay's always-on-top and
configured placement were silently ignored, leaving it able to fall behind other
windows and mis-placed (offset from center by the display's scale factor).
Two changes:
1. Run the overlay through XWayland on Wayland sessions (spawn with
WAYLAND_DISPLAY removed when an X server is reachable). As an X11 client,
KWin honors _NET_WM_STATE_ABOVE and window positioning. On a pure X11 session
this is a no-op.
2. Compute the overlay position inside the overlay process from its own winit
monitor + window size, instead of receiving pre-scaled pixel coordinates from
the app process. The two processes can see different display scales, which
put the window at a fraction of the intended offset (up-and-left on HiDPI).
The position messages now carry the anchor ("top"/"bottom"/"center") + monitor
preference; the overlay resolves pixels in a single coordinate space and
applies them via winit (uncached) so a level change can't strand it.
Removes the now-unused app-side coordinate math (calculate_overlay_coordinates /
calculate_overlay_y); the vertical-anchor formula moves to the overlay as a pure,
tested `anchor_y` helper.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GpXWVxBBJvtRgkdMWasSH6
Moonshine ONNX speech-to-text backend (opt-in, now built into releases) and the Wayland overlay fixes (always-on-top + correct positioning via XWayland). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GpXWVxBBJvtRgkdMWasSH6
JRufer
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two features plus the 0.3.0 version bump.
1. Real Moonshine ONNX speech-to-text backend
The
moonshinebackend was selectable in Settings/config/docs but had no implementation — it silently ran whisper-cpp, and the "model not downloaded" guards were skipped for it, so a Moonshine selection could leave a permanently silent, unexplained pipeline.Implemented for real against the current upstream ONNX export (two graphs:
encoder_model+ a KV-cacheddecoder_model_mergedwithuse_cache_branch), greedy decode from SOT→EOT, tokenizer embedded in the binary. Model files download on demand to~/.local/share/voxctrl/models/moonshine/<size>/. Wired a download button + "not-in-this-build" notice into Settings → Engine, and fixed the two model-check guards to only skip the Whisper check when Moonshine is actually compiled in.Moonshine is now built into every release (
--features moonshineinbuild_appimage.shandrelease.yml); ONNX Runtime links statically, so binaries stay self-contained.2. Overlay always-on-top + positioning on Wayland
On native Wayland a regular client can't control its own stacking or position, so the overlay fell behind windows and was mis-placed (scale-offset from center). Fixed by running the overlay through XWayland (where KWin honors
_NET_WM_STATE_ABOVE+ positioning) and computing the position inside the overlay from its own monitor scale (the app now sends the anchor, not pre-scaled pixels). Confirmed working on KDE Plasma Wayland.3. Cleanup
Deduplicated
expand_tilde/model-dir helpers into a sharedutilmodule; optimized the Moonshine decode loop (precomputed input binding, vector KV caches, no per-step allocations/string-matching); removed the now-orphaned app-side overlay coordinate math.Testing
cargo test -p voxctrl-inference— 34 pass (default), 48 with--features moonshine.cargo check -p voxctrl-appclean; overlayanchor_yunit tests pass.npm run build,npm run test:unit(25),svelte-check(0 errors).🤖 Generated with Claude Code
https://claude.ai/code/session_01GpXWVxBBJvtRgkdMWasSH6