feat(cala): Phase 5 — browser app scaffold + four-worker runtime#139
Merged
Conversation
Phase 5 opener. Adds the `bindings/` module on top of the pure-Rust core: a natively-testable JSON config bridge and the `#[wasm_bindgen]` veneer (`AviReader`, `Preprocessor`, `Fitter`, `MutationQueueHandle`, `SnapshotHandle`) that apps/cala workers will consume. - Config types (`PreprocessConfig`, `FitConfig`, `ExtendConfig`, `RecordingMetadata`, grayscale/motion enums, `ComponentClass`) pick up cfg-gated `serde` derives. New `serde` feature that `jsbindings` and `pybindings` both enable — single source of truth per tuning knob, no JS-side magic numbers. - `io::OwnedAviReader` owning variant of `AviUncompressedReader` so WASM can hold the file bytes across the JS/WASM boundary. Shared grayscale decode helper keeps borrowed and owning paths byte- identical. - 9 new config-JSON roundtrip tests + 5 owned-AVI-reader parity tests, all written before the binding code (§4.1). Compiles on both `x86_64-unknown-linux-gnu` and `wasm32-unknown-unknown --features jsbindings`; clippy + rustfmt clean. Total suite: 347 native tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 5 runtime package — per CALA_DESIGN §7 — lands its first piece: the SAB-backed single-producer/single-consumer ring channel used for frame data between the decoder, fit, and extend workers. - Package scaffolding: package.json, tsconfig.json, vitest.config.ts, README, barrel index.ts (follows packages/compute + packages/io structure). - `src/channel.ts`: SabRingChannel with writeSlot / readSlot / tryWrite / waitRead / stats. Every tuning knob (slot bytes, slot count, timeouts) is a ChannelConfig field — no literals in the channel hot path. - Tests written first per §4.1: FIFO order, ring wrap, tryWrite backpressure, byte-level payload parity, config validation. - Mutation queue / snapshot / event-bus / orchestrator modules left as TODO stubs documenting the full §7 surface; they land in tasks 16–18. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Turns the Rust WASM surface from task 12 into a consumable workspace package. Mirrors @calab/core's wrapping of crates/solver — same lazy init pattern, same deep-import block, same test shape. - New workspace package `@calab/cala-core` with wasm-adapter.ts. Lazy, idempotent `initCalaCore()` boots the WASM module once and installs the panic hook on first call. Re-exports AviReader, Preprocessor, Fitter, MutationQueueHandle, SnapshotHandle. - Root `build:wasm` now runs both solver and cala-core builds via `build:wasm:solver` and `build:wasm:cala` sub-scripts. - ESLint `no-restricted-imports` extended to block `**/crates/cala-core/pkg/*` imports anywhere but the adapter, same guardrail applied to the solver pkg. - `.prettierignore` updated to skip `crates/cala-core/pkg/` and `crates/cala-core/target/`, matching the solver exemptions. - 4 adapter tests (module-level singleton via `vi.resetModules()`): idempotent init, single-shot panic hook install, concurrent-caller promise sharing, public re-export surface. Real WASM execution is exercised at Phase 5 exit in the browser (task 25). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 5 decoder worker needs random-access grayscale frames from a user-dropped file. This lands the abstraction and the one format we ship in v1. - `FrameSource` interface (design §10): `meta()`, `readFrame(n, method)`, `close()`, plus `FrameOutOfRangeError` and `FrameSourceParseError`. TIFF / compressed AVI / MP4 decoders plug in here without the pipeline caring which parser produced a frame. - `openAviUncompressed(File)` and `openAviUncompressedFromBytes(Uint8Array)` implementations — thin JS veneer over `@calab/cala-core`'s WASM `AviReader`. Parses the RIFF container once on open (O(frame_count) scan), then O(1) random-access reads. Reads the full file in v1; `File.slice()` streaming for huge files is deferred to a future `avi-uncompressed-streaming.ts` that reuses the same contract. - 9 new tests (mock `@calab/cala-core`): meta forwarding, readFrame argument forwarding, FrameOutOfRangeError for invalid indices, constructor/read WASM error wrapping, close() idempotency + free lifecycle, and `initCalaCore()` await on the File path. Real WASM round-trip lands at Phase 5 exit in the browser (task 25). - `@calab/io` picks up `@calab/cala-core` as a workspace dep. Vitest alias updated so the mock resolves cleanly in Node. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Ports crates/cala-core/src/extending/mutation.rs to TypeScript for the runtime orchestration layer. Single-threaded VecDeque-equivalent semantics; cross-worker SAB backing lands with the orchestrator in task 18. - MutationQueue with drop-oldest overflow, FIFO drain, bigint drops counter matching the Rust u64 — all parameters sourced from MutationQueueConfig, no magic numbers in the class body. - PipelineMutation discriminated union (register / merge / deprecate) mirroring the Rust variants field-for-field. DeprecateReason and ComponentClass string unions for the corresponding Rust enums. - Tests written first (§4.1): capacity assertion, FIFO, drop-oldest on overflow, drainAll, snapshotEpoch helper, DeprecateReason round- trip, and one byte-for-byte Rust-parity test mirroring a scenario from tests/extending_mutation.rs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 5 runtime gains the two cross-worker coordination surfaces from design §7.2 and §9.2: - `SnapshotProtocol` handles extend→fit snapshot requests with correlation ids, fit-side polling / ack publication, and ack-timeout diagnostics. Single-threaded in-memory transport for now; SAB-backed swap lands with the orchestrator in task 18. - `EventBus` carries the six `PipelineEvent` variants (birth, merge, split, deprecate, reject, metric) plus `FootprintSnap` payloads (sparse pixel_idx / value pairs, design §9.3) from fit to archive. Drop-oldest under pressure, drops counter for dashboard metrics. - All tuning knobs (ring capacities, timeouts, subscriber caps) live on the config structs — no literals in the class bodies. - Tests written first (§4.1): round-trip + timeout + capacity for the snapshot protocol; publish/subscribe/unsubscribe/drop-oldest/close + FootprintSnap byte parity for the event bus. Orchestrator module stub untouched — it lands in task 18 wiring all three channels (SAB frames, mutation queue, snapshot protocol, events) to real workers. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Copies the app-template structure into apps/cala and wires in every
dependency the CaLa workers will need: @calab/{cala-core, cala-runtime,
compute, core, io, ui} aliases in vite and tsconfig, vite-plugin-wasm
on both the main and worker plugin chains.
- vite.config.ts sets `Cross-Origin-Opener-Policy: same-origin` and
`Cross-Origin-Embedder-Policy: require-corp` on the dev and preview
servers so SharedArrayBuffer is available for the SAB-backed
runtime channels landed in tasks 15-17. Addresses design §13's
"test COOP/COEP early before it blocks UI work."
- `scripts/verify-sab.mjs` — smoke check that boots Vite, fetches `/`,
and asserts both headers are set to the expected values. Runnable
via `npm run verify-sab -w apps/cala`. All configurable values
(timeouts, header names, expected values) are const-named at the
top — no literals scattered through the fetch path.
- README calls out the GitHub Pages limitation: the host does not
support custom response headers, so the production SAB story is a
coi-serviceworker deliverable for Phase 6+. Phase 5 exit only
requires local dev end-to-end.
- `apps/cala` added to the root `typecheck` target; eslint rule for
Node globals in scripts now covers `apps/*/scripts/**/*`.
`npm run dev -w apps/cala` boots the placeholder shell; `npm run
verify-sab -w apps/cala` confirms SAB headers are live;
`npm run build -w apps/cala` produces a 172 KB (45 KB gzipped)
bundle.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Ties channels, mutation queue, snapshot protocol, and event bus together into a single RuntimeController the app layer drives. - `createRuntime(cfg)` spawns four workers via caller-provided factories (decode-preprocess, fit, extend, archive), wires SAB channels between them, and waits for all four to ack `ready` within `startupTimeoutMs`. - Epoch tracker owned by the orchestrator — increments only on mutation-applied acks from fit, matching the Rust `FitPipeline::epoch` semantics (frame-processed does not bump). - Lifecycle states `idle -> starting -> running -> stopping -> stopped` with `error` as the terminal state on any spawn / timeout / worker-crash; `onStatus` + `onEvent` subscription APIs. - Stats aggregator pulls every drop counter (frame channel full, mutation queue overflow, event bus drop-oldest, snapshot ack timeouts) from the underlying modules so the dashboard can render them. - `worker-protocol.ts` codifies the orchestrator-worker message union so workers in tasks 21-23 import the types directly. - Tests (section 4.1) use a fake-worker harness — no real Worker instances — covering ready-handshake timeout, lifecycle transitions, epoch monotonicity + exact-once-per-mutation semantics, stats aggregation, graceful + hard shutdown paths, and onEvent subscription. Phase 5 runtime surface is now complete; workers plug into this API in tasks 21-23. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wire W1 — opens an AVI via @calab/io, builds a Preprocessor from @calab/cala-core, runs the decode→preprocess→SAB-write loop, and handles init/run/stop lifecycle with throttled frame-processed heartbeats. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wires apps/cala's file-drop UI and run-control lifecycle: adds a createStore data-store, a RuntimeController wrapper with stub factories, ImportOverlay (.avi drag-drop + metadata + start button), and a CaLaHeader run-state pill. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extend ships as a heartbeat-only stub so the orchestrator's 4-worker lifecycle is exercisable end-to-end; the real snapshot/segmentation loop lands post-Phase 5. Archive is full: drop-oldest event log + per-name metric snapshot, served via new request-archive-dump / archive-dump protocol variants (design §9.2, §10). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wires W2 to Fitter + frame channel consumer + MutationQueue/Handle + SnapshotProtocol + EventBus; every stride (heartbeat / snapshot / mutation-drain cap / event-bus capacity) is a FitConfig-overridable DEFAULT_* constant per the no-magic-numbers rule. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wires the dashboard to real worker output: ArchiveClient + dashboard store + SingleFrameViewer replace the task-20 placeholder, and run-control now spawns real workers and forwards W1 preview frames. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Path B (Node vitest harness) — Playwright + chromium downloads were blocked by the task 25 sandbox, so the E2E pipes real AVI bytes through the real W1/W2/W4 worker modules via the existing WorkerHarness pattern instead of a real browser. Fixture: .test_data/anchor_v12_prepped.avi (448x288, 10 frames run). Observed: 5 decode heartbeats, 5 fit heartbeats, 2 preview frames, 2 archived metric events, 0 worker errors, 18 ms end-to-end. Opt-in via `npm run test:e2e:cala`; default `npm test` continues to pass on 401 tests across 45 files. Real browser E2E + real WASM + coi-serviceworker remain Phase 6+. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Seven fixes surfaced during the first real in-browser run: - App: gate viewer on runState (not file presence) so the ImportOverlay + Start button stay visible until a run begins. - ImportOverlay: surface state.errorMsg alongside local errors so failures in the starting→error transition don't get lost on remount. - run-control: send a default RecordingMetadata JSON (pixel_size_um = 2.0 µm, standard UCLA miniscope) — Rust side requires the field. - orchestrator: strip non-clonable source.frameSourceFactory from the worker init config; postMessage structured-clone would crash. - orchestrator: forward fit/extend 'event' outbound messages to the archive worker — archive never heard them otherwise. - orchestrator: mirror snapshot-ack to extend (not just fit) so the extend stub's snapshot-latch actually fires. - extend: emit a metric on any pending ack, not only on strictly advancing epoch — live runs with no mutations keep epoch at 0. All 50 apps/cala + 79 cala-runtime tests still green. typecheck + lint clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- crates/cala-core/src/io/mod.rs: cfg-gate decode_grayscale_f32 re-export
to match its only caller (wasm bindings). Keeps --features native-cli
-D warnings clean.
- crates/cala-core/tests/bindings_config_json.rs: #![cfg(feature =
"serde")] so the test file compiles out under native-cli builds that
don't enable the config_json module.
- apps/cala/src/workers/__tests__/{archive,extend}.worker.test.ts:
prettier --write (flagged by format:check in CI).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Matches the existing `crates/solver/pkg/` convention — built `.d.ts`, `.js`, `.wasm`, `.wasm.d.ts`, and `package.json` land in the repo so typecheck resolves the `@calab/cala-core` wasm-adapter import without needing `wasm-pack` in the CI `check` job. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 5 of CaLa — the browser-native streaming calcium-imaging demixing app (port of Raymond Chang's Python
cala). Lands the app scaffold, the four-worker runtime (W1 decode+preprocess → W2 fit → W3 extend stub → W4 archive), the SolidJS UI (file drop → run control → single-frame viewer → dashboard), and an E2E run on a real AVI.What ships
New crates / packages:
New app:
End-to-end verification:
Live-session fixes (last commit): structured-clone hazard in worker init config, missing `pixel_size_um` default, event-forwarding gap between orchestrator and archive worker, snapshot-ack routing to extend, overlay/error UX gates.
What's deferred to Phase 6+
Test plan
🤖 Generated with Claude Code