Out-of-core preprocessing (qc/normalize/log1p/hvg/pca) with deterministic parallelism

[iandriver]

Summary

Implements native out-of-core (disk-backed) preprocessing for SingleRust — the flagship item
from the performance roadmap. The whole Tier-1 pipeline (QC → normalize_total → log1p → HVG → PCA)
now runs streamed against an .h5ad in bounded memory, the larger-than-RAM niche scanpy fills
with Dask (where, per scanpy's own issues, the sparse path is weak). All compute-bound passes are
parallelized with a determinism guarantee: results are bit-identical regardless of thread count.

Stacked on demo/scverse-integration, so this PR's diff is just the OOC engine + benchmarks.

What's new

OOC engine (src/backed/processing/) — streams X in row-chunks (x().iter) and writes back
via set_x_from_iter (extendable HDF5 dataset); peak memory ≈ one chunk + small accumulators:

• transformation: log1p_backed, normalize_total_backed
• qc: qc_metrics_backed (one pass; obs/var written in place)
• pipeline: preprocess_backed — fused QC+normalize+log1p (the per-cell total is reused as the
  normalization row-sum, so it's computed once)
• hvg: highly_variable_genes_backed (Seurat) — shares seurat_select with the in-memory path
• pca: pca_backed — exact covariance/eigendecomposition PCA over the HVGs (pass-1 Gram matrix,
  symmetric eigendecomp, pass-2 projection → obsm["X_pca"])

Deterministic parallelism (backed::processing::det) — naive parallel FP reduction is not
reproducible (non-associative addition + work-stealing). All reductions use a fixed-block,
ordered-merge scheme so the summation order depends only on data size + a constant block size,
never on threads/scheduling. Applied to PCA Gram, HVG sum/sumsq, and QC accumulation.

Drop-in scanpy interface — examples/sr_ooc.rs CLI (qc|normalize_total|log1p|preprocess|hvg|pca,
in-place via temp+rename) and demo/singlerust.py mirroring sc.pp.*:

scanpy	SingleRust (out-of-core)
sc.pp.calculate_qc_metrics(adata)	sr.pp.calculate_qc_metrics(adata)
sc.pp.normalize_total(adata, target_sum=1e4)	sr.pp.normalize_total(adata, target_sum=1e4)
sc.pp.log1p(adata)	sr.pp.log1p(adata)
sc.pp.highly_variable_genes(adata, …)	sr.pp.highly_variable_genes(adata, …)
sc.pp.pca(adata, …)	sr.pp.pca(adata, …)

adata is a path or a backed AnnData; the shim releases the HDF5 handle before invoking Rust.

Benchmark (demo/bench_ooc.py, 3 lanes): scanpy in-memory vs scanpy+Dask vs SingleRust OOC.

Results

Memory, 500k × 48,788 genes (48 GB / 18-core):

lane	time	peak RSS
scanpy in-memory	13.5 s	13.6 GB
scanpy + Dask OOC	35.4 s	14.5 GB
SingleRust OOC (fused)	45.0 s	2.5 GB

SingleRust OOC uses ~5.4× less memory than scanpy in-memory and ~5.8× less than scanpy+Dask,
with a chunk-bounded footprint that stays flat as cells grow (scanpy OOMs ~46 GB at 2M). Dask's
sparse OOC path gave no memory benefit (RSS ≥ in-memory) at 2.6× the time — the immaturity
scanpy's issues describe.

Correctness & determinism

Each step is unit-tested against its same-algorithm counterpart: log1p/normalize match scanpy's X
exactly; QC matches; HVG selects the same genes as in-memory HVG; PCA matches a dense
covariance-PCA reference (variance ratios exact, embedding norms match up to sign).

Determinism verified two ways:

• unit tests run QC/HVG/preprocess/PCA under 1 vs 8 rayon threads → bit-identical;
• at scale, the full pipeline on 50k real cells with RAYON_NUM_THREADS=1 vs 18 → bit-identical
  X, obs/var, obsm["X_pca"], and variance ratios.

Notes for reviewers

• Adds rayon as a direct dependency. 21 lib tests pass; clippy clean (one pre-existing dead-code
  warning unrelated to this PR).
• End-to-end PCA axes differ from scanpy only because SingleRust's Seurat HVG selects a different
  gene set than scanpy's — a pre-existing in-memory difference, not introduced here.
• Surfaced (and flagged separately) a pre-existing convert_to_array_f64 CSR densify bug; the OOC
  tests densify manually to avoid it.
• The Dask benchmark lane needs an isolated .venv-dask (anndata ≥ 0.11 / scanpy ≥ 1.11); the main
  3.9 demo env is untouched.

🤖 Generated with Claude Code