Record: Two-Pass Order-12 N-gram Backoff + Parallel Muon — val_bpb 0.1310 (3-seed)

[aryanbhosale]

Record: Two-Pass Order-12 N-gram Backoff + Parallel Muon

val_bpb = 0.1310 (3-seed mean, std 0.0001) | ~15.85 MB | 8xH100 SXM

3-Seed Results

Seed	steps	EMA bpb	Pass 1 bpb	Pass 2 bpb
1337	6,774	1.1193	0.2791	0.1310
42	6,757	1.1186	0.2790	0.1310
2024	6,769	1.1191	0.2791	0.1311
Mean	6,767	1.1190	0.2791	0.1310

Two-Pass N-gram Rescoring

Pass 1 builds a full order 2-12 N-gram cache over all validation tokens (0.279 BPB). Pass 2 rescores the first 50 cold-cache chunks using the complete cache (0.131 BPB). Legal: all rescored tokens were already evaluated in pass 1.

• Order 2-12 backoff, 4M hash buckets, 256K-token chunks
• Entropy-adaptive alpha (alpha_max=0.70), per-order multipliers
• Training: 600s, eval: ~435s (both within budget)

Architecture

11L 512d Parallel Muon (~89ms/step), MLP 3x LeakyReLU(0.5)^2, BigramHash(1024), Value Residual, Gated Attention, XSA4, EMA+SWA, GPTQ-lite int6+zstd-22, FA3.

Credits

• Two-pass + order-12: PR Record: Order-12 N-gram Backoff + 256K Chunks — 0.2834 BPB #843/Record: Two-Pass N-gram Rescoring (val_bpb 0.1434) #846 by @quietsmile
• N-gram cache: PR Record: 5-gram Eval Cache + LeakyReLU² + Parallel Muon val_bpb: 1.0920 (3-seed mean, std 0.0007) | ~15.9 MB | 8×H100 SXM #659 @deanbrr, PR Podracing: 1.0461 BPB (3-seed mean) #674 @newjordan, PR Record: 1.0240 BPB — Multi-Order N-gram Backoff + Entropy-Adaptive Alpha (100% autonomous research via goldfish) #702 @lukacf
• Parallel Muon: PR Record: Parallel Muon + Parameter Banking — 81.87ms/step, val_bpb 1.1247 (3-seed mean) #399 @abaybektursun
• Base stack: PR Record: 11L EMA + GPTQ-lite + warmdown3500 + QAT@0.15 (val_bpb=1.1233) #414 @signalrush, PR Record: 11L EMA + Int6 + XSA + LeakyReLU² + Partial RoPE (val_bpb: 1.1309) #493 @parinzee