Record: 0.9076 BPB — 10L + N-gram Backoff + Matrix LR 0.03

[bigbag]

Summary

val_bpb = 0.9076 (seed 42, additional seeds pending) | 15.32 MB | 8xH100 SXM, 600s

Single change from PR #802: MATRIX_LR=0.03 (was 0.02). Discovered through systematic hyperparameter screening (74 experiments across steps 10-12).

Results

Seed	Steps	ms/step	Pre-quant BPB	N-gram BPB	Artifact
42	6,693	89.6	1.1528	0.9076	15,320,749

Key Change

MATRIX_LR=0.03 vs PR #802's default 0.02.

Architecture

• 10L, 512d, GQA 8H/4KV, MLP 3x LeakyReLU(0.5)²
• BigramHash(4096), SmearGate, Value Residual, Gated Attention
• Mixed int5-MLP/int6-attn + zstd-22, EMA(0.997)

Eval: Multi-Order N-gram Backoff (from PR #802)

• Score-first backward-looking n-gram cache (orders 2-7)
• Entropy-adaptive alpha mixing
• 138s eval time (well within 600s budget)

Reproduction

MATRIX_LR=0.03 torchrun --standalone --nproc_per_node=8 train_gpt.py

Test plan

• 8xH100 SXM, seed 42
• 3-seed validation (seeds 1337, 2024)
• Artifact ≤ 16MB (15.32 MB)
• Training ≤ 600s
• Eval ≤ 600s (138s)

Based On

• PR 10L + Multi-Order N-gram Backoff (0.9123 BPB) #802: 10L + Multi-Order N-gram Backoff (0.9123 BPB)
• Systematic screening: 74 experiments across RTX4500 + 8xH100

🤖 Generated with Claude Code