Int6 GPTQ-lite + LeakyReLU(0.5)^2 + EMA + 11L MLP3x

[zeytx]

Summary

• 11 transformer layers (vs 9 baseline) with 3x MLP expansion
• LeakyReLU(0.5)^2 activation (~0.003 BPB improvement over ReLU^2)
• Int6 per-row GPTQ-lite quantization with clip search + zstd-22 compression
• Late QAT via STE (activates when LR scale < 0.15)
• EMA weight averaging (decay 0.997)
• GQA: 8 query heads, 4 KV heads

Results

• val_bpb: 1.1807 (pre-quantization, 8xH100 SXM, 9881 steps in 10 min)
• Compressed model size: ~3.9 MB (well under 16MB limit)
• 26.5M parameters

Run command

torchrun --standalone --nproc_per_node=8 train_gpt.py

Requires zstandard package (pip install zstandard).