val_bpb: 1.1720 | 19.4 MB (unlimited compute) | 1xA6000, 9500 steps, 14.5hr

Summary

• Value Residual (ResFormer, arXiv:2410.17897): caches layer-0 V vectors, mixes into subsequent layers via learnable scalars. -0.015 BPB, 22 params added.
• Gated Attention (arXiv:2505.06708): per-head sigmoid gate after SDPA, eliminates attention sinks. -0.003 BPB, ~37K params added.
• Techniques stack additively (-0.0172 combined), validated via controlled ablation on 9L baseline.
• Full community meta-stack: 11L MLP3x + SmearGate + BigramHash(2048) + OrthoInit + WD0.04 + XSA(4) + EMA(0.997) + Partial RoPE + LN Scale + Logit Softcap.
• Both techniques independently adopted by 5+ community submissions, including a record-tier entry (1.1101 BPB).

Ablation (9L v1024, 1000 steps, 131K batch, 1x3090)

Production Results

Files

• README.md — full writeup with ablations and reproducibility command
• submission.json — metadata
• train_gpt.py — training script
• train.log — complete training log