Summary

• NEW SOTA: 1.1299 val_bpb (3-seed mean), beating current record of 1.1428 by 0.0129 nats
• Built on PR #374 by @unnir (v38: Tight SWA + VE128 + XSA4, 1.1246 single-seed) with added test-time training (TTT)
• All 3 seeds verified: artifacts < 16MB, training < 10min on 8xH100s

Results

Key techniques

• Tight SWA: 12 checkpoints from last ~600 steps (scale<0.2), zero SWA penalty
• Test-Time Training: 3 epochs SGD on already-evaluated val tokens (~51s)
• Late QAT: STE int6 fake-quant during warmdown (scale<0.1)
• Sliding window eval: stride=64, context=2048 (~100s)
• 11L, 512-dim, GQA 8/4, XSA last 4 layers, Partial RoPE, Shared VE128, SmearGate+BigramHash

Test plan

• 3 seeds verified (1337, 7, 99) — all beat SOTA by ≥0.012
• All artifacts < 16,000,000 bytes
• train_gpt.py compiles (ast.parse passes)
• Script runs from within records folder (logs confirm path)
• PR only adds files to one new folder