Summary

• 11-layer parameter-banking GPT with XSA on 7 layers, BigramHash(2048×96) + TrigramHash(1024×128), ValueResidual, ValueEmbedding at layers 5/9/10, LeakyReLU(0.5)², int6+lzma compression, and legal score-first TTT
• Best run: val_bpb=1.12265 (legal TTT), val_bpb=1.12468 (sliding window stride=64)
• 3-seed mean: val_bpb=1.12327 ± 0.00082
• Artifact: 15,944,685 bytes (under 16,000,000 cap)
• Training: 600s on 8×H100 SXM (~90.8 ms/step, 6,487 steps)
• Evaluation: <600s (quant roundtrip ~90s + sliding window ~300s + TTT ~475s)

3-Seed Evidence

Submission Checklist

• One new folder under records/track_10min_16mb/
• Included README.md
• Included submission.json
• Included train_gpt.py
• Included train.log
• Artifact ≤ 16,000,000 bytes (15,944,685)
• Training completes in <600s on 8×H100 SXM (600,069 ms)
• Evaluation completes in <600s on 8×H100 SXM
• Legal score-first TTT (no two-pass full-rescore leakage)
• No tokenizer/dataset modifications
• 3-seed statistical evidence provided
• No other files modified

Key Techniques

• Flash Attention 3 (Hopper kernel) for ~90ms/step
• Parallel Muon optimizer with parameter banking and batched Newton-Schulz
• Cross-Sequence Attention (XSA) on last 7 layers
• BigramHash + TrigramHash n-gram hash embeddings
• Value Residual (ResFormer-style) connections
• Value Embedding token identity reinjection
• SWA + EMA + Late QAT for quantization-friendly convergence
• Legal score-first TTT with SGD (lr=0.002, 4 epochs, all blocks unfrozen)