Summary

val_bpb: 1.1477 (sliding window, stride=64, post int5/int6 quantization roundtrip)
Artifact size: 15.94 MB

Novel Contributions

• Quantization-Aware Training (QAT): STE with int5 MLP / int6 attention during training — reduces quant gap from ~0.016 to ~0.005 BPB
• Backout: Learned residual subtraction (λ·x_mid) from final output
• U-Net Skip Connections: 5+5 encoder-decoder with learned per-dim skip weights
• SVD Embedding Init: Spectral decay 1/√k for tied embeddings

Shared Techniques

BigramHash(10240), SmearGate, Simple Muon + compiled NS5, SWA every 50 steps, orthogonal init, 8% magnitude pruning, zstd-22

Run Details

• 6457 steps in 10 min, 92.9ms/step avg
• 8xH100 SXM (RunPod)
• seed=42, single run

See README.md for full details.