Summary

Non-record submission exploring recursive weight sharing — a novel approach where 6 unique transformer blocks are looped 2x each, providing 12 effective layers of depth with only 6 blocks of stored parameters. The freed parameter budget enables MLP 4x expansion, which is the primary quality driver.

• val_bpb: 1.1478 (sliding window stride=64) | 15.19 MB | 8xH100 SXM, 600s
• 28.2M params, 4,396 steps at 136.5ms/step
• Full pipeline: Muon + SWA + Late QAT + Training Replay + Self-Distillation + EMA

Key Insight

MLP 4x gives ~2% relative BPB improvement over MLP 3x, but doesn't fit in 16MB with 12 unique layers. Recursive weight sharing (6 unique x 2 loops) fits it in 15.19 MB. The weight sharing is the compression technique; the MLP 4x is the quality lever.

Architecture

• 6 unique blocks x 2 loops = 12 effective depth
• dim=640, 10 heads, 5 KV (GQA), head_dim=64
• MLP 4x (hidden=2560), relu-squared
• Orthogonal loop positions, U-Net skips, SmearGate, BigramHash, VE128, XSA last 2

Results

No TTT on Validation Data

All training uses training data only. Late replay buffers training batches. Self-distillation uses EMA teacher on training data.

Test plan

• 8xH100 SXM, 600s
• Artifact under 16MB (15.19 MB)
• No TTT on validation data
• Post-quant int6 roundtrip verified
• Sliding window eval (stride=64)