Summary

• val_bpb: 1.1518 (int6 sliding window, stride=64, seed 1337)
• 11-layer GPT, 26.8M params, 15.2MB artifact (int6+zstd-22)
• Trained in 600s on 8×H100 SXM (9,906 steps at 60.6ms/step)

Key Techniques

• Per-row int6 quantization (MLP + attention) + zstd-22 compression
• 3× MLP expansion (hidden=1536) with relu² activation
• SmearGate: learned token-predecessor blending at input
• BigramHash embedding: 2048-bucket hash table (dim=128) for token-pair context
• U-Net skip connections: encoder→decoder with learned per-dimension weights
• Muon optimizer with WD=0.04, momentum warmup 0.92→0.99
• SWA: 7 snapshots every 200 steps during warmdown
• Sliding window eval (stride=64) as primary score
• TTT LoRA eval also included (1.1535 BPB)

Eval Results (seed 1337)

Submission Checklist

• Artifact < 16MB (15,202,515 bytes)
• Trains in < 10 min on 8×H100 (600s)
• Eval < 10 min (sliding 97s + TTT 83s = 180s)
• train_gpt.py compiles (1191 lines, under 1500 limit)
• README.md with technique descriptions
• submission.json with metadata
• Training log included

Note on Seeds

Single seed submission. The improvement margin over the current #3 (1.1502) is modest, but the submission includes independently developed techniques (U-Net skips, seq_len=1024 tradeoff for more layers/steps) that may be of interest to the community. Additional seeds available on request.

Differences from Existing Submissions

Developed independently from PR #162 (raahilshah). Key architectural differences:

• 11 layers (vs 9) with seq_len=1024 (vs 2048) — more layers + more steps
• U-Net skip connections with learned weights
• BigramHash 2048 buckets (vs 4096)
• SWA every 200 steps (vs 50)
• Includes TTT LoRA as alternative eval