Summary

Mean val_bpb: 1.1565 (3 seeds) | Best: 1.1538 (seed 1337) | Artifact: ~15.9 MB

23 GPU runs on 8xH100 SXM5. Systematic exploration of XSA, EMA vs SWA, depth recurrence, seq curriculum, LR/WD sweep, and MLP scaling.

Techniques

• 11 transformer layers + XSA on last 4 layers
• SmearGate + BigramHash(2048) + OrthoInit
• INT6 per-row quantization + zstd-22 + FP16 tied embedding + Late-K FP16
• SWA every 50 steps (fp32 accumulation) — bf16 causes catastrophic loss
• Muon WD=0.04 + grad clip 0.3 + RoPE base 50K
• Overtone SVD init + Phase-transition residual mixing
• MLP 2.75x — sweet spot (3x exceeds 16MB with SmearGate at 11L)

Results (3 seeds)

Key Findings from 23 Runs

• EMA(0.997) causes 0.14 BPB quant gap — SWA far better for our stack
• 11L MLP 3x exceeds 16MB with SmearGate+BigramHash
• SmearGate removal loses more than MLP 3x gains — bigram context matters
• XSA needs GQA-compatible v expansion (repeat_interleave, bug found and fixed)
• Seq curriculum doesn't work — SWA checkpoint incompatibility across seq lengths
• Depth recurrence works but dim=640 too narrow; dim=768+ exceeds 16MB
• Higher LR (0.03) improves BPB but worsens compression (larger weights)
• Late QAT (75%) reduces quant gap (0.023 -> 0.006) but fewer steps

Run command

NUM_LAYERS=11 XSA_LAST_N=4 BIGRAM_VOCAB_SIZE=2048 \
TRAIN_SEQ_LEN=2048 TRAIN_BATCH_TOKENS=524288 MLP_MULT=2.75 \
MATRIX_LR=0.025 SCALAR_LR=0.025 TIED_EMBED_LR=0.035 \
MUON_MOMENTUM=0.99 MUON_WEIGHT_DECAY=0.04 WARMDOWN_ITERS=3000 \
SWA_ENABLED=1 SWA_EVERY=50 ROPE_BASE=50000 EVAL_STRIDE=64 \
python3 -m torch.distributed.run --standalone --nproc_per_node=8 train_gpt.py

Test plan

• Runs reproducibly on 8xH100 SXM in under 10 minutes
• Artifact under 16 MB (15.87-15.99 MB)
• 3-seed validation (mean 1.1565, std 0.0028)
• Sliding window eval completes within 10 minutes

Built with Claude Code