Summary

Mean val_bpb = 1.1488 (3-seed verified, std=0.0006)

Builds on @baudrillardsgh0st's technique stack (PR #194). Two hyperparameter changes informed by 30+ experiments:

1. Muon WD=0.04 (vs 0.038) — improves int6 quantization quality
2. SWA_START_FRAC=0.4 (vs 0.5) — 33% more checkpoint diversity for smoother weight averaging

Std: 0.0006 (4× tighter than SOTA's 0.0024)

Key finding

SWA_START_FRAC=0.4 captures ~33% more checkpoints, producing smoother weight distributions that survive int6 quantization better. Combined with WD=0.04, the quantization gap is minimized.

Submission checklist

• 3-seed verification (mean=1.1488, std=0.0006)
• All artifacts < 16MB (max 15.3MB)
• Wallclock < 600s on 8×H100
• Train logs included (3 seeds)
• Reproducible train_gpt.py included
• README with detailed explanation