Entry for the OpenAI Parameter Golf competition: train the best language model that fits in 16MB and trains in 10 minutes on 8xH100 SXM, scored by bits-per-byte (bpb) on the FineWeb validation set.

Submission: PR #379

Built on PR #374's SOTA stack (1.1246 bpb) with a novel post-training quantization improvement.

Standard int6 quantization uses row-wise absolute max for clipping. GPTQ-lite searches 5 clip percentiles per weight matrix (100%, 99.9%, 99.5%, 99%, 98%) and selects the one minimizing reconstruction error ||W - dequant(quant(W))||. This reduces quantization degradation at zero training cost. Zero new parameters, ~10s extra quantization time.

All experiments on 8xH100 SXM via Modal. Sliding window bpb (stride=64).

• GPTQ-lite: Per-layer clip percentile search reduces quantization degradation. Simple, fast, effective.
• FA3 from source: Building flash_attn_interface from the hopper/ subdirectory of Dao-AILab/flash-attention. Required for competitive step times.
• Leaderboard scraping: Built leaderboard.py to track 200+ PRs and extract technique frequency. Critical for strategy.
• Trusting community data: Other participants' validated results (3-seed) are reliable data points. Don't re-run what's already proven.

• Differential Attention (4-call): 0.046 bpb better per step, but 4 FA calls = 154-322ms/step. Net negative. The architecture is right but the implementation is too expensive.
• Even/odd head differential: Zero-cost DiffAttn variant. Crashed 3 times with FA3 + torch.compile + DDP incompatibilities.
• Self-distillation TTT: KL-divergence adaptation post-training. Slightly negative (-0.0003 bpb). The KL constraint was too weak to improve on the already well-trained model.
• ResFormer value residual: Zero params, zero cost, but neutral at 26M scale. May help at larger scales.
• Depth recurrence: 50+ competition attempts by various participants, best result 0.044 behind SOTA. Unique layers beat shared layers at 16MB.
• NorMuon: Training instability without warmup. Noisy initial EMA + weight decay = destructive feedback loop.
• Too many changes at once: Stacking 6 changes simultaneously regressed badly. Always bisect.

• Quantization is the biggest untapped gap: SOTA loses 0.007 bpb to int6 quantization (~30% of total improvement over baseline). Most competition focus is on training quality.
• Eval-time compute is unconstrained: The 10-min limit applies to training only. Post-training optimizations (GPTQ, TTT, SWA averaging) have unlimited time.
• FA3 vs FA2 matters: ~15% step time difference = ~0.005 bpb from more training steps alone.
• The competition converges fast: With 200+ participants, simple technique combinations are found within hours. Novel contributions need to be non-obvious.

See RUN_GUIDE.md for running on your own 8xH100 node.

# Clone and run
git clone https://github.com/dannywillowliu-uchi/parameter-golf-entry.git
cd parameter-golf-entry
GPTQ_ENABLED=1 torchrun --standalone --nproc_per_node=8 pr374_train_gpt.py

• Competition: OpenAI Parameter Golf (March 18 - April 30, 2026)
• Constraint: 16MB artifact, 10 min on 8xH100 SXM
• Metric: Bits-per-byte (bpb) on FineWeb validation set
• Leaderboard: parameter-golf.github.io
• Current SOTA: 1.1246 bpb (PR #374)
• Our position: 1.1257 bpb, rank ~3 (PR #379)