11L, XSA all layers, partial RoPE 16/64, LN scale, VE128 (layers 9,10),
LeakyReLU(0.5)² activation, BigramHash(2048), INT6+zstd-22.
Legal score-first TTT: 32K chunks, all blocks, SGD(0.002,mom=0.9), 3ep.
Base: PR openai#503 (EthanYangTW) + LeakyReLU² from openai#518/openai#549 + SGD from openai#549.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Multiple top PRs (openai#535, openai#549, openai#569) demonstrate -0.0015 to -0.003 bpb
from this change. LeakyReLU preserves gradient flow through negative
pre-activations while maintaining the sparsity/gating benefits of
squaring. At 22M params, dead neurons from hard ReLU are expensive.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Four novel improvements over PR openai#549 (1.1194 BPB) base:
- Full GPTQ quantization with Hessian-guided error compensation
- Soft-round QAT with tanh-based temperature annealing
- LoRA-based test-time training (rank-8 adapters on Q/K/V/O)
- Entropy-coded compression (Huffman+LZMA adaptive selection)

Made-with: Cursor

Track: 10min_16mb
Based on: PR openai#549 (LeakyReLU+ParallelMuon), PR openai#606 (Soft-Round+AdamW TTT), PR openai#609 (XSA-all+Full GPTQ)

Changes from SOTA (openai#549):
- XSA on all 11 layers (was 4)
- Soft-Round QAT with tanh-based differentiable rounding (alpha 1->16)
- Full GPTQ with Hessian-aware column-reordered Cholesky error compensation
- MHA 8/8 (was GQA 8/4)
- MLP 3.5x expansion (1792 hidden, was 3.0x/1536)
- BigramHash vocabulary 8192 (was 2048)
- AdamW TTT with grouped LR and cosine schedule (was SGD)
- Early QAT threshold 0.5 (was late 0.15)
- Selective ±1 magnitude pruning to hit size target

Run 0: PR openai#549 UNMODIFIED (merged SOTA 1.1194, verified 3-seed)
Run 1: PR openai#549 + TTT_ENABLED=1 + TTT_LR=0.0005 (2 lines changed)

Both have FA3→FA2→SDPA fallback for non-Hopper GPUs.
Following retro: one change per run, baseline first.

Expected: Run 1 should achieve ~1.094-1.104 (beats 1.1144 target).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Documents merged SOTA of 1.1194 (PR openai#549, LeakyReLU² + Legal TTT + Parallel Muon),
confirmed technique deltas, enforcement ruling on GPTQ calibration, and the path
forward to beat 1.1144.

https://claude.ai/code/session_01U3LXGzTkedd9ZcHF2qgW7d

Run 0: PR openai#549 UNMODIFIED (merged SOTA 1.1194, verified 3-seed)
Run 1: PR openai#549 + TTT_ENABLED=1 + TTT_LR=0.0005 (2 lines changed)

Both have FA3→FA2→SDPA fallback for non-Hopper GPUs.
Following retro: one change per run, baseline first.

Expected: Run 1 should achieve ~1.094-1.104 (beats 1.1144 target).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Case study: reordering training shards by model difficulty (hardest
first) gives -0.0033 BPB improvement over sequential ordering. Zero
architecture changes, zero compute cost, ten lines of code.

Key finding: token-level statistics (KL divergence) find 0.0009 range
across shards. Model perplexity finds 0.0475 range -- 100x more
variation. The two metrics are uncorrelated (r = -0.056).

3-seed validated on PR openai#549 (merged openai#1):
  Seed 1337: 1.1217 -> 1.1183 (-0.0034)
  Seed 42:   1.1222 -> 1.1181 (-0.0041)
  Seed 2025: 1.1221 -> 1.1198 (-0.0023)
  Mean:      1.1220 -> 1.1187 (-0.0033)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>