BPB (bits per byte) measures how many bits your model needs to encode each byte of text. A model that perfectly predicts every next character needs zero bits — it already "knows" what comes next. A model with no understanding of language needs the maximum (~8 bits per byte).

A model's cross-entropy loss IS its compression rate. Shannon proved in 1948 that prediction and compression are mathematically equivalent — a model that predicts well compresses well, and vice versa. The competition measures the compression side of that equivalence.

This framing matters because it legitimizes approaches beyond pure language modeling: sliding window eval improves compression by giving more context. Backward-looking TTT adapts to already-scored tokens for better compression. These are valid compression strategies.

There is no separate held-out test set — the FineWeb validation set is the fixed evaluation target. However, val tokens cannot be stored in the artifact (paid prefix ruled out), and pre-eval adaptation on val data is also ruled out. Only backward-looking TTT (adapting on tokens already graded) is permitted.

"Tokenizer-agnostic" means BPB normalizes across tokenizers. A bigger vocabulary uses fewer tokens but more bits per token — BPB cancels that out, measuring compression of raw bytes regardless of how they're tokenized.

Three failure patterns. (1) Throughput cost exceeds quality gain. In a 600s budget, anything adding >10% step overhead needs >10% per-step improvement to break even. QAT (#236: 115ms vs 67ms baseline), NorMuon (#236: 110ms), and MTP (#212, #236: 86ms) all fail this test. Partial reversal: #1031 uses MTP as auxiliary-only training signal (2-head, weight=0.1, discarded at export) with -0.0037 BPP claimed at zero artifact/eval cost — a different usage pattern (1 seed, unvalidated). (2) Mechanism redundancy. Stacking two techniques that extract the same signal yields diminishing returns — TTT+XSA underperforms XSA-alone (#290 vs #265), error-guided TTT doesn't improve over uniform TTT (#296), EMA without XSA hurts (#201). (3) Regime incompatibility. Techniques optimized for int6 break under different weight representations — the standard stack (XSA, SmearGate, WD, EMA/SWA, TTT) all fail on ternary (#367), and recurrence amplifies quantization error 900× (#363).

• 12 layers at seq2048 (slower steps cancel extra capacity) — Non-record: 12L Int5-MLP + Int6-Attn mixed quantization, val_bpb=1.1541 #219's 12L at seq2048 runs at 107ms/step, fitting only ~5,590 steps. Result: 1.1541 vs 11L's 1.1326. However, 12L Int5-MLP + SmearGate + BigramHash + SWA (val_bpb 1.1433) #76 shows 12L at seq1024 (59ms/step, ~9000 steps) reaches 1.1468 — the tradeoff depends on sequence length.
• Late QAT at 12L is step-budget-dependent. @saml212's #332 found that at 12L, Late QAT added ~7ms/step overhead, costing ~770 training steps. At 11L, those steps would cost ~7ms each — but at 12L, each step is already more expensive and step count is already lower, so the overhead-to-gain ratio worsens. Result: Late QAT was dropped from the 12L submission. Takeaway: the same technique's cost-benefit flips depending on step time and total step count. Always re-evaluate overhead techniques when changing layer count.
• Int5-MLP tradeoff is layer-dependent — At 11L, Record: 11L Int6 + SmearGate + Batch Optimization (val_bpb=1.1400) #236 found int5 quant penalty (0.029) outweighs artifact savings. But at 10L, Record: 10L Int5-MLP + BigramHash(10240) + SWA(0.4) + WD=0.04 (val_bpb=1.1428, mean 3 seeds) #180 used int5 to fund BigramHash(10240) (previous official SOTA at 1.1428; now superseded by Record: 11L EMA + GPTQ-lite + warmdown3500 + QAT@0.15 (val_bpb=1.1233) #414 at 1.1228). New data from Non-record: 27M params at Int5 QAT / train larger, quantize harder (val_bpb=1.1418) #469: all-int5 on a larger model (d=576, 27M params) with early QAT activation (threshold 0.50 = ~1700 adaptation steps) reaches 1.1418 (1-seed) — validating the "train larger, quantize harder" principle.
• Larger vocabularies + fewer layers — Record: Vocab 4096 + MLP 3x + Sliding Window Eval (mean val_bpb=1.1642, 3 seeds) #123 (vocab 4096, 8L) at 1.1642 and Record: SP4096 + Int6 QAT + NorMuon (val_bpb=1.2012) #200 (SP4096, 9L) at 1.2012 both underperform 11-Layer Int6 + WD=0.04 + SWA + FA3 (val_bpb: 1.1318) #198's 11L at 1.1326. The embedding matrix gets 4x larger, forcing fewer layers. At current artifact sizes, depth wins over vocab breadth. New data from Record: 10L d=512 Int5-MLP Int6-Attn sp1024 (val_bpb=1.1508) #465: Int6 embedding quantization costs only +0.0005 BPB — enabling sp8192 at d=512, but even then sp8192 8L (1.1794) loses to sp1024 10L (1.1508). More layers still dominate.
• SmearGate without OrthoInit — hurts BPB by 0.003 (see SmearGate deep dive).
• SWA with bf16 accumulation — Non-record: Negative findings on codebook quantization, magnitude pruning, multi-token prediction, embedding factorization #212 found catastrophic precision loss when accumulating SWA checkpoints in bf16. Must use fp32. (However, @kellyvv's #238 found that with enough SWA checkpoints (84), the quant gap actually reverses — quantized BPB becomes 0.037 better than pre-quant. SWA smoothing eliminates quantization-sensitive outliers.)
• MTP (multi-token prediction) — Non-record: Negative findings on codebook quantization, magnitude pruning, multi-token prediction, embedding factorization #212's controlled test: no BPB improvement (1.1947 vs 1.1929 control).
• Curriculum learning (content-based) — Non-record: Negative findings on codebook quantization, magnitude pruning, multi-token prediction, embedding factorization #212 found no effect.
• LAWA-EMA replacing SWA — context-dependent. LAWA-EMA frontier fork (pr198 base, SWA -> LAWA val_bpb=1.1551) #201 tested EMA alone on 11-Layer Int6 + WD=0.04 + SWA + FA3 (val_bpb: 1.1318) #198 base: 1.1551 (0.023 worse than SWA). But Record: 11L XSA + EMA + Int6 MLP3x + WD=0.04 (val_bpb: 1.1271) #287 uses EMA (decay=0.997) WITH XSA and reaches 1.1280 — beating SWA. EMA needs XSA to work. EMA decay=0.999 was also tried on Record: 11L XSA + EMA + Int6 MLP3x + WD=0.04 (val_bpb: 1.1271) #287 and hurt BPB — too slow to average (per @jfprincz). The sweet spot is 0.997.
• cuDNN SDP vs Flash SDP — Non-record: val_bpb=1.1374, FA2+SWA adaptation of Farnsworth #281 found cuDNN is 40% faster per attention op but produces worse BPB (1.1455 vs 1.1418). More steps doesn't help — different internal accumulation precision hurts quality.
• SwiGLU activation — worse than relu² on the standard architecture (V2 Prototype: SwiGLU + Dropout + MuonWD + MidLayerLoop #340, Non-record: 11L MLP3.5x LeakyReLU(0.5)^2 + Full SOTA Stack (mean val_bpb=1.1330, 8xH100) #344). However, GEPA's AI-discovered architecture uses SwiGLU successfully — both with TTT (Record: SwiGLU + XSA4 + U-Net + AdamW TTT (3-seed mean val_bpb=1.0672) #462: 1.0672) and without TTT (Record: SwiGLU+VE128+NoTTT val_bpb=1.1181 (3-seed mean) #505: 1.1181, GEPA non-TTT). Clarification: Record: SwiGLU+VE128+NoTTT val_bpb=1.1181 (3-seed mean) #505's code reveals the "SwiGLU" label is misleading — the actual activation is Star-ReLU (relu²+learned affine scale+bias, arXiv:2210.13452), NOT true SwiGLU gating. The MLP uses a single up_proj, not the dual-projection gated structure of SwiGLU. Star-ReLU works when co-optimized with U-Net skip gates and hidden=1792.
• Step-based LR schedule — Non-record: 11L MLP3.5x LeakyReLU(0.5)^2 + Full SOTA Stack (mean val_bpb=1.1330, 8xH100) #344 found −0.483 BPB vs wallclock-based warmdown. Catastrophic because the 600s budget varies by hardware; step-count schedules can't adapt.
• Error-guided TTT — concentrating TTT on highest-loss tokens doesn't help; they're genuinely unpredictable. Also: focal loss TTT (Record: Cosine TTT scheduling with per-layer lr — mean val_bpb=1.0970 (3 seeds) #481: no improvement over CE) and KL-divergence from pre-quant model (Record: Cosine TTT scheduling with per-layer lr — mean val_bpb=1.0970 (3 seeds) #481: no improvement). 7 failed TTT objective/targeting variants total.
• Advanced quantization algorithms at int6 (Non-record: Negative results — quantization algorithms, TTT, architecture, and self-generated GPTQ calibration study #756, @abaybektursun, SOTA holder). Qronos iterative Hessian (+0.0007 worse) and CDQuant coordinate descent (+0.0005 worse) both fail to improve over standard GPTQ at int6. Reason: the int6 quant gap is only +0.0036 BPB — most weights are already at their optimal grid point. At int6, GPTQ is near-optimal. Also: TTT alone is marginal on val-calibrated GPTQ stacks — 25 total failed TTT attempts across two stacks (full, MLP-down, MLP-all: all +0.0001 or neutral). Val-calibrated GPTQ rounding decisions are disrupted by gradient updates.
• MoE at small scale — Non-record: MoE exploration + multi-bit quantization analysis #480: 2-expert soft-routing MoE = −0.06 to −0.08 BPB vs dense baseline on 8xH100. Apple scaling laws (ICML 2025) confirm optimal sparsity = 0 below ~500M params. MoE is definitively unviable at competition scale.
• Lightweight TTT — 5 variants dead at frontier. Naive SGD (Record: 11L XSA+EMA+TTT, sliding val_bpb=1.1254 (3-seed mean 1.1256) #338: neutral), MLP-only (Non-record: Negative results & insights from 24hrs on 8xH100 #375: neutral), Reptile (Non-record: Negative results & insights from 24hrs on 8xH100 #375: +0.008 worse), Self-Distillation (Record: 11L GPTQ-lite + Int6 MLP3x (val_bpb=1.1257) #379: −0.0003), MAML (Non-record: Meta-TTT + Cache/OGD Eval Stacking + Tokenizer Ablation #384: +0.085 worse). All used conservative settings. But aggressive TTT works in principle: backward-looking TTT reached 1.1162 (Record: int5 GPTQ + Soft-Round QAT (3-seed mean 1.1162) #606, 3-seed — now closed for eval-time GPTQ). The lightweight variants failed; the aggressive variants (AdamW, cosine LR, selective freezing) succeeded — but need GPTQ within training budget.
• Multi-epoch TTT — memorization is a gradient, not a threshold (Record: PROTEUS v8 — 11L INT6 + LoRA TTT 5ep cosine (mean val_bpb=0.7853, 4 seeds) #568, Record: PROTEUS v7 — 11L INT6 + LoRA TTT (mean val_bpb=0.9512, 3 seeds) #512, Non-record: Empirical Bayes Adaptive TTT (val_bpb=1.1185) #484). The PROTEUS series reveals a clear memorization gradient: 3ep→0.9512 (Record: PROTEUS v7 — 11L INT6 + LoRA TTT (mean val_bpb=0.9512, 3 seeds) #512), 5ep→0.7853 (Record: PROTEUS v8 — 11L INT6 + LoRA TTT 5ep cosine (mean val_bpb=0.7853, 4 seeds) #568, −0.166 BPB from just 2 more epochs). At 0.78 BPB the model compresses to <1 bit per byte — near-certain data reproduction. Non-record: Empirical Bayes Adaptive TTT (val_bpb=1.1185) #484's original diagnostic (10+ constant-LR = memorization, 3ep = genuine) doesn't capture this: cosine LoRA TTT memorizes progressively, not at a fixed threshold. The ~0.95 floor was for full-weight TTT; LoRA TTT memorizes differently. Cosine TTT at 100ep (Record*: val_bpb=0.978 BPB — Goldfish ML Autonomous Research (100ep Cosine *leaky* TTT) #517: 0.978) stays above the 0.95 floor — suggesting cosine full-weight and cosine LoRA have different memorization dynamics.
• Systematic frontier negatives (Non-record: Negative results & insights from 24hrs on 8xH100 #375, 13 techniques on Record: 11L Partial RoPE + LN Scale + EMA + XSA4 (val_bpb: 1.1248) #315 base, $500/24hrs on 8xH100). Reptile meta-TTT: +0.0076 worse (1.1332, 20% training budget consumed). All 3 TTT variants (low-LR, high-LR MLP-only, Reptile) failed. Also failed: memory tokens (+0.016), Canon layers (48% overhead), MTP (+0.028), gradient-guided quant (noise), cautious WD (breaks torch.compile: 710× slower), label smoothing, L1 reg, 1M batch, full-run QAT. Key positive findings: EMA > standard SWA by 0.003 (3-seed); 786K > 524K by 0.004 (total tokens > gradient steps at frontier). Throughput heuristic: each 1ms step overhead ≈ 0.006 BPB — any technique adding Nms must deliver >N×0.006 to break even. INT4 quant gap: 0.048-0.060 (exponential from int6's 0.006). 3-seed std: 0.0007 BPB.
• INT4 quantization — catastrophic. Non-record: MoE exploration + multi-bit quantization analysis #480's controlled grid: int6→int5 MLP costs +0.007, but int6→int4 MLP costs +0.065 (10× worse). Full grid: attn6/mlp4 = 1.2111, attn5/mlp4 = 1.2183 vs baseline 1.1456. Int4 is a dead end.
• MLA (Multi-Head Latent Attention) — [Non-record] MLA + SmearGate + BigramHash + SWA — pre-quant 1.2838 bpb #354's kv_rank=128 MLA on 13L runs at 83ms/step vs ~43ms baseline, halving token throughput (~3.7B vs ~7.2B tokens). Pre-quant 1.2838 — architecture quality may be there but throughput cost makes it infeasible in the 600s budget.
• Block-wise weight sharing / depth recurrence — aggressive recurrence (3+ cycles) fails at 512d, but shallow recurrence works. Non-record: 11L MLP3.5x LeakyReLU(0.5)^2 + Full SOTA Stack (mean val_bpb=1.1330, 8xH100) #344: 2x slower. Non-record: 12L Low-Rank Q + QAT (1xH100, pre-quant 1.2035) #316: 0.09 BPB cost. Non-record: Depth Recurrence 5x3 — Weight-Shared Looping Transformer (6xH200, val_bpb=1.2716) #319: loop gates collapse. Non-record: Depth Recurrence in Parameter-Constrained Transformers — What Works, What Doesn't, and Why #363: quant error amplifies ~900× over 3 cycles. Non-record: Empirical Bayes Adaptive TTT (val_bpb=1.1185) #484 (EBLS): gammas → 0 for MLP (fully shared). The Frugendorff: Recursive Weight Sharing Research — Cadence Laws + Challenges (1.1325 BPB) #579: 6×2 loops — 1.1478 (1-seed). GPTQ compounds multiplicatively: 2 loops survive, 3+ catastrophic (+4.3 BPB). Exception: Record: Depth Recurrence (layers 4 and 5 repeated): val_bpb 1.1182 #686 (@msisovic) uses shallow recurrence (layers 4+5 repeated once each, 11→13 virtual layers) with per-pass learnable block scalars (~2K params). Reaches 1.1182 (3-seed). Recovers ~70% of independent 12L gain at minimal step cost. Key: staying within the "2 loops survive" zone.
• AttnRes (learned softmax over depth) — Record: 11L Int6+Zstd MLP3x SmearGate BigramHash OrthoInit MuonWD EMA (mean val_bpb=1.1497) #362: 54% throughput penalty from routing attention over layer outputs. Infeasible in 600s budget.
• MUD optimizer (Non-record: MUD optimizer — triangular Gram preconditioning (arxiv:2603.17970) #510) — Triangular Gram preconditioning replacing Muon's Newton-Schulz. 1.1989 BPB at 118ms/step (4.5× slower than Muon's ~26ms). Only 5,087 steps in 600s. Alternative optimizers remain unviable: throughput cost dwarfs quality gains.
• FTLE per-row precision (Non-record: 12L Low-Rank Q + QAT (1xH100, pre-quant 1.2035) #316) — Dynamical systems-inspired row-level quantization (Lyapunov exponent tracking). Clean negative result: uniform int-N beats FTLE-guided mixed precision at every bit width, because mixing bit widths within a row increases quantized value entropy, which defeats zstd compression. Lower RMSE does not imply smaller artifact.
• Non-record: 11L XSA-all + Full GPTQ + Selective Pruning (val_bpb=1.1154, 3-seed) #609 frontier ablations (16 techniques on Record: Full GPTQ + LeakyReLU² + Parallel Muon + BigramHash 3072 (val_bpb 1.1163, 3-seed mean) #593 stack). On the current best non-TTT base: VRL +0.0012 (conflicts with VE128), Gated Attention +0.0011 (3% step overhead), Catalytic Residuals −0.0001 (redundant with existing scaling), Backout −0.0005 (redundant with U-Net skips), TrigramHash +0.0049 (hurts compression), Hadamard rotation −0.0002 but +0.5MB (net negative for artifact), Temperature scaling +0.0002 (model well-calibrated at T=1.0), seq4096 eval catastrophic (RoPE breaks beyond training length), lzma at 99.7% Shannon limit (entropy coding gains capped at 0.05MB).
• Knowledge Distillation — dead at this budget (#1029). Hard distillation catastrophic (+0.090 to +0.407 BPP). Soft KL (top-32 cached logits from 105M teacher): +0.003 BPP worse at all alpha values. ~11ms/step I/O overhead costs ~280 steps; knowledge transfer doesn't compensate. Extended 2-hour training shows no crossover — curves track identically. Online distillation (761ms/step) is a non-starter. At 600s budgets, per-step overhead is fatal.
• Compression moonshots — MSE ≠ artifact size (#1048, 8 findings). Procrustes symmetry-transport: 91% MSE reduction but 380% larger artifact (rotation matrices are dense/high-entropy). Low-rank rotation: rank-128 captures only 16.6% of variance. 3% pruning increases artifact by 728KB (zeroing hurts zstd-22). Takeaway: int6 + zstd-22 is near-optimal. Always measure compressed artifact, not RMSE.
• MC Dropout ensembling (#1021) — K=16 passes at dropout=0.30: +0.005 BPP. dropout=0.05: +0.002. Sub-networks lack diversity at 17M params. Deterministic single pass strictly better.
• AdamW TTT at high learning rate (#1045) — AdamW at lr=0.002 degrades from 1.1509 to 1.2804 (+0.13 BPP). Per-document optimizer state resets interact badly with adaptive LRs. Not universally dead: lower-LR AdamW TTT works (Record: 11L VR + GA + LeakyReLU² + Legal Score-First TTT (val_bpb=pending) #490, Record: 1.0400 BPB -- Hedge Mixer + VRL + AdamW TTT + Polyak EMA #731, 1.1085 BPB: JEPA + AdamW TTT + Full GPTQ + FA3 + LZMA #1006). The negative is LR-transfer from SGD to AdamW without re-tuning.
• kNN-LM at eval time (#1103, @abaybektursun). Single-layer (k=64, L2, 2M store): +0.0026 BPP. Multi-layer (11 layers concatenated, cosine): +0.0031. XSA-all already captures the inter-position patterns that kNN-LM tries to exploit. From the SOTA holder, on the Record: AR Self-Gen GPTQ + XSA-all + BigramHash 3072×112 — val_bpb 1.11473 (3-seed mean) #1019 stack.
• Sliding window logit averaging (#1103). 32-window average: +0.024 BPP. Destroys sharp predictions. Catastrophically negative.
• SelfExtend / extended context 4096 (#1103). +0.48 BPP. The model was trained at seq2048; extending to 4096 at eval time causes massive degradation.
• Mixed-precision GPTQ int4 attn / int8 MLP (#1103). +0.047 BPP. Int4 attention weights lose too much information. Hessian sensitivity says MLP is more important, but int8 MLP still can't compensate for int4 attention.

2:4 Structured Sparsity (#1105): +0.672 BPB. Definitively dead at competition scale.

Turbo-Muon on 8xH100 (#1105): +0.0018 BPP worse AND artifact over 16MB. Early convergence advantage at step 500 doesn't hold at 7000+ steps.

SLOT causality concern (#1105): @abaybektursun found shared delta optimized over all positions then applied to all positions leaks future tokens. Removed from their submission. However, #1176 and #1172 use SLOT successfully — implementation details may differ.

1xH100 not a viable proxy (#1186): 8x fewer optimizer steps means results don't transfer to 8xH100.

SGD+momentum TTT (#1186): +0.065 BPP vs AdamW. Use AdamW for TTT.

Scale deception (#1227): Local experiments can be 180° wrong at full scale. SSM hybrid showed -18% CE improvement at dim=192 but was +2.7% BPP worse at dim=512 on H100. Systematic bias, not noise. Takeaway: always validate on the target hardware and model size.

Product Quantization (#1227): +292% BPP. Catastrophic at competition scale.

PAQ Logistic Mixing (#1227): BPC=19. Fundamentally broken for multi-class prediction — compression-domain techniques don't transfer to neural LM output heads.

Complementary Training (#1227): +2.6% BPP at full scale. Does not help.

LN Scaling (#1227): +11.4% BPP. Harmful.

QAT as regularizer (#1227, positive): Quantized model beats float32 by 0.66%. QAT can improve BPP, not just preserve it — acts as beneficial regularization.

1. Int6 Quantization (instead of Int8)

Standard post-training quantization maps each weight to an 8-bit integer (256 levels). Int6 uses only 6 bits (64 levels, range [-32, 31]) with per-row scale factors, then compresses with zstd (level 22) instead of the baseline's zlib-9. Int6 frees ~25% more artifact space than int8, reinvested in a bigger model. Some submissions keep sensitive layers in fp16 (tied embedding) or int8 (embeddings) to limit compounding precision loss.

Origin: @nanlliu introduced int6 mixed precision in #39.

2. MLP 3x Expansion

The baseline uses 2x MLP expansion (hidden dim 1024 for 512-dim model). Top submissions use 3x (1536). Wider MLP = more expressive capacity, funded by int6 artifact savings.

Origin: @jfprincz in #70 (Mar 19 08:57 UTC). @saml212 independently reached the same insight in #61 later that day.

3. Sliding Window Evaluation

Overlapping windows (stride=64, window=2048) give each scored token 1984+ tokens of context vs minimal context with non-overlapping chunks. Purely eval-time. Worth 0.034 BPB per @samacqua's ablation in #77.

Origin: @mattqlf in #50. Stride debate: stride=256 gives marginally better BPB at 4x less eval time (#114). Doc isolation hurts at stride=64 — use flat-stream eval (#199).

4. FP16 Tied Embedding

The tied embedding matrix (input + output) is uniquely sensitive to quantization — errors compound in both directions. Keeping it in fp16 (~1MB) is the single highest-value precision decision.

Origin: @chonchiog in #42.

5. Zstd-22 Compression

Zstandard at level 22 squeezes int6 data significantly tighter than zlib-9 — enough to fit ~1-2M more parameters. Compression happens once after training; decompression is fast. Free lunch.

Val data ruled out (Mar 20, @0hq): Val tokens cannot be in the artifact. Paid prefix (#168), error correction (#108), val-only training all banned for record track. Now in README FAQ.

TTT ruling (Mar 20, @0hq on #152): Only backward-looking TTT allowed — adapt on tokens already graded, not future tokens. Pre-eval adaptation invalid. Causal TTT (#267-style) remains allowed. In README FAQ.

Mar 22, @cocohearts on #317: TTT is "not in the spirit of the challenge." Broader organizer signal — even backward-looking TTT may face scrutiny.

Mar 23, @0hq on #402: Explicit TTT clarification — token-stream model is correct. You may use any preceding eval tokens already graded. You may NOT re-order the evaluation set. Invalid TTT PRs (train-on-val-then-measure) will be closed. Auto-review process being built.

Mar 23, @cocohearts: #374 rejected for insufficient statistical significance vs new SOTA. #505 needs packaging fixes.

Mar 24, @valerio-oai — enforcement sweep (15+ PRs closed). Two categories: (1) TTT information leakage: multi-epoch TTT with min-NLL selection, and adapting-then-scoring same tokens, both ruled equivalent to "training on the val set." #593, #576, #573, #568, #596, #605, #614, #620, #518, #548 closed. (2) Training data at eval time: GPTQ calibration using training data during eval budget disallowed. #593, #576, #569 closed for this. Calibration must count within training 600s. #589 ruled valid but closed — fails 0.005-nat threshold vs #549 SOTA. valerio-oai confirmed: "TTT is a valid approach in theory" but "very easy to unintentionally leak val data into."

Mar 25, @valerio-oai — second enforcement sweep (issue #677). Comprehensive audit. (1) Eval-time GPTQ: Training for full 600s then doing GPTQ calibration afterward (even 3-4s) is "accessing training data at eval time" — disallowed. #606, #615, #626, #639, #656 closed. (2) N-gram eval cache ruling: The concept is "directionally legal" — building a cache from already-scored tokens is allowed. The specific #659 implementation was illegal (hindsight selection: comparing n-gram vs LM on the true next token). Legal alternatives: fixed-weight blending or entropy-adaptive alpha (using model uncertainty, not ground truth). (3) #706 flagged: @valerio-oai told @newjordan that #706's GPTQ calibration still runs after 600s training time — needs fix. (4) Broad invalid TTT list: #410, #415, #417, #442, #462, #481, #486, #517, #518, #532, #555, #581, #595 all flagged for adapting on validation before the reported eval pass.

Mar 27, @valerio-oai — mass n-gram cache closure (33+ PRs, #677). Hashed n-gram caches disallowed: they score only the correct token via hashing without normalizing over the full token distribution, producing invalid probabilities. Two-pass rescoring (score → TTT → rescore) explicitly disallowed as "training on the eval set." PRs closed include #846, #853, #868, #869, #870, #876, #881, #888, #893, #900, #907, #912, #918, #982, and many more. Only reviewing PRs after #988 for potential merging. Recommended: @NoesisGenesis's formal criteria for valid causal prediction — (a) distribution depends only on artifact + strict prefix, (b) full normalized distribution over token vocabulary required before scoring, (c) score computed from pre-update probability only, (d) single left-to-right pass.

Mar 28, @valerio-oai on #728: Val-calibrated GPTQ "breaks autoregressivity" — disallowed. Self-generated calibration data (as in #1019) is "probably legal." PR left open pending fix.

Mar 28, @valerio-oai on #991: Closed for double-pass TTT (score, train, rescore same tokens).

Mar 28, @valerio-oai on #1028: GPTQ calibration running after 600s training cap = accessing training data at eval time. Author confirmed bug and resubmitted as #1047 with GPTQ within budget. Under organizer review.

Muon (MomentUm Orthogonalized by Newton-Schulz) is the optimizer at the heart of this competition's baseline, created by Keller Jordan for the NanoGPT speedrun. It runs standard SGD with Nesterov momentum, then post-processes each 2D parameter's gradient update by replacing it with the nearest orthogonal matrix via Newton-Schulz iteration. Intuitively: compute the gradient direction, then "clean it up" so the update is maximally informative without redundant directions. It's equivalent to steepest descent under the spectral norm, which improves the conditioning of the optimization landscape. ~35% faster training than AdamW on language models.

NorMuon extends Muon by adding per-neuron adaptive learning rates from accumulated second-order statistics. Vanilla Muon can produce updates with highly non-uniform norms across neurons, causing some neurons to dominate training. NorMuon normalizes row-wise after orthogonalization, combining Muon's conditioning benefits with Adam-style balanced per-neuron learning. It also improves distributed scaling by avoiding full momentum gathering across GPUs. Used by @mtybadger (#122), @vmfunc (#89), @abhishekgahlot2 (#137), and others.

Muon Weight Decay — The competition baseline's Muon optimizer has no weight decay. Decoupled weight decay for Muon (p.mul_(1 - wd * lr)) existed in modded-nanogpt since Nov 2025, but wasn't in the baseline. @notapplica was the first to bring it into this competition in #60, improving BPB from 1.2160 to 1.2094. Weights stay smaller and better-distributed, improving both generalization and compressibility.

Post-enforcement status (Mar 27): The Mar 27 enforcement sweep closed 33+ n-gram cache PRs after discovery of the normalization bug — implementations scored only the correct token without full-vocabulary normalization, producing artificially low BPP. #978 proved that properly normalized n-gram achieves only 1.51 BPB (worse than neural baseline). The question of whether a correctly normalized eval-time statistical method can improve on pure neural remains open.

Instead of training in full precision and quantizing afterward, QAT simulates quantization during training. In the forward pass, weights are rounded to their quantized values. The problem: rounding is non-differentiable, so gradients can't flow through it.

The Straight-Through Estimator (STE) solves this by pretending the rounding operation is the identity function during the backward pass. It's mathematically "wrong" but works remarkably well — the model learns weight configurations that are robust to precision loss because it's been "seeing" quantized weights throughout training.

Late QAT outperforms full-training QAT: The later, the better. @trovatochris (#117) activates at 70%, @mohosy (#130) at 75%, @unixmadtoonslab (#76) at 85%. #76 even dropped QAT entirely at 12L (1.1468), finding WD=0.04 alone sufficient. @jfprincz's #315 pushes this to the extreme: STE activates only in the final 4% of training (lr_scale < 0.1, during low-LR warmdown). This cuts the int6 roundtrip gap to ~0.007 BPB while preserving full-precision convergence. The lesson: QAT activation is a spectrum — later = cleaner convergence, better int6 gap.

Int8 vs int6 QAT tradeoff: @mrdavtan's ablation in #145 shows that int8 QAT is not worth it under the 10-min wallclock cap. The torch.quantile call for exact percentile matching adds ~20% per-step overhead (64ms → 77ms), costing ~2,000 training steps. Result: 1.2052 BPB with QAT vs 1.1925 without — the lost training tokens hurt more than closing the ~0.007 int8 quantization gap. Int6 QAT, however, likely pays off because its larger ~0.01+ BPB gap justifies the overhead — confirmed by #128 and #137.

@unnir introduced SmearGate in #102 and refined it in #135. This appears to be a novel technique for this competition — no published papers found.

SmearGate: A tiny learned gate (~512 params) that blends each token's embedding with the previous token's. This injects bigram (two-token) context directly into the embedding layer before the transformer starts processing. Normally a transformer must discover token pair relationships through self-attention; SmearGate provides this signal for free.

Bigram Hash: A hash table (commonly 2048-10240 buckets, dim=128, projected to 512) that maps token pairs to learned embeddings. Together with SmearGate, this gives the model token-pair awareness at nearly zero parameter cost.

@unnir's original combination with orthogonal initialization achieved 1.1539 BPB in #135. @jfprincz's #198 (1.1326) extended this with 11L + SWA + FA3 + WD 0.04, and #287 (1.1280) extended further with XSA + EMA.

OrthoInit appears critical for SmearGate. @mrdavtan's ablation in #212 found that adding SmearGate + BigramHash without OrthoInit hurt BPB (1.1739 vs 1.1708 without). Every successful SmearGate submission uses OrthoInit — the two techniques may be co-dependent.

XSA (arXiv:2603.09078, Shuangfei Zhai, 2026) removes self-value bias from attention output via orthogonal projection. In standard attention, each token's value vector contributes to its own output — XSA subtracts this self-component, forcing the model to rely on information from other tokens. Applied to the last 3-4 layers only ("Partial XSA"), where self-attention bias is highest.

Zero parameters, minimal overhead. @unnir's #265 GQA-aware implementation reduces XSA overhead from ~7ms/step to ~2ms/step. Near-universal among frontier submissions. Best non-TTT (#609, 1.1154) uses XSA on all 11 layers; official SOTA (#1019, 1.1147) uses XSA-all.

XSA coverage depth: 4 layers appears near-optimal. @gowtham0992's #478 tested XSA on ALL 11 layers: 1.1268 (3-seed) vs XSA-4 at 1.1327 on the same base (−0.006 from XSA-all). But #414 (XSA-4 + VE128 + Partial RoPE + LN Scale) reaches 1.1228 — better than #478's XSA-all(11) at 1.1268. XSA-all adds ~3ms/step overhead (−230 steps), and removing self-value from ALL layers may degrade the model's own-representation capacity. The progression: 3 layers (#265: 1.1307) → 4 layers (#414: 1.1228) → 11 layers (#478: 1.1268) suggests 4-6 layers is the sweet spot for non-TTT. However, #609 (1.1154, best non-TTT) uses XSA-all(11) and #606 (1.1162, best legal TTT) also uses XSA-all — at the current frontier, XSA-all with Full GPTQ overcomes the overhead penalty.

@samacqua introduced a creative approach in #77: adapting the model during evaluation.

For each validation document, rank-8 LoRA (Low-Rank Adaptation) adapters are trained on the document's own text using only backward-looking context (no data leakage). The model essentially "studies" each document briefly before being scored on it. LoRA makes this practical by only training tiny low-rank matrices (~1.5% of params) rather than the full model, enabling batched per-document adaptation within the eval time budget.

Original #77 ablation showed TTT itself adds ~0.003 BPB on early baselines (most gain came from doc isolation + sliding window). Full-model SGD TTT (#152) was ruled invalid by @0hq — only backward-looking (score-first) TTT is legal. The best legal TTT submissions (#606 at 1.1162, #615 at 1.1169) were later closed for eval-time GPTQ on training data (Mar 25 sweep).

TTT on XSA+EMA is a spectrum, not a binary. On SmearGate bases: #254 shows 0.014 BPB gain. Three XSA+EMA data points, sorted by base strength: (1) #317 (weak base, pre-quant 1.1581, no FA3): TTT gains 0.024 BPB. (2) #338 (@alertcat, #315 base — frontier at 1.1250, Partial RoPE + LN Scale + Late QAT): TTT neutral ±0.001 (3 seeds). (3) #303 (@sseanliu, #287 base — 1.1280, without #315's additional regularization): TTT +0.016 BPB worse. The pattern suggests TTT interacts with how tightly converged the base model is: under-trained bases benefit from local adaptation; over-regularized frontier bases are disrupted; the current frontier (#315) sits in a neutral zone. #338's neutral result is informative — it means TTT is not a meaningful lever at the frontier.

Reptile meta-TTT: gains on SmearGate, fails at frontier. @sseanliu's #296 shows 0.011 BPB on SmearGate models vs 0.001 naive. But #375 tested Reptile on #315's XSA+EMA base: +0.0076 worse, consuming 20% of training budget. The SmearGate gain does not transfer to the frontier. All three TTT variants (naive, MLP-only, Reptile) are now confirmed dead ends at ~1.125. Error-guided TTT is also negative — hardest tokens are genuinely unpredictable.

TTT optimizer recipe matters. @Christopher-Lee-McClendon's #461 (non-record, 4xA100) found that SGD+momentum(0.9), 3 epochs per 32K chunk, freezing first 2 blocks gets −0.0165 BPB TTT gain — 2.4× better than AdamW 1-epoch over all params (−0.0068 in their prior #456). Pre-TTT baselines nearly identical, so the entire improvement comes from the TTT recipe. This partially contradicts the #442 narrative (AdamW >> SGD) — the comparison is more nuanced: selective freezing + multi-epoch SGD with momentum can outperform single-epoch full-network AdamW.

Legal TTT survivors — none remain after Mar 25 sweep. #606 (1.1162) and #615 (1.1169) were closed — eval-time GPTQ calibration on training data. #576 (1.1164) closed in Mar 24 sweep. #573 (Multi-Pass min(NLL)) ruled invalid. All frontier TTT submissions used eval-time GPTQ and are now invalid. TTT optimizer matters for GPTQ: SGD TTT hurts Full GPTQ models (+0.030, #601), but AdamW with cosine LR works. The remaining legal TTT avenue requires GPTQ calibration within the 600s training budget.

Cosine TTT scheduling is a 3× multiplier. @mrdavtan's #481 (3-seed, 1.0970) introduced two TTT innovations on top of AdamW TTT: (1) cosine LR decay over 30 epochs — high LR early to repair quant damage, low LR late to refine; (2) per-layer LR groups based on measured quantization error — 3× base LR for MLP output projections (3.4× higher quant error), 0.5× for input projections. Result: TTT gain of −0.061 BPB vs #442's −0.019 with flat LR — a 3× improvement from scheduling alone. Pre-TTT ~1.158 (weaker base, FA2 not FA3). Also tested: focal loss and KL-divergence from pre-quant model — both failed to improve over CE. ⚠️ Pre-eval TTT.

The single biggest BPB lever discovered in the competition. During sliding window evaluation, a backward-looking n-gram cache is built from already-scored tokens and mixed with model predictions. The concept is simple: if the model has already scored "the cat sat on the", and the 5-gram "cat sat on the" was followed by "mat" last time, weight that prediction into the next token's distribution.

How it works:

1. After scoring each token, record the preceding n-gram context and the actual next byte
2. For new tokens, look up the current n-gram context in the cache
3. If found, mix the empirical n-gram distribution with the model's distribution: p_final = (1-α) * p_model + α * p_ngram
4. Store counts in a hash table (count-min sketch, ~4M buckets)

Three generations of implementation:

• Fixed 5-gram, fixed alpha (Podracing: 1.0461 BPB (3-seed mean) — 5-gram eval + LeakyReLU² #706, @newjordan): alpha=0.20 always. Simple. Drops BPB by ~0.07 (1.1202→1.0461).
• Multi-order backoff 2-7 (Record: 1.0240 BPB — Multi-Order N-gram Backoff + Entropy-Adaptive Alpha (100% autonomous research via goldfish) #702, @lukacf): Try 7-gram first, cascade to 6,5,4,3,2 on miss. Coverage jumps dramatically. Additional −0.02 over fixed 5-gram.
• Entropy-adaptive alpha (Record: First Legal Sub-1.0 BPB — Multi-order N-gram Backoff + Entropy-Adaptive Alpha (val_bpb=0.9674, 3-seed) #727, @Asukabot0): alpha = 0.05 + 0.55 * sigmoid(2 * (H - 4.0)). When the model is uncertain (high entropy), trust n-grams more. Additional −0.02 over fixed alpha. Combined with backoff: 1.1271 neural-only → 0.9674.

Why it's so effective: Language has enormous local repetition — names, technical terms, formatting patterns — that a small transformer can't memorize but n-grams capture perfectly. The n-gram cache acts as a lossless "local memory" that costs zero artifact bytes (built on-the-fly from eval data).

Legality argument: All implementations claim score-first backward-looking compliance — the cache uses only previously-scored tokens, alpha depends on the model's own entropy (not ground truth), and there's no oracle selection. #702 cites @valerio-oai suggesting entropy-adaptive alpha as a legal alternative in the #659 review. But the technique hasn't been officially ruled on yet.

#738 adds kNN-LM: @gowtham0992 stores 512-dim hidden states in a GPU ring buffer and finds k=32 nearest neighbors for uncertain tokens. RBF kernel builds a non-parametric distribution. Additive −0.007 BPB on top of n-gram cache. Based on Khandelwal et al. 2019 (ICLR 2020). Captures semantic patterns that pure n-gram statistics miss.

Ablation data from #727:

@jfprincz's #315 (1.1250) adds two effective zero-parameter techniques on top of #287's XSA+EMA base, gaining 0.0023 BPB. Note: Late QAT was also included in the code, but torch.compile constant-folded the _qat_enabled flag, making the STE branch dead code — Late QAT never activated (discovered by @152334H, confirmed in #453). Update: @wfproc (#1032) confirmed this dead-code bug persists in the current SOTA #549 codebase. A fix via tensor-scale STE actually worsened the int6 gap — suggesting WD+EMA already compensate for what QAT was supposed to do. The 0.0023 gain comes entirely from Partial RoPE + LN Scale.

Partial RoPE (16 of 64 head dimensions). Rotary Position Embedding (RoPE) injects position information by rotating query/key vectors. Standard RoPE applies to all head dimensions. Partial RoPE applies to only 25% (16 of 64 dims) — the remaining 48 dims attend without position encoding. Why this helps: the position-free dims learn semantic similarity independent of token distance, improving generalization across different position ranges. The model can learn both "what things are" (position-free) and "where things are" (position-encoded) using different parts of the same head. Zero new parameters.

LN Scale (output scaled by 1/√(layer_idx+1)). After each RMSNorm, the output is multiplied by a layer-dependent scale factor that shrinks with depth. Layer 0: ×1.0; Layer 5: ×0.408; Layer 10: ×0.302. This damps the contribution of deeper layers to the residual stream, preventing later layers from "overwriting" early representations. Training is more stable — the model can use depth incrementally rather than being forced to route everything through deep layers. The 1/√(layer+1) schedule is related to the "depth scaling" used in some architecture papers. Zero new parameters.

Late QAT (STE enabled only when lr_scale < 0.1) — ⚠️ was dead code in #315. torch.compile constant-folded the _qat_enabled class attribute, so the STE branch never activated (discovered by @152334H, confirmed in #453). The concept is sound — late activation avoids corrupting Muon's momentum — but #315's actual gains came from Partial RoPE + LN Scale alone. Working Late QAT: @unnir (#374, scale<0.1), @signalrush (#414, threshold 0.15), @fbedev (#417). Downstream submissions copying #315's code may also have inactive Late QAT.

The two active techniques (Partial RoPE + LN Scale) gain 0.0023 BPB vs #287 — statistically clear (3-seed variance 0.0005 BPB, t-stat -101.9 vs SOTA, p << 0.01).

Validated against the SOTA at submission time.

Sorted by BPB ascending.

Competitive submissions that haven't demonstrated statistical significance.