His 0.9695 BPB is val-only training (separate track). Standard score is
1.1629, close to Larson's 1.1574. No novel architecture — wins with
tuning: stride-64 sliding window, seq_len=4096, mixed int6/int8 quant,
Muon momentum=0.99. Crucially, he uses NO weight sharing, meaning our
fractal approach is an orthogonal improvement on top of his full stack.

https://claude.ai/code/session_01RtoPPgJGUFS7XfcFCPwYtq

New features:
- MUON_WD: decoupled weight decay for Muon optimizer (from #1 entry)
- FP16_EMBED: keep tok_emb in fp16 instead of int6 (from #1 entry)
- Both controlled via env vars, off by default

Leaderboard chase script tests stride=64, fp16 embed, muon WD, and
full stack on MLP 3× base. All runs use early QAT (25%).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

SWA: average model weights every 50 steps over last 50% of training.
Smoother weight distributions quantize better. Controlled via
SWA_EVERY and SWA_START_FRAC env vars.

MuonWD bumped from 0.01 to 0.04 (PR 162 swept 0.01-0.05, optimal at 0.04).
MLP_HIDDEN=1408 to fit under 16MB.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

#1 untried combination from competition commentary:
TTT (from #254) + XSA (from #265) = estimated 1.117-1.121 BPB
XSA_LAST_N=3 excludes self-attention in final 3 layers.
Zero extra params, frees attention capacity for cross-token focus.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Copy of pr374_safe — EMA(0.997) + Tight SWA + QAT(0.15) + warmdown(3500).
3-seed mean 1.1248, best seed 1.1243. #2 overall, #1 non-TTT on the leaderboard.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

#1 untried combination from competition commentary:
TTT (from #254) + XSA (from #265) = estimated 1.117-1.121 BPB
XSA_LAST_N=3 excludes self-attention in final 3 layers.
Zero extra params, frees attention capacity for cross-token focus.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Copy of pr374_safe — EMA(0.997) + Tight SWA + QAT(0.15) + warmdown(3500).
3-seed mean 1.1248, best seed 1.1243. #2 overall, #1 non-TTT on the leaderboard.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

#1 untried combination from competition commentary:
TTT (from #254) + XSA (from #265) = estimated 1.117-1.121 BPB
XSA_LAST_N=3 excludes self-attention in final 3 layers.
Zero extra params, frees attention capacity for cross-token focus.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Copy of pr374_safe — EMA(0.997) + Tight SWA + QAT(0.15) + warmdown(3500).
3-seed mean 1.1248, best seed 1.1243. #2 overall, #1 non-TTT on the leaderboard.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Three variants targeting the 0.187 BPB gap to #1:
- bwing_alpha: clip 0.95, alpha 0.05-0.60 (isolate alpha curve)
- bwing_entropy_shift: per-order entropy center shift (isolate)
- bwing_full_port: all #809 techniques + fixed order mults (fire first)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Three improvements over Green v1 (baseline: sliding=1.1129, ngram9=0.4489):

PR #931 (packed training oracle): after training, reads 2 train shards
(~200M tokens) and seeds eval n-gram tables before val token #1.
Eliminates cold-start penalty where early val chunks score with empty cache.
Legal: oracle is training-data-only, eval remains single-pass causal.

PR #900 (Dirichlet smoothing): replaces linear alpha mixing with
  p = (ng_count + c * neural_p) / (ctx_count + c)
Count-sensitive weighting: high-count matches trust n-gram, low-count
matches stay close to neural prior. No hand-tuned alpha per-order needed.
NGRAM_EVAL_MIN_COUNT=1 (formula handles low counts naturally).

PR #859 (matrix_lr): MATRIX_LR=0.03 vs 0.025 in Green — higher LR
found across 79-experiment sweep to train stronger base model.

Both new features are independent toggles (ARTIFACT_NGRAM, NGRAM_DIRICHLET)
for A/B isolation.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>