Built on PR #720 by @agalimova. Acknowledges the HedgeMixer framework from that submission.

val_bpb: 1.0722 (3-seed mean, std 0.009) | artifact <15.7MB | eval <546s

3-Seed Results

What's New (vs PR #720)

Novel TTT recipe that improves test-time adaptation:

1. Per-layer LR groups: 3x LR for output projections (attn.proj, mlp.proj), 0.5x for input (mlp.fc) — targets the highest-value adaptation parameters
2. Cosine LR schedule within TTT: starts high, anneals to prevent overfitting in later chunks
3. 4 TTT epochs (vs 3) + freeze 1 block (vs 2) — more adaptation capacity
4. Skip standalone sliding eval to reclaim 86s for the extra epoch

Ablation

The mixer scoring framework and TTT adaptation work as a joint system — the Hedge-weighted expert ensemble provides calibrated probability estimates that make chunk-sequential TTT effective.

Rule Compliance (per Issue #677)

• ✅ Score-first TTT: Each chunk scored under inference_mode() before any model weight update. Every token evaluated before the model trains on it.
• ✅ Backward-looking mixer: N-gram count tables built exclusively from already-scored tokens. No token's own data is used in its prediction.
• ✅ No validation data during training: Mixer starts empty, builds statistics only during eval phase.
• ✅ No training data during evaluation: No access to fineweb_train_* shards during eval.
• ✅ No oracle/hindsight selection: Mixer blends expert predictions BEFORE observing ground truth. Hedge weights updated after scoring, not used to select predictions.
• ✅ No GPTQ calibration during eval: GPTQ calibration runs during training phase only.
• ✅ Artifact self-contained: All code in single optimize.py, model weights in compressed artifact.
• ✅ Time budgets: Train <600s, eval <546s, total <1200s.
• ✅ Artifact size: <15.7MB (<16MB limit).

Base Architecture (from PR #720)

• 11L, 512d, 8 heads / 8 KV heads, MLP 3.5x, LeakyReLU(0.5)²
• XSA on all 11 layers, BigramHash(6144, dim=128), SmearGate
• EMA(0.997) + SWA, int5 GPTQ-lite + zstd compression
• Parameter Banking with Parallel Muon optimizer

Test Plan

• 3-seed verification complete (1337, 42, 2025)
• All runs emit final_int6_ttt_exact metrics
• Artifact size < 16,000,000 bytes for all seeds
• Eval time < 600s for all seeds
• Ablation run confirming contribution breakdown