Summary

Builds on the merged SOTA (PR #549, 2026-03-23_LeakyReLU_LegalTTT_ParallelMuon, 1.1194 bpb) by adding SLOT (Sample-specific LM Optimization at Test-time) as a complementary eval-time adaptation on top of the existing legal TTT.

Results

All seeds: artifact < 16MB (pass), eval < 600s (pass), beats merged SOTA 1.1194 (pass)

Architecture

No changes to training. Identical to PR #549 base:

• 11 layers, 512 dim, 8H/4KV GQA, Partial RoPE (rope_dims=16)
• LeakyReLU(0.5)^2 MLP (mlp_mult=2.80)
• BigramHash (vocab=1536, dim=128), VE128 at layers 9-10, XSA last 4 layers
• LN Scale, DTG gate, U-Net skips, SmearGate
• EMA(0.997) + SWA -> GPTQ-lite int6 + LZMA-6 -> ~15.95-15.96MB artifact
• Parallel Muon + AdamW optimizer

Eval-Time Additions

Legal TTT (unchanged from PR #549)

• SGD, 3 epochs per chunk, score first, no future token leakage
• TTT_LR=0.002, TTT_CHUNK_TOKENS=32768, TTT_BATCH_SEQS=32

SLOT (new)

SLOT (Sample-specific LM Optimization at Test-time) adds a per-batch residual delta in R^512 that adapts the hidden->logit mapping without modifying model weights.

Protocol:

1. H = forward_hidden(x_batch) -- full 11-layer forward, single pass, torch.no_grad()
2. H.detach() -- gradient graph cut from model weights
3. 5 AdamW steps optimizing delta through compute_logits(H + delta) only -- backprop through the final linear projection only (~524K weights, ~1% of full forward cost)
4. Score with compute_logits(H + delta.detach()) -- this is the reported BPB

Legality: Model weights are frozen throughout. delta is optimized only on tokens that have already been scored in the current batch window (same guarantees as the existing TTT). Score-first ordering is preserved. One left-to-right pass.

Efficiency: The expensive forward (11 layers) runs once. The 5 optimization steps are through a single linear layer -- total SLOT overhead ~14ms per batch, well within the 600s budget.

Hyperparameters: SLOT_LR=0.003, SLOT_STEPS=5 (ablated: default lr=0.001/steps=3 gives -0.0007 bpb; tuned config gives -0.0042 bpb)

Compliance Verification

• Rule 1 (no future tokens in p_t): pass -- delta optimized only on already-scored tokens
• Rule 3 (score before update): pass -- score-first at every position
• Rule 4 (one left-to-right pass): pass -- no rescoring
• GPTQ-lite: data-free percentile clipping on weight tensors only (torch.quantile on param.data; no training data reopened post-600s)
• N-gram cache: not used
• Multi-epoch TTT before scoring: not present
• Eval time: 568.5s-571.8s across all seeds
• Artifact: 15.947MB-15.960MB across all seeds