Record: SLOT + LeakyReLU² + Legal Score-First TTT + Parallel Muon — val_bpb 1.1154 (3-seed mean)

val_bpb = 1.1154 (3-seed mean, std 0.0002) | ~15.9 MB | 8×H100 SXM

3-Seed Results (8×H100 80GB SXM)

vs Previous SOTA (PR #549)

Key Innovation: SLOT (Sample-specific LM Optimization at Test-time)

First SLOT-based entry in Parameter Golf. SLOT optimizes a single additive δ ∈ ℝ^512 vector at the last hidden layer during TTT scoring, adapting the model's hidden-to-logit mapping per-batch.

Source: Hu et al., arXiv:2505.12392v2, "SLOT: Sample-specific Language Model Optimization at Test-time" (Westlake University, 2025)

How SLOT Works

The model's forward_logits() is split into forward_hidden() + compute_logits(). During TTT Phase 1 (scoring), SLOT optimizes δ between the two:

for each batch of windows:
    # 1. Get hidden states from TTT-adapted model
    H = model.forward_hidden(x_batch)           # [bsz, seq_len, 512]

    # 2. Optimize delta (5 AdamW steps, lr=0.003)
    delta = zeros(1, 1, 512)                    # broadcasts across batch + seq
    optimizer = AdamW([delta], lr=0.003)
    for step in range(5):
        logits = model.compute_logits(H + delta)
        loss = CE(logits[:, :-1], targets[:, 1:])
        loss.backward()                          # gradients only through lm_head
        optimizer.step()

    # 3. Score with adapted logits
    final_logits = model.compute_logits(H + delta)
    nll = CE(final_logits, targets)              # used for BPB

Why SLOT Works

SLOT and TTT address complementary bottlenecks:

• TTT adapts all 27M model weights to local data distribution (chunk-level, SGD, 3 epochs)
• SLOT fine-tunes the final hidden→logit mapping per-batch (5 AdamW steps on 512 params)

TTT gives SLOT better hidden states; SLOT gives TTT-adapted representations a final per-batch correction. The two stack because they operate at different granularities (chunk vs batch) and different model depths (all layers vs last layer only).

SLOT Properties

• Zero artifact cost: δ is optimized from scratch per-batch during eval
• Minimal overhead: +217s to eval (569s total vs 352s baseline TTT)
• Score-first compliant: δ optimizes using autoregressive shift on the same tokens being scored; model weights are frozen during δ optimization
• Clean toggle: SLOT_ENABLED=0 reproduces PR Record: LeakyReLU² + Legal Score-First TTT + Parallel Muon — val_bpb 1.1194 (3-seed mean) #549 baseline exactly

SLOT Hyperparameters

Hyperparameter Ablation (seed 1337)

Also Tested: CTW — Negative Result

Context Tree Weighting (Willems et al., 1995) was integrated and tested across three progressively improved implementations. All degraded BPB.

Root cause: The 11-layer transformer at 1.12 BPB already captures all n-gram patterns a depth-4 Markov model knows. Mixing in a weaker predictor adds noise regardless of implementation quality.

Also Tested: Stacking Hacks — Negative Results

Base Architecture (PR #549 by @abaybektursun)

• 11L, 512d, 8H/4KV, LeakyReLU(0.5)² MLP 3×
• Parameter Banking + Parallel Muon (FlashAttention 3)
• BigramHash(1536), XSA4, Partial RoPE(16), LN Scale, VE128
• EMA(0.997) + Tight SWA(50), GPTQ-lite int6 + LZMA-6
• Legal Score-First TTT (SGD, lr=0.002, 3 epochs, 32K chunks)

Run Command

cd /workspace/parameter-golf && SEED=1337 SLOT_ENABLED=1 SLOT_LR=0.003 SLOT_STEPS=5 \
CTW_WEIGHT=0 NUM_LAYERS=11 BIGRAM_VOCAB_SIZE=1536 XSA_LAST_N=4 \
EMA_ENABLED=1 EMA_DECAY=0.997 SWA_ENABLED=1 SWA_EVERY=50 \
ROPE_DIMS=16 LN_SCALE=1 LATE_QAT=1 LATE_QAT_THRESHOLD=0.15 \
VE_ENABLED=1 VE_DIM=128 VE_LAYERS=9,10 \
TTT_ENABLED=1 TTT_LR=0.002 TTT_EPOCHS=3 TTT_CHUNK_TOKENS=32768 \
TTT_FREEZE_BLOCKS=0 TTT_MOMENTUM=0.9 TTT_BATCH_SEQS=32 TTT_GRAD_CLIP=1.0 \
MUON_WD=0.04 ADAM_WD=0.04 \
MATRIX_LR=0.025 SCALAR_LR=0.025 TIED_EMBED_LR=0.035 \
MUON_MOMENTUM=0.99 MUON_MOMENTUM_WARMUP_START=0.92 \
MUON_MOMENTUM_WARMUP_STEPS=1500 WARMDOWN_ITERS=3500 \
ITERATIONS=9000 MAX_WALLCLOCK_SECONDS=600 EVAL_STRIDE=64 \
torchrun --standalone --nproc_per_node=8 train_gpt.py

Credits

• SLOT integration, tuning, and analysis: Anubhav (@AnubhavBharadwaaj)
• SLOT algorithm: Yang Hu et al. (arXiv:2505.12392v2, Westlake University)
• CTW negative result analysis: Anubhav (@AnubhavBharadwaaj)
• LeakyReLU²: PR Record: 11L EMA + Int6 + XSA + LeakyReLU² + Partial RoPE (val_bpb: 1.1309) #493 by @parinzee, PR Record: 11L XSA4 + LeakyReLU(0.5)² + Cosine TTT 50ep (val_bpb=1.0622) #518 by @sofiabod
• Parallel Muon + Parameter Banking: PR Record: Parallel Muon + Parameter Banking — 81.87ms/step, val_bpb 1.1247 (3-seed mean) #399 by @abaybektursun
• TTT recipe: PR Non-record: 11L Depth Recurrence + High-Yield Legal TTT (1.14458 BPB) #461 by @Christopher-Lee-McClendon (adapted: freeze=0)
• Base model: PR Record: 11L EMA + GPTQ-lite + warmdown3500 + QAT@0.15 (val_bpb=1.1233) #414 by @signalrush

@0hq or @valerio-oai
Hey @0hq, I've applied for the Development grant several times but no response yet. GitHub: AnubhavBharadwaaj. Could you help check the status?