Summary

• val_bpb: 0.7227 (single seed, 8xH100 SXM)
• val_loss: 1.2203
• Artifact: 15.45 MB (96.5% of 16MB limit)
• Training: 600s (7274 steps at 82.5ms/step)
• Eval (TTT + scoring): 569s (within 600s budget)

Architecture

• 10L, 512d, GQA 8/4, MLP 3x with ReLU-squared
• EMA (decay 0.999, every 10 steps) + SWA (11 checkpoints)
• SmearGate, BigramHash(2048), U-Net skip connections
• Late QAT, int6 uniform quantization + zstd-22
• Compiled Muon Newton-Schulz optimizer, train_seq_len=1024

LoRA TTT (per-document adaptation)

• Rank-8 LoRA on Q/V projections + rank-16 on LM-head
• Per-block bias tuning during TTT
• Per-document reset at BOS boundaries (batched 64 docs/GPU)
• Adam optimizer, lr=0.01, 6 epochs per document batch
• Per-step cosine LR decay, gradient clipping 1.0
• Post-TTT temperature rescaling (T=0.98)
• Wall-clock deadline 550s with base-model fallback for remaining docs
• Score-every-epoch (backward-looking, Issue Invalid submissions due to information leakage during TTT #402 compliant)

Our additions over base (PR #596)

• FlashAttention-3 (flash_attn_func) — 3% faster attention on H100
• Rotary cache .clone() fix — resolves CUDA graph conflict with FA3

Results breakdown

• Pre-quantization BPB: 1.1621
• Post-quantization BPB: 1.1750 (quant gap: 0.013)
• Post-TTT BPB: 0.7227 (TTT gain: 0.4522)
• Short docs (no TTT): 2393 docs, 27s
• Long docs (LoRA TTT): 3857 docs, 61 batches, 377s

Test plan

• Artifact under 16MB (15.45MB)
• Training under 600s (600s)
• Eval under 600s (569s)