Summary

Non-TTT submission: 11L XSA-all + LeakyReLU(0.5)² + Value Residual + Gated Attention + 7-gram backward-looking eval cache.

3-seed mean val_bpb = 1.0337 (std=0.0010) on 8xH100 SXM, 600s wallclock. Artifact ~15.99MB (int6+zstd).

3-Seed Results

Key Techniques

N-gram Cache Compliance

The 7-gram cache is a deterministic, eval-time-only statistical post-processing step:

• Score-first: Each token is scored by the model before entering the n-gram table
• Fixed alpha: alpha=0.40 is baked into the code, not tuned per-sample
• No oracle selection: Same alpha and order for every token
• Zero learned parameters: Purely statistical, built from the eval data stream
• Deterministic: Identical results regardless of hardware or random seeds

Training Config

8xH100 SXM, 600s wallclock (~5589 steps at 107ms/step)
WARMDOWN_ITERS=3000, MATRIX_LR=0.025, SCALAR_LR=0.025
XSA_LAST_N=11, LEAKY_RELU=1, EMA=0.997
NGRAM_CACHE=1, NGRAM_ALPHA=0.40, NGRAM_ORDER=7 (eval-time only)

Test plan

• 3-seed validation (seeds 1337, 42, 7) on 8xH100 SXM
• All artifacts under 16MB
• Sliding window eval (stride=64) with n-gram cache
• Logs included in logs/ directory