Summary

• val_bpb: 0.9485 (3-seed mean, std 0.0008)
• ~15.6 MB artifact, 8×H100 SXM, 88ms/step, ~6700 steps
• Combines Scylla tokenizer (PR Record: Scylla (novel tokenizer) + Legal Score-First TTT (val_bpb: 1.08056553) #1143) with PR Record: 1.1122 BPB — Coprime-Stride Loader + Full GPTQ + XSA-all (3-seed mean) #1060 training stack
• Beats current Update README.md little things #1 (PR Record: Scylla (novel tokenizer) + Legal Score-First TTT (val_bpb: 1.08056553) #1143, 1.0806) by 0.132 BPB (12.2%)

3-Seed Results

Key Innovation

Scylla tokenizer (998 tokens, @simon-marcus PR #1143) + modern training stack:

• Full Hessian GPTQ (Cholesky error compensation)
• XSA on all 11 layers
• Coprime-stride multi-shard loader (194 shards)
• FlashAttention 3 (Hopper native)
• No TTT needed (neutral on this stack)

PR #1143 used the old SOTA base. This submission applies the modern stack to the same tokenizer, yielding 12% better BPB.

Test plan

• 3-seed verification on 8×H100 SXM (mean 0.9485, std 0.0008)
• All artifacts under 16,000,000 bytes
• All training under 600s
• No TTT
• Tokenizer byte accounting via validated metadata (candidate.meta.npz)

Credits

• Scylla tokenizer: @simon-marcus (PR Record: Scylla (novel tokenizer) + Legal Score-First TTT (val_bpb: 1.08056553) #1143)
• Training stack: @resouer (PR Record: 1.1122 BPB — Coprime-Stride Loader + Full GPTQ + XSA-all (3-seed mean) #1060), @abaybektursun (PR Record: LeakyReLU² + Legal Score-First TTT + Parallel Muon — val_bpb 1.1194 (3-seed mean) #549)