Summary

• val_bpb: 1.0321 (3-seed mean, std 0.0011) — beats current SOTA (1.1194) by 0.0873 BPB
• Replaces 659/765 merge tokens by ablation leverage scoring (β=1.0)
• Vanilla 12L 384d transformer — no SmearGate, no BigramHash, no XSA, no EMA, no TTT, no sliding window eval
• The vocabulary alone accounts for the entire improvement
• 15.6 MB artifact, ~591s training time, all constraints met with margin

3-Seed Results

Approach

At 1024 vocabulary tokens, every merge slot matters. Standard BPE allocates by frequency. The Gravity Tokenizer allocates by ablation leverage — the downstream loss increase when a token is shattered back to bytes. This is a measurement of structural importance, not frequency.

The scoring pipeline uses a frozen GPT-2 reference model to measure each candidate token's leverage across 100 FineWeb contexts. The top 765 candidates by gravity score replace the BPE merge tokens. The vocabulary size stays exactly 1024.

Tokenizer Correctness

The val_bpb calculation uses the competition's own build_sentencepiece_luts() and eval_val() functions with zero modifications. The gravity tokenizer's lower compression ratio (1.05 vs 2.45 bytes/token) results in a higher tokens_per_byte multiplier, which penalizes the gravity tokenizer. The improvement is entirely in per-token prediction quality. Detailed correctness documentation included in tokenizer_scrutiny_doc.md.

Setup

bash setup.sh   # Downloads stock FineWeb + retokenizes with gravity vocabulary

The train_gpt.py is the unmodified competition baseline. All config via env vars.

Test plan

• 3 seeds with p << 0.01 statistical significance
• All artifacts under 16,000,000 bytes
• All runs under 600 seconds on 8×H100 SXM
• Tokenizer correctness documented and defended
• Retokenization is deterministic and reproducible from stock FineWeb

🤖 Generated with Claude Code