Thanks for the contribution! We tested the custom tokenizer on our PR #1204 reproduction pipeline (parallel residuals + mini depth recurrence + mixed int5/int6 GPTQ) and wanted to share findings.

Results

We ran on 8×H100 SXM (GCP) with seed 1337:

The custom tokenizer produces +0.002 BPP worse results on the same architecture. The +2.82% more validation tokens means the model needs to predict ~1.75M additional tokens with the same number of parameters, which creates a BPP handicap that the improved token boundaries don't fully compensate for.

Suggestions for improvement

1. Reduce token count: The current tokenizer creates more tokens than SP1024 (63.8M vs 62.0M). A tokenizer that creates fewer tokens per byte while maintaining coverage would be more competitive — see PR Record: Scylla (novel tokenizer) + Legal Score-First TTT (val_bpb: 1.08056553) #1143 (Scylla/TokenMonster) which uses a 998-token vocab and achieves significant BPP gains through token efficiency.

2. Model co-optimization: Simply swapping the tokenizer without retuning the model (BigramHash primes, LR schedule, warmdown) leaves potential gains on the table. The hash function constants (36313, 27191, etc.) were tuned for SP1024 — they may not be optimal for different token distributions.

3. Byte accounting verification: Per the README: "If changes are made to the tokenizer, prove with certainty that the val_bpb is correctly calculated." We verified that build_sentencepiece_luts() produces correct byte counts for this tokenizer (the U+2581 leading space token exists as expected), but note this is something organizers will scrutinize carefully.

Community rules compliance (Issue #1017)

This PR appears to comply with all four conditions:

• ✅ Condition 1 (Causality): Standard autoregressive prediction, no future token dependence
• ✅ Condition 2 (Full normalized distribution): Standard softmax over full vocab
• ✅ Condition 3 (Score-before-update): No TTT, no eval-time adaptation
• ✅ Condition 4 (Single L→R pass): Standard sliding window evaluation

The tokenizer change itself is explicitly permitted by the README ("novel tokenizers" are listed as an encouraged direction), subject to extra scrutiny on BPP calculation correctness.

The main concern is not legality but competitiveness — the +2.82% token overhead is a significant handicap that would need to be overcome through model quality improvements or a tokenizer redesign that reduces rather than increases token count.

Thanks for sharing the dataset and tokenizer — this kind of open contribution helps the community explore the tokenizer direction!