Motivation

The CachedTokenizer's L0 cache layer was previously using only the raw input string as a lookup key. This caused a key collision when the same string was tokenized with different values for the add_special_tokens flag (e.g., once for an embedding request where special tokens are required, and once for a chat length check where they are omitted). The second call would incorrectly return the cached result of the first, leading to incorrect token IDs and counts.

Modifications

• src/tokenizer/cache/l0.rs: Updated the internal DashMap to use a composite key of (String, bool) representing the input text and the add_special_tokens flag. Updated get, insert, and insert_arc methods to handle this new key format.
• src/tokenizer/cache/mod.rs: Modified the encode method to pass the add_special_tokens flag into the L0 cache lookup and insertion logic. Updated documentation comments to reflect that the cache is now parameter-aware.
• Tests: Updated unit tests in l0.rs to reflect the new method signatures.

Accuracy Tests

Created a new integration test tests/tokenizer/cache_collision_test.rs to verify the fix. The test confirms:

1. Isolation: Tokenizing the same string with add_special_tokens=true followed by add_special_tokens=false results in two distinct cache misses, proving they no longer collide.
2. Correctness: A third call using a previously cached combination (e.g., true) results in a cache hit.

Test Result:

test tokenizer::cache_collision_test::test_l0_cache_key_collision ... ok

Benchmarking and Profiling

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review Process

1. Ping Merge Oncalls to start the PR flow. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
   • /tag-run-ci-label, /rerun-failed-ci, /tag-and-rerun-ci
4. After green CI and required approvals, ask Merge Oncalls to merge.