we don't check the tokenization of all the model paths we have in the transformers repo. Fixes #45488, Related to #44255

We have models that don't have their own dedicated Tokenizer class and use another model's tokenizer (ex. Granite which uses GPT2Tokenizer - related issue: #45813) ). The different model tokenizer class would be mapped in the tokenization_auto.py mapping, or in the tokenization_config.json. Sometimes the mapped tokenizer isn't actually the one that is being used, and v5 surfaced these incorrect mappings. In order to "stay true to" the pre-v5 behavior of these models, we can map them to TokenizersBackend (eq. to PreTrainedTokenizerFast in v4) which loads the tokenizer.json as is. This happens because in v5 we actually try to load the mapped tokenizer class and force the same tokenizer type.

Anyway we only test tokenization of models that have their own tokenizer class but we should test tokenization for every checkpoint we have in the repo!

This PR

tools/tokenizer_compare/run_equivalence_comparison_all_checkpoints.py script

(based on that in #44255)

python tools/tokenizer_compare/run_equivalence_comparison_all_checkpoints.py
  --rerun-from-results tools/tokenizer_compare/output/tokenizer_compare_result.json
  --output output/tokenizer_compare_results_rerun.json

scans tests/models/test_modeling_*.py for .from_pretrained(...) and extracts all the checkpoint paths we list, and compares _tokenizers loaded via AutoTokenizer.from_pretrained vs TokenizersBackend.from_pretrained.

tools/tokenizer_compare/compare_tokenizer_xlni_roundtrip.py script

check if TokenziersBackend and AutoTokenizer behave the same way, producing same roundtrip results on xlni and generate the same token ids

python tools/tokenizer_compare/compare_tokenizer_xlni_roundtrip.py bigcode/tiny_starcoder_py v1 --backend tokenizers --compare-backend auto

Report

PRE this PR / Fixes - on all the checkpoints we list: report

POST this PR / Fixes: report

Remaining patterns:

Pattern 1 & 2

Expected and safe to ignore

• these behave identically as per (tools/tokenizer_compare/compare_tokenizer_xlni_roundtrip.py). These are legacy llama (already caught here: [vllm + v5 fix] handle TokenizersBackend fallback properly for v5 #44255)

Pattern 3

• opened a PR on the hub https://huggingface.co/allenai/dolma2-tokenizer/discussions/1 , there is no config.json so we cannot force TokenizersBackend from tokenization_auto.py

Pattern 4

• opened a PR on the hub https://huggingface.co/google/umt5-small/discussions/2 google/umt5-small

When we need precompiled

the charsmap is unordered and when we use a tokenizers.normalizer it will reorder mappings (like NFKC)

TODO

adding a test that will check AutoTokenizer and TokenizersBackend loads equivalent _tokenizer objects for each path we mention in the repo