The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

You need to run python stub.py !

Awesome! 💯

Hmm, having a bit of trouble here? 😅

• I had to make some changes because running python stub.py gives the error python: can't open file '/Users/stevhliu/tokenizers/stub.py': [Errno 2] No such file or directory.
• Instead, I run the command python bindings/python/stub.py and it creates a new py_src/tokenizers/ directory (with a bunch of empty files) instead of writing the changes to the existing one.
• So I modify stub.py to write the changes to "bindings/python/py_src/tokenizers/" but then it just overwrites and replaces everything in those files with a blank line which doesn't seem right.

I'll merge this a fix in a follow-up stub.py is right even if it's just a newline.

Oh you may have forgotten to rebuild tokenizers, stub.py looks at the binary and extracts the pyi from the built binary, so if your binary is outdated you may not see the new strings.