The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@Narsil @ArthurZucker excuse me, i am facing a memory leaks problem may caused by this. And could you help me to disable the cache in this version?
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(_path)
tokenizer.backend_tokenizer.model.cache_capacity(0)
This code cann't work.

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(_path)
tokenizer._tokenizer.model.cache_capacity(0)

where does backend_tokenizer come from?

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(_path)
tokenizer._tokenizer.model.cache_capacity(0)

where does backend_tokenizer come from?

AttributeError: 'tokenizers.models.BPE' object has no attribute 'cache_capacity'

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(_path)
tokenizer._tokenizer.model._clear_cache()

or

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(_path)
tokenizer._tokenizer.model._resize_cache(0)

You can find the functions using dir(tokenizer._tokenizer.model)

do you want to open a PR to add some doc?

do you want to open a PR to add some doc?

no, i just won't to disable cache. but i only can do it by backend_tokenizer.

tokenizer._tokenizer.model._resize_cache(0)

will disable it

backend_tokenizer does not exist sir

backend_tokenizer does not exist sir

the tokenizers generated by transformers seems have.

No, the backend_tokenizer is _tokenizer