System Info

• transformers version: 4.30.2
• Platform: Linux-5.4.0-146-generic-x86_64-with-glibc2.35
• Python version: 3.11.3
• Huggingface_hub version: 0.15.1
• Safetensors version: 0.3.1
• PyTorch version (GPU?): 2.0.1+cu117 (False)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?: (NA)
• Using distributed or parallel set-up in script?: (NA)

Who can help?

@arthu

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

In [1]: from transformers import AutoTokenizer

In [2]: tokenizer = AutoTokenizer.from_pretrained("./models/t5-base/")

In [3]: tokenizer.add_tokens(["asdfg"], special_tokens=False)
Out[3]: 1

In [4]: tokenizer.tokenize("asdfgwordtimeasdfgtime")
Out[4]: ['asdfg', '▁word', 'time', 'asdfg', '▁time']

Expected behavior

tokenizer return ['asdfg', 'word', 'time', 'asdfg', 'time']