There is a bug in offset mapping that actually affects all the fast tokenizers converted from SentencePiece. During the pre-tokenization step, we first split everything on whitespaces (WhitespaceSplit pre-tokenizer), and in a second step, we add the ▁ character in front of each word (Metaspace pre-tokenizer). This process is accurate in terms of tokenization, but it makes the offset tracking very difficult:

• All the whitespaces get removed, so we won't have any token pointing back to them.
• We add a "new" ▁ in front of each word, so these tokens actually point back to the beginning of each word: the first character.

How we fix it

The initial idea of using the WhitespaceSplit in a first step was simply to deduplicate the whitespaces but since it leads to loss of information we replace it with the following process:

• Normalization step that replaces groups of whitespaces with a single one, effectively mapping the single whitespace to the group in the original input.
• Pretokenization step: we just keep the Metaspace pre-tokenizer.

Related to huggingface/transformers#9633 and huggingface/transformers#9637