NormalizedString.clear() broken? #1636
Description
Hello. I think there are some problems with NormalizedString (tokenizers 0.15.2).
In the following example, append() works as expected.
from tokenizers import NormalizedString
s = NormalizedString("Hi.") # NormalizedString(original="Hi.", normalized="Hi.")
s.append("Hello.") # NormalizedString(original="Hi.", normalized="Hi. Hello.")
After using clear(), append() no longer modifies the normalized attribute.
from tokenizers import NormalizedString
s = NormalizedString("Hi.") # NormalizedString(original="Hi.", normalized="Hi.")
s.clear() # NormalizedString(original="Hi.", normalized="")
s.append("Hello.") # NormalizedString(original="Hi.", normalized="")
This is also a problem with prepend.