However, this change makes read_index unused, so I am not entirely certain if it aligns with the intended logic of the original implementation. Please let me know if further adjustments or clarifications are needed, or if there is additional context regarding the intended use of read_index.

I'm very confused by my own implementation re-reading it. Your fix is definitely correct and we can remove than internal pointer.
The prefix/read difference stems from another implementation I had which tried to keep all "side effects contained (in the read segment part, which isn't ever compared) and keep the prefix segment distinct.
However in this implementation because we recalculate the prefix *prefix = tokenizer.decode(..) then we re-encapsulate the side effect which is simpler.

What's weird is that I remember having only that in my first iteration of this. I have no idea why I re-added half-brokenly that read_index.

Thanks a lot for the fix.

Note: I will rewrite your unit test. They are good but I find them not really linearly readable and also panicking within tests is perfectly fine (it's a failure).