SharpToken seems to be actively maintained, we are evaluating a possible switch from TiktokenSharp. I'm interested in whether the performance is the same or how SharpToken performs compared to TiktokenSharp and Tokenizer. Quickly gazing through the codebase, core BPE is not taking advantage of SSE, AVX2/512, or other SIMD implementations (like arm neon) yet.

Do you know if any of this is planned?