* workflows

* fix

* fix

* fix

* fix

* fix

* fix

* fix 1

* fix 2

* fix

* fix4

* fix

* dotnet-version fix

* needs: call-reusable-build-test

* actions/checkout@v4

* actions/setup-dotnet@v4

…rmance improvements) (#36)

* feat(performance): add benchmark project

* feat(performance): reduce public API surface

* feat(performance): reduce allocations

* feat(performance): use compiled regex for better performance

* feat(performance): run benchmark

* chore: move reorganize files

* feat(performance): replace SpecialTokenPatternRegex with faster alloc free solution

* feat(performance): reduce string allocations

* feat(performance): run benchmark

* feat(performance): add BytePairIndex class to support faster implementation in net8.0

* feat(performance): cache model parameters to do params preparation only once

* feat(performance): run benchmark

* feat(performance): improve allowedSpecialTokens handling + bug fix

* feat(performance): add support for ReadOnlySpan<char> in net8.0

* feat(performance): run benchmark

* feat(performance): use compile time generated regex in net8.0

* feat: run tests for netstandard2.0 over net471

* feat(performance): run benchmark

* feat: remove unused lastTokenLength + refactoring

* feat(performance): implement fast MultiBytePairEncoder with almost zero allocations

* feat(performance): run benchmark

* feat(performance): reduce allocations

* feat(performance): backport some optimizations to net6.0 and netstandard

* feat(performance): run benchmark

* chore: cosmetics

* feat(refactor): extract Encoding.Encode() logic into priv EncodeCore to support broader use

* feat(token-count): implement low allocation token count public method

* feat(performance): run benchmark

* feat(benchmark): don't make allocations in benchmark methods

* feat(performance): reduce minor allocations

* feat(benchmark): add benchmark for large file token count

* feat(performance): run benchmark

* fix: add test for allowedSpecialTokens and fix code

* chore: fix naming

* feat(benchmark): add another benchmark to show in README.md

* feat(readme): add benchmark to README.md and add docs for TokenCount method

* feat(performance): use multibyte cpu instructions for FastPartitionList.RemoveAt

* feat(performance): re-add fast path - got lost in refactoring

* feat(benchmark): add comparison to other tokenizer

* feat(performance): improve ByteArrayEqualityComparer

* feat(performance): small improvement

* chore: update benchmarks in README.md

* fix

* improved pipleline

* switch (key)

* i += size - 1;

* CountTokens renamed

---------

Co-authored-by: René Larch <renelarch@gmail.com>

* Adding o200k_base.tiktoken

* Support for o200k_base and gpt-4o (omni)

* Fixing typo

Accidentally removed a using statement in my last update

* Fixed issue with some tests

Still failing in a handful of tests of the new o200k

* All tests now passing (fixed typo)

* architecture: x64  # Add this line

* attempt

* Update dotnet-build-test.yml

---------

Co-authored-by: Tom Winzig <winzig@users.noreply.github.com>
Co-authored-by: Tom Winzig <thomas@winzig.com>