Add FFI bindings for tiktoken-rs by yethee · Pull Request #27 · yethee/tiktoken-php

yethee · 2025-03-30T12:54:43Z

Added an alternative implementation of the encoder using tiktoken-rs library to improve performance in some cases.

Fixes: #6, #25

Benchmark

NOTE: Memory measurement for the LibEncoder is not relevant.

> phpbench run -l dots --report=agg_by_subject --report=enc_chart --profile=jit
PHPBench (1.4.1) running benchmarks...
with configuration file: /workspace/phpbench.json
with PHP version 8.3.19, xdebug ❌, opcache ✔

................................................

Subjects: 4, Assertions: 0, Failures: 0, Errors: 0
encode
+--------------------+---------------------------------+------+-----+----------+-----------+---------+
| benchmark          | set                             | revs | its | mem_peak | mode      | rstdev  |
+--------------------+---------------------------------+------+-----+----------+-----------+---------+
| LibEncoderBench    | p50k_base,baconipsum            | 5    | 3   | 1.317mb  | 7.176ms   | ±9.33%  |
| LibEncoderBench    | cl100k_base,baconipsum          | 5    | 3   | 1.317mb  | 8.147ms   | ±1.83%  |
| LibEncoderBench    | o200k_base,baconipsum           | 5    | 3   | 1.317mb  | 8.218ms   | ±1.73%  |
| LibEncoderBench    | p50k_base,cyrillic              | 5    | 3   | 1.317mb  | 1.826ms   | ±1.51%  |
| LibEncoderBench    | cl100k_base,cyrillic            | 5    | 3   | 1.317mb  | 2.145ms   | ±0.58%  |
| LibEncoderBench    | o200k_base,cyrillic             | 5    | 3   | 1.317mb  | 1.942ms   | ±1.04%  |
| LibEncoderBench    | p50k_base,latin                 | 5    | 3   | 1.317mb  | 758.602μs | ±0.16%  |
| LibEncoderBench    | cl100k_base,latin               | 5    | 3   | 1.317mb  | 1.110ms   | ±13.83% |
| LibEncoderBench    | o200k_base,latin                | 5    | 3   | 1.317mb  | 1.510ms   | ±16.32% |
| LibEncoderBench    | p50k_base,without-whitespaces   | 5    | 3   | 4.833mb  | 2.053s    | ±1.05%  |
| LibEncoderBench    | cl100k_base,without-whitespaces | 5    | 3   | 4.833mb  | 2.413s    | ±1.77%  |
| LibEncoderBench    | o200k_base,without-whitespaces  | 5    | 3   | 2.736mb  | 31.133ms  | ±0.44%  |
| NativeEncoderBench | p50k_base,baconipsum            | 5    | 3   | 7.271mb  | 9.862ms   | ±1.86%  |
| NativeEncoderBench | cl100k_base,baconipsum          | 5    | 3   | 13.994mb | 7.598ms   | ±3.17%  |
| NativeEncoderBench | o200k_base,baconipsum           | 5    | 3   | 27.583mb | 6.488ms   | ±1.06%  |
| NativeEncoderBench | p50k_base,cyrillic              | 5    | 3   | 7.271mb  | 4.217ms   | ±5.89%  |
| NativeEncoderBench | cl100k_base,cyrillic            | 5    | 3   | 13.994mb | 4.682ms   | ±1.93%  |
| NativeEncoderBench | o200k_base,cyrillic             | 5    | 3   | 27.583mb | 3.561ms   | ±1.75%  |
| NativeEncoderBench | p50k_base,latin                 | 5    | 3   | 7.271mb  | 256.463μs | ±4.12%  |
| NativeEncoderBench | cl100k_base,latin               | 5    | 3   | 13.994mb | 274.299μs | ±1.57%  |
| NativeEncoderBench | o200k_base,latin                | 5    | 3   | 27.583mb | 299.513μs | ±13.34% |
| NativeEncoderBench | p50k_base,without-whitespaces   | 5    | 3   | 34.318mb | 49.407s   | ±0.45%  |
| NativeEncoderBench | cl100k_base,without-whitespaces | 5    | 3   | 39.993mb | 56.818s   | ±0.69%  |
| NativeEncoderBench | o200k_base,without-whitespaces  | 5    | 3   | 27.583mb | 35.300ms  | ±0.33%  |
+--------------------+---------------------------------+------+-----+----------+-----------+---------+

decode
+--------------------+---------------------------------+------+-----+----------+-----------+---------+
| benchmark          | set                             | revs | its | mem_peak | mode      | rstdev  |
+--------------------+---------------------------------+------+-----+----------+-----------+---------+
| LibEncoderBench    | p50k_base,baconipsum            | 5    | 3   | 1.317mb  | 750.609μs | ±0.91%  |
| LibEncoderBench    | cl100k_base,baconipsum          | 5    | 3   | 1.317mb  | 657.150μs | ±2.18%  |
| LibEncoderBench    | o200k_base,baconipsum           | 5    | 3   | 1.317mb  | 668.732μs | ±2.42%  |
| LibEncoderBench    | p50k_base,cyrillic              | 5    | 3   | 1.317mb  | 407.333μs | ±18.12% |
| LibEncoderBench    | cl100k_base,cyrillic            | 5    | 3   | 1.317mb  | 268.550μs | ±42.99% |
| LibEncoderBench    | o200k_base,cyrillic             | 5    | 3   | 1.317mb  | 238.260μs | ±1.02%  |
| LibEncoderBench    | p50k_base,latin                 | 5    | 3   | 1.317mb  | 105.187μs | ±8.90%  |
| LibEncoderBench    | cl100k_base,latin               | 5    | 3   | 1.317mb  | 123.266μs | ±18.81% |
| LibEncoderBench    | o200k_base,latin                | 5    | 3   | 1.317mb  | 114.973μs | ±1.97%  |
| LibEncoderBench    | p50k_base,without-whitespaces   | 5    | 3   | 3.121mb  | 3.781ms   | ±0.98%  |
| LibEncoderBench    | cl100k_base,without-whitespaces | 5    | 3   | 3.100mb  | 3.756ms   | ±2.45%  |
| LibEncoderBench    | o200k_base,without-whitespaces  | 5    | 3   | 2.031mb  | 3.669ms   | ±0.85%  |
| NativeEncoderBench | p50k_base,baconipsum            | 5    | 3   | 7.271mb  | 1.238ms   | ±0.37%  |
| NativeEncoderBench | cl100k_base,baconipsum          | 5    | 3   | 13.994mb | 1.103ms   | ±0.99%  |
| NativeEncoderBench | o200k_base,baconipsum           | 5    | 3   | 27.583mb | 954.397μs | ±2.41%  |
| NativeEncoderBench | p50k_base,cyrillic              | 5    | 3   | 7.271mb  | 700.969μs | ±4.67%  |
| NativeEncoderBench | cl100k_base,cyrillic            | 5    | 3   | 13.994mb | 337.236μs | ±3.64%  |
| NativeEncoderBench | o200k_base,cyrillic             | 5    | 3   | 27.583mb | 230.186μs | ±2.12%  |
| NativeEncoderBench | p50k_base,latin                 | 5    | 3   | 7.271mb  | 139.868μs | ±5.20%  |
| NativeEncoderBench | cl100k_base,latin               | 5    | 3   | 13.994mb | 133.633μs | ±2.12%  |
| NativeEncoderBench | o200k_base,latin                | 5    | 3   | 27.583mb | 137.276μs | ±3.20%  |
| NativeEncoderBench | p50k_base,without-whitespaces   | 5    | 3   | 32.217mb | 6.593ms   | ±1.03%  |
| NativeEncoderBench | cl100k_base,without-whitespaces | 5    | 3   | 37.892mb | 6.274ms   | ±1.01%  |
| NativeEncoderBench | o200k_base,without-whitespaces  | 5    | 3   | 27.583mb | 5.968ms   | ±4.05%  |
+--------------------+---------------------------------+------+-----+----------+-----------+---------+

ASCII text ~40k characters long:

UTF8 text ~7k characters long:

ASCII text ~6k characters long:

Text 100k characters long without any spaces:

codacy-production · 2025-03-30T12:54:47Z

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation	Diff coverage
❌ -11.29% (target: -1.00%)	✅ 69.05%

Coverage variation details

	Coverable lines	Covered lines	Coverage
Common ancestor commit (`5ce64fe`)	186	155	83.33%
Head commit (`4add7b3`)	279 (+93)	201 (+46)	72.04% (-11.29%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

	Coverable lines	Covered lines	Diff coverage
Pull request (#27)	210	145	69.05%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings Change summary preferences

flexchar · 2025-03-31T10:53:24Z

Just to be clear, lib means using FFI and native means using existing way?

yethee · 2025-03-31T11:25:39Z

@flexchar Yes, that's right.

flexchar · 2025-03-31T11:51:06Z

Out of curiosity, I'm so surprised that native is faster in certain cases. Any idea how that is possible?

yethee · 2025-03-31T17:58:15Z

Using FFI we have a performance overhead (marshalling costs). For example, strings need to be copied from C to Rust, etc.

This approach can be profitable when there are a lot of CPU-bound computations. Mainly for encoding text into tokens. In the case of decoding, both implementations are close in performance, since we only need to traverse array of tokens once and concat the string.

flexchar · 2025-03-31T20:12:36Z

Thank you for sharing your wisdom, dear person!

yethee added 2 commits March 30, 2025 18:13

Add FFI bindings for tiktoken-rs

5730812

Add benchmark for the lib encoder

1629654

yethee force-pushed the ffi branch from a9586bf to 60e619f Compare March 31, 2025 10:32

yethee marked this pull request as ready for review March 31, 2025 10:42

Update readme

4add7b3

yethee force-pushed the ffi branch from 60e619f to 4add7b3 Compare March 31, 2025 10:48

yethee merged commit 6866101 into master Mar 31, 2025
19 of 21 checks passed

yethee deleted the ffi branch March 31, 2025 10:50

yethee mentioned this pull request Mar 31, 2025

Performance degrades to quadratic for some input strings #25

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add FFI bindings for tiktoken-rs#27

Add FFI bindings for tiktoken-rs#27
yethee merged 3 commits intomasterfrom
ffi

yethee commented Mar 30, 2025 •

edited

Loading

Uh oh!

codacy-production bot commented Mar 30, 2025 •

edited

Loading

Uh oh!

Uh oh!

flexchar commented Mar 31, 2025

Uh oh!

yethee commented Mar 31, 2025

Uh oh!

flexchar commented Mar 31, 2025

Uh oh!

yethee commented Mar 31, 2025 •

edited

Loading

Uh oh!

flexchar commented Mar 31, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

yethee commented Mar 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmark

Uh oh!

codacy-production bot commented Mar 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage summary from Codacy

See diff coverage on Codacy

See your quality gate settings Change summary preferences

Uh oh!

Uh oh!

flexchar commented Mar 31, 2025

Uh oh!

yethee commented Mar 31, 2025

Uh oh!

flexchar commented Mar 31, 2025

Uh oh!

yethee commented Mar 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

flexchar commented Mar 31, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

yethee commented Mar 30, 2025 •

edited

Loading

codacy-production bot commented Mar 30, 2025 •

edited

Loading

yethee commented Mar 31, 2025 •

edited

Loading