Skip to content

Rust conversion#278

Merged
agronholm merged 154 commits intomasterfrom
rust
Mar 25, 2026
Merged

Rust conversion#278
agronholm merged 154 commits intomasterfrom
rust

Conversation

@agronholm
Copy link
Copy Markdown
Owner

@agronholm agronholm commented Feb 4, 2026 •
edited
Loading

Changes

This replaces the encoder and decoder code (Python and C) with a new Rust-based implementation. This was done to
reduce the maintenance cost (by having only a single implementation) and to prevent memory corruption
issues and vulnerabilities that have been appearing in the C version.

The Rust version also supports free-threading and subinterpreters out of the box thanks to native multi-stage initialization
and synchronization.

There is also a significant performance boost in both decoding and encoding, up to 400%, partly due to
using internal buffers more than reading/writing from/to files which cause round-trips to the Python interpreter.

Finally, it implements tags 52 and 54 for the official IP address/network/interface
specification. The deprecated tags 260 and 261 are still supported but only for decoding.
Additionally, tag 261 will now decode to a IPv4Interface or IPv6Interface if there are host bits present in the address.

Fixes #286. Fixes #285. Fixes #272. Fixes #257. Closes #232. Closes #226. Closes #224. Closes #192. Closes #169. Fixes #157. Closes #41.

Checklist

If this is a user-facing code change, like a bugfix or a new feature, please ensure that
you've fulfilled the following conditions (where applicable):

  • You've added tests (in tests/) which would fail without your patch
  • You've updated the documentation (in docs/), in case of behavior changes or new
    features
  • You've added a new changelog entry (in docs/versionhistory.rst).

If this is a trivial change, like a typo fix or a code reformatting, then you can ignore
these instructions.

Updating the changelog

If there are no entries after the last release, use **UNRELEASED** as the version.
If, say, your patch fixes issue #123, the entry should look like this:

- Fix big bad boo-boo in the encoder
  (`#123 <https://github.com/agronholm/cbor2/issues/123>`_; PR by @yourgithubaccount)

If there's no issue linked, just link to your pull request instead by updating the
changelog after you've created the PR.

agronholm and others added 30 commits January 11, 2026 02:19
@agronholm
WIP Rust conversion
9c14eff
@agronholm
Milestone: 42 failing encoder tests left
0b4e5d5
@mschwager @agronholm
Moved OSS-Fuzz target file under tests (#234)
436a413
@agronholm
Fixed docs not building
fccb039
@agronholm
Moved SECURITY.md to .github
2749f53
@agronholm
Added funding link and Tidelift badge
01f0c60
@agronholm
Updated script to match single implementation
128cf2d
@agronholm
Fixed types and their tests
4d786c4
@agronholm
Added missing types and functions to the cbor2 module
daf7bb7
@agronholm
Removed obsolete script
3add3c0
@agronholm
Removed debugging code
56b1138
@agronholm
Merge branch 'master' into rust
c120f5b
@agronholm
Implemented encoding for many more Python types
ffb4c07
@agronholm
Implemented RFC 9164 compatible encoding of IP addresses and prefixes
a64137c
@agronholm
Updated the benchmark script
76e3fa5
@agronholm
Fixed timezone being replaced with default (+other misc changes)
c7e991a
@agronholm
Fixed value sharing
518b7af
@agronholm
Fixed calling the default encoder hook raising PyBorrowMutError
70800b5
@agronholm
Started implementing decoders
e5d63d6
@agronholm
Implemented more decoders
9b3e027
@agronholm
Implemented decoders for epoch datetime, big integers and decimals
9fa3e31
@agronholm
Implemented many more decoders and updated the docs
65ec4fa
@agronholm
Fixed several test failures
9268b55
@agronholm
Updated the ruff alias to ruff-check
250496a
@agronholm
All encoder tests pass now
40b4910
@agronholm
Removed debugging code and fixed decoder test parametrization
0c82bd5
@agronholm
All tests pass now
0d879a6
@agronholm
Fixed the speed test
546d7f1
@agronholm
Updated pyo3 and dependencies
1496d34
@agronholm
Don't use BytesIO in the encoder with dumps()
897c5f0
@agronholm
Copy link
Copy Markdown
Owner Author

agronholm commented Mar 21, 2026 •
edited
Loading

Updated benchmarks after switching to iterative decoding and optimizing string decoding in strict mode:

                  +-----------------------------------------------+-----------------------------------------------+    
                  |                      Encoding                 |                  Decoding                     |     
 +----------------+----------+-----------+-----------+------------+----------+------------+----------+------------+     
 | Test           | cbor     | c-cbor2   | py-cbor2  | rust-cbor2 | cbor     | c-cbor2    | py-cbor2 | rust-cbor2 |     
 +----------------+----------+-----------+-----------+------------+----------+------------+----------+------------+     
 |           None | 102.9 ns |    1.7 µs |    1.8 µs |   153.5 ns |  72.5 ns |   641.5 ns |   1.7 µs |    81.6 ns |     
 |           10e0 | 120.8 ns |    1.7 µs |    2.1 µs |   233.3 ns |  71.7 ns |   640.6 ns |   1.7 µs |    86.1 ns |     
 |          10e12 | 136.0 ns |    1.7 µs |    2.1 µs |   274.8 ns |  82.4 ns |   659.7 ns |   2.0 µs |    99.0 ns |     
 |          10e29 | 655.7 ns |    2.0 µs |    2.9 µs |   349.2 ns | 458.3 ns |   863.8 ns |   2.9 µs |   160.8 ns |     
 |          -10e0 | 121.0 ns |    1.7 µs |    2.1 µs |   251.6 ns |  72.4 ns |   671.3 ns |   1.7 µs |    99.7 ns |     
 |         -10e12 | 135.1 ns |    1.7 µs |    2.2 µs |   335.6 ns |  85.8 ns |   694.9 ns |   2.0 µs |   110.0 ns |     
 |         -10e29 | 712.4 ns |    2.0 µs |    2.9 µs |   409.4 ns | 483.2 ns |   906.6 ns |   3.1 µs |   198.1 ns |     
 |         float1 | 134.6 ns |    1.7 µs |    2.1 µs |   186.0 ns |  81.1 ns |   648.2 ns |   2.0 µs |    96.0 ns |     
 |         float2 | 134.2 ns |    1.7 µs |    2.1 µs |   187.4 ns |  79.2 ns |   644.0 ns |   2.0 µs |    95.4 ns |     
 |            str | 156.7 ns |    1.7 µs |    2.2 µs |   176.2 ns |  84.6 ns |   687.3 ns |   2.0 µs |   106.4 ns |     
 |         bigstr | 466.7 ns |    3.0 µs |    2.6 µs |   480.1 ns | 461.1 ns |     1.4 µs |   2.9 µs |   571.7 ns |     
 |          bytes | 134.1 ns |    1.7 µs |    2.1 µs |   179.6 ns |  83.3 ns |   680.2 ns |   2.0 µs |   102.0 ns |     
 |       bigbytes | 261.2 ns |    2.0 µs |    2.3 µs |   447.9 ns | 149.4 ns |   871.2 ns |   2.5 µs |   303.8 ns |     
 |       datetime |        - |    2.5 µs |    4.2 µs |     1.8 µs | 565.5 ns |     1.1 µs |   3.5 µs |   280.3 ns |     
 |        decimal |        - |    3.4 µs |    6.4 µs |     1.3 µs | 584.7 ns |     1.4 µs |   4.2 µs |   851.8 ns |     
 |       fraction |        - |    3.6 µs |    6.8 µs |     1.2 µs | 588.5 ns |     1.1 µs |   4.2 µs |   485.9 ns |     
 |        intlist | 200.4 ns |    1.9 µs |    3.9 µs |   795.7 ns | 112.9 ns |   702.4 ns |   2.9 µs |   163.0 ns |     
 |     bigintlist |  60.1 µs |  103.0 µs |  656.1 µs |   245.5 µs |  25.4 µs |    36.1 µs | 856.7 µs |    37.5 µs |     
 |        strlist | 307.6 ns |    2.0 µs |    4.1 µs |   619.5 ns | 151.0 ns |   811.0 ns |   3.7 µs |   229.1 ns |     
 |     bigstrlist | 141.2 µs |  168.9 µs |  807.2 µs |   110.5 µs |  62.9 µs |   104.2 µs |   1.5 ms |   100.0 µs |     
 |           dict | 374.2 ns |    2.2 µs |    4.9 µs |   917.7 ns | 186.7 ns |   881.7 ns |   4.5 µs |   276.7 ns |     
 |        bigdict | 204.7 µs |  128.8 µs |  595.4 µs |   244.8 µs | 231.1 µs |   292.2 µs |   1.4 ms |   205.3 µs |     
 |            set |        - |    2.0 µs |    4.7 µs |     1.2 µs | 612.2 ns |   789.5 ns |   3.7 µs |   339.4 ns |     
 |         bigset |        - |   45.0 µs |  253.5 µs |    90.7 µs |  14.5 µs |    24.2 µs | 454.0 µs |    27.2 µs |     
 |    bigdictlist | 822.0 µs |    1.2 ms |    5.6 ms |     1.4 ms | 695.8 µs |  1023.0 µs |  10.3 ms |   844.2 µs |     
 |     objectdict |        - |    3.1 µs |    7.8 µs |     2.3 µs | 851.9 ns |     1.4 µs |   7.9 µs |   660.9 ns |     
 | objectdictlist |        - |  131.0 µs |  569.6 µs |   199.1 µs |  72.6 µs |    73.9 µs | 635.4 µs |    47.8 µs |     
 +----------------+----------+-----------+-----------+------------+----------+------------+----------+------------+

A massive speedup was achieved by this effort, cutting run times by 50-66%. Most importantly, the Rust version no longer segfaults due to stack overflow, as stack frames are stored on the heap.

@agronholm agronholm mentioned this pull request Mar 22, 2026
Closed
3 tasks
@agronholm agronholm marked this pull request as ready for review March 23, 2026 17:05
This was referenced Mar 24, 2026
Merged
Merged
@agronholm
Copy link
Copy Markdown
Owner Author

agronholm commented Mar 25, 2026

Benchmarks after the latest changes:

                 +-----------------------------------------------+-----------------------------------------------+
                 |                      Encoding                 |                  Decoding                     |
+----------------+----------+-----------+-----------+------------+----------+------------+-----------+-----------+
| Test           | cbor     | c-cbor2   | py-cbor2  | rust-cbor2 | cbor     | c-cbor2    | py-cbor2 | rust-cbor2 |
+----------------+----------+-----------+-----------+------------+----------+------------+-----------+-----------+
|           None | 102.9 ns |    1.9 µs |    1.7 µs |   161.7 ns |  72.5 ns |   585.3 ns |  978.3 ns |   82.4 ns |
|           10e0 | 120.8 ns |    1.8 µs |    1.9 µs |   238.2 ns |  71.7 ns |   572.9 ns | 1008.8 ns |   84.4 ns |
|          10e12 | 136.0 ns |    1.9 µs |    1.9 µs |   285.3 ns |  82.4 ns |   631.7 ns |    1.1 µs |   98.4 ns |
|          10e29 | 655.7 ns |    2.3 µs |    2.5 µs |   363.5 ns | 458.3 ns |   875.0 ns |    1.6 µs |  160.4 ns |
|          -10e0 | 121.0 ns |    1.8 µs |    1.9 µs |   263.3 ns |  72.4 ns |   595.2 ns |    1.0 µs |   99.2 ns |
|         -10e12 | 135.1 ns |    1.9 µs |    2.0 µs |   352.5 ns |  85.8 ns |   685.7 ns |    1.2 µs |  108.4 ns |
|         -10e29 | 712.4 ns |    2.4 µs |    2.5 µs |   434.8 ns | 483.2 ns |   917.5 ns |    1.7 µs |  216.8 ns |
|         float1 | 134.6 ns |    1.9 µs |    1.8 µs |   197.4 ns |  81.1 ns |   624.6 ns |    1.1 µs |   94.6 ns |
|         float2 | 134.2 ns |    1.9 µs |    1.8 µs |   200.5 ns |  79.2 ns |   624.4 ns |    1.1 µs |   96.5 ns |
|            str | 156.7 ns |    1.9 µs |    1.9 µs |   189.6 ns |  84.6 ns |   635.6 ns |    1.1 µs |  108.9 ns |
|         bigstr | 466.7 ns |    2.2 µs |    2.2 µs |   495.3 ns | 461.1 ns |     1.2 µs |    1.8 µs |  540.8 ns |
|          bytes | 134.1 ns |    1.9 µs |    1.9 µs |   190.5 ns |  83.3 ns |   639.4 ns |    1.1 µs |  104.7 ns |
|       bigbytes | 261.2 ns |    2.1 µs |    2.0 µs |   463.5 ns | 149.4 ns |   838.0 ns |    1.4 µs |  274.7 ns |
|       datetime |        - |    2.8 µs |    3.6 µs |     1.8 µs | 565.5 ns |     1.1 µs |    2.5 µs |  253.6 ns |
|        decimal |        - |    3.9 µs |    5.6 µs |     1.3 µs | 584.7 ns |     1.5 µs |    2.6 µs |  887.7 ns |
|       fraction |        - |    3.9 µs |    6.0 µs |     1.2 µs | 588.5 ns |     1.0 µs |    2.3 µs |  497.0 ns |
|        intlist | 200.4 ns |    2.1 µs |    3.3 µs |   813.8 ns | 112.9 ns |   724.8 ns |    1.8 µs |  174.1 ns |
|     bigintlist |  60.1 µs |  110.1 µs |  426.1 µs |   254.5 µs |  25.4 µs |   100.2 µs |  629.4 µs |   40.1 µs |
|        strlist | 307.6 ns |    2.2 µs |  3.4   µs |   627.4 ns | 151.0 ns |   866.9 ns |    2.1 µs |  237.8 ns |
|     bigstrlist | 141.2 µs |  171.6 µs |  508.4 µs |   112.0 µs |  62.9 µs |   229.7 µs |  958.0 µs |  101.5 µs |
|           dict | 374.2 ns |    2.3 µs |  4.0   µs |   946.5 ns | 186.7 ns |   958.6 ns |    2.7 µs |  287.5 ns |
|        bigdict | 204.7 µs |  138.4 µs |  499.9 µs |   248.5 µs | 231.1 µs |   337.6 µs |  963.7 µs |  205.5 µs |
|            set |        - |    2.2 µs |  3.9   µs |     1.2 µs | 612.2 ns |   876.8 ns |    2.3 µs |  302.1 ns |
|         bigset |        - |   51.6 µs |  176.1 µs |    92.3 µs |  14.5 µs |    72.2 µs |  317.6 µs |   27.2 µs |
|    bigdictlist | 822.0 µs |    1.2 ms |  3.9   ms |     1.4 ms | 695.8 µs |     1.9 ms |    6.8 ms |  851.2 µs |
|     objectdict |        - |    3.4 µs |  6.3   µs |     2.3 µs | 851.9 ns |     1.8 µs |    5.3 µs |  670.8 ns |
| objectdictlist |        - |  142.8 µs |  445.9 µs |   204.1 µs |  72.6 µs |   111.9 µs |  439.7 µs |   48.3 µs |
+----------------+----------+-----------+-----------+------------+----------+------------+-----------+-----------+

@agronholm agronholm merged commit 726cad5 into master Mar 25, 2026
15 checks passed
@agronholm agronholm deleted the rust branch March 25, 2026 22:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment