Stack allocate small strings by andreer · Pull Request #270 · agronholm/cbor2

andreer · 2026-01-02T11:29:10Z

Changes

Do temporary allocations for small strings on the stack, and avoid creating an
intermediate pybytes object. This improves performance, around 9-17%, with no
regressions in any of my testing.

Also fixes #255, and removes a nonexistent "error" string handler, since it was in the same code.

Checklist

You've added tests (in tests/) which would fail without your patch
You've updated the documentation (in docs/), in case of behavior changes or new
features
You've added a new changelog entry (in docs/versionhistory.rst).

'error' was never a valid Python string error handler. Accept it for backwards compatibility but normalize to 'strict' internally. Update error messages to only mention valid options.

The C implementation of decode_definite_short_string was not respecting the str_errors setting - it always used PyUnicode_FromStringAndSize which only supports strict mode. Now uses PyUnicode_DecodeUTF8 with the str_errors field passed directly. Store str_errors as const char* (NULL for strict, "replace" for replace) instead of PyObject*. This eliminates a conditional in the hot path since PyUnicode_DecodeUTF8 accepts NULL to mean strict mode. Fixes agronholm#255

Skip creating an intermediate Python bytes object by using fp_read directly into a PyMem_Malloc buffer. This avoids Python allocator and reference counting overhead.

Use a stack buffer for strings <= 256 bytes to avoid heap allocation overhead. Larger strings continue to use PyMem_Malloc.

for more information, see https://pre-commit.ci

coveralls · 2026-01-02T11:31:31Z

coverage: 94.565% (+0.02%) from 94.55%
when pulling ad6cc16 on andreer:stack-allocate-small-strings-v4
into a7ac10d on agronholm:master.

topher200

Overall, LGTM! It definitely solves #255 for smaller strings, which very important to me 😍

topher200 · 2026-01-03T17:17:50Z

source/decoder.c

I found a test which fails, even on this PR:

def test_str_errors_long_string_over_65536_bytes(impl): """Issue #255: str_errors not respected for strings >65536 bytes.""" # 65537 bytes: 65536 'a' + 1 invalid UTF-8 byte payload = unhexlify("7a00010001" + "61" * 65536 + "c3") result = impl.loads(payload, str_errors="replace") assert len(result) == 65537 assert result[-1] == "\ufffd"

Claude came up with this fix. I'm not going to stand by it, but it does make the test pass:

diff --git a/source/decoder.c b/source/decoder.c index 3751e7a..8518a54 100644 --- a/source/decoder.c +++ b/source/decoder.c @@ -905,7 +905,7 @@ decode_definite_long_string(CBORDecoderObject *self, Py_ssize_t length) } consumed = chunk_length; // workaround for https://github.com/python/cpython/issues/99612 - string = PyUnicode_DecodeUTF8Stateful(source_buffer, chunk_length, NULL, &consumed); + string = PyUnicode_DecodeUTF8Stateful(source_buffer, chunk_length, self->str_errors, &consumed); if (!string) goto error; @@ -946,9 +946,28 @@ decode_definite_long_string(CBORDecoderObject *self, Py_ssize_t length) chunk = NULL; } + // Process any remaining buffered bytes (e.g., incomplete multi-byte UTF-8 sequences) + if (buffer_length > 0) { + string = PyUnicode_DecodeUTF8(buffer, buffer_length, self->str_errors); + if (!string) + goto error; + + if (ret) { + PyObject *joined = PyUnicode_Concat(ret, string); + Py_DECREF(string); + if (!joined) + goto error; + ret = joined; + } else { + ret = string; + } + } + if (ret && string_namespace_add(self, ret, length) == -1) goto error; + if (buffer) + PyMem_Free(buffer); return ret; error: Py_XDECREF(ret);

I'll let you choose if you want to fix this issue. This was also not fixed on my PR; I think all the strings I decode are on the smaller side. I found the issue while checking your PR.

Thanks, I'll take a look tomorrow. Funny that we did this at the same time for something that's been there so long 😄

agronholm

I have a couple questions to start with.

tests/test_decoder.py

agronholm · 2026-01-03T21:43:43Z

I've been looking into rewriting this library in Rust, and allocating strings on the stack was one issue that I had to look into, as the Rust String::new function always allocates a string on the heap. While it's not clear to me yet if I really need a Rust String in this case, I found the heapless crate that should let me stack allocate plenty of useful types - once I get that far along.

andreer · 2026-01-04T11:18:35Z

Seems there's another pre-existing memory leak here too, when there is an utf-8 sequence that spans chunk boundaries. That's not really valid cbor, but I'll try to fix it while I'm in there

  import tracemalloc
  import cbor2

  # 65535 'a' + "€" (E2 82 AC) = 65538 bytes
  # Chunk 1: 65535 'a' + E2 (incomplete, buffer allocated)
  # Chunk 2: 82 AC (completes €, buffer_length=0 but buffer never freed)

  payload = b"\x7a\x00\x01\x00\x02" + b"a" * 65535 + "€".encode()

  tracemalloc.start()
  for i in range(10000):
      cbor2.loads(payload)
  current, peak = tracemalloc.get_traced_memory()
  tracemalloc.stop()

  print(f"Leaked: {current / 1024 / 1024:.0f} MB")  # ~625 MB

agronholm · 2026-01-04T12:30:56Z

Seems there's another pre-existing memory leak here too, when there is an utf-8 sequence that spans chunk boundaries. That's not really valid cbor, but I'll try to fix it while I'm in there

  import tracemalloc
  import cbor2

  # 65535 'a' + "€" (E2 82 AC) = 65538 bytes
  # Chunk 1: 65535 'a' + E2 (incomplete, buffer allocated)
  # Chunk 2: 82 AC (completes €, buffer_length=0 but buffer never freed)

  payload = b"\x7a\x00\x01\x00\x02" + b"a" * 65535 + "€".encode()

  tracemalloc.start()
  for i in range(10000):
      cbor2.loads(payload)
  current, peak = tracemalloc.get_traced_memory()
  tracemalloc.stop()

  print(f"Leaked: {current / 1024 / 1024:.0f} MB")  # ~625 MB

Adding tests that already pass is not a sin in itself, but they would need to at least increase the coverage. We don't have coverage tracking for the C code, and I don't know how to even do that.

- Pass str_errors to PyUnicode_DecodeUTF8Stateful (fixes agronholm#255) - Handle remaining bytes after loop for incomplete UTF-8 at string end - Fix reference leak: DECREF old ret before reassigning to joined - Fix memory leak: free buffer in success path

cbor2/_decoder.py

agronholm · 2026-03-21T22:38:33Z

I seems like I made a poo-poo with the merge from master. I maybe too tired to fix it right now, so I'll look at this with fresh eyes tomorrow.

agronholm · 2026-03-22T15:24:59Z

Thanks!

andreer added 5 commits January 2, 2026 11:01

normalize str_errors='error' to 'strict'

9cca4a3

'error' was never a valid Python string error handler. Accept it for backwards compatibility but normalize to 'strict' internally. Update error messages to only mention valid options.

use fp_read directly in string decoder

db18cb5

Skip creating an intermediate Python bytes object by using fp_read directly into a PyMem_Malloc buffer. This avoids Python allocator and reference counting overhead.

stack allocate small strings in C decoder

05253aa

Use a stack buffer for strings <= 256 bytes to avoid heap allocation overhead. Larger strings continue to use PyMem_Malloc.

add changelog entry for str_errors fix and string decoding optimization

3088b54

andreer changed the title ~~Stack allocate small strings v4~~ Stack allocate small strings Jan 2, 2026

[pre-commit.ci] auto fixes from pre-commit.com hooks

8544cef

for more information, see https://pre-commit.ci

andreer added 2 commits January 2, 2026 12:32

fix mypy type error for str_errors property

f6d83a8

fix pull request reference in version history

2098306

agronholm mentioned this pull request Jan 3, 2026

Fix str_errors parameter regression in C extension (#255) #271

Closed

3 tasks

topher200 approved these changes Jan 3, 2026

View reviewed changes

agronholm reviewed Jan 3, 2026

View reviewed changes

tests/test_decoder.py Outdated Show resolved Hide resolved

tests/test_decoder.py Outdated Show resolved Hide resolved

andreer added 3 commits January 12, 2026 14:09

remove superfluous tests

8d4ac77

fix missed reference to str_errors

b7bd728

agronholm reviewed Mar 2, 2026

View reviewed changes

cbor2/_decoder.py Outdated Show resolved Hide resolved

Merge branch 'master' into stack-allocate-small-strings-v4

88e792a

agronholm added 2 commits March 22, 2026 17:16

Accept all error handling modes for str_errors

8274682

Fixed errors from the merge

ad6cc16

agronholm merged commit 2b53b28 into agronholm:master Mar 22, 2026
14 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Stack allocate small strings#270

Stack allocate small strings#270
agronholm merged 14 commits intoagronholm:masterfrom
andreer:stack-allocate-small-strings-v4

andreer commented Jan 2, 2026 •

edited by agronholm

Loading

Uh oh!

coveralls commented Jan 2, 2026 •

edited

Loading

Uh oh!

topher200 left a comment

Uh oh!

topher200 Jan 3, 2026

Uh oh!

andreer Jan 3, 2026

Uh oh!

agronholm left a comment

Uh oh!

Uh oh!

Uh oh!

agronholm commented Jan 3, 2026

Uh oh!

andreer commented Jan 4, 2026

Uh oh!

agronholm commented Jan 4, 2026

Uh oh!

Uh oh!

agronholm commented Mar 21, 2026

Uh oh!

Uh oh!

agronholm commented Mar 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

andreer commented Jan 2, 2026 • edited by agronholm Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Checklist

Uh oh!

coveralls commented Jan 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

topher200 left a comment

Choose a reason for hiding this comment

Uh oh!

topher200 Jan 3, 2026

Choose a reason for hiding this comment

Uh oh!

andreer Jan 3, 2026

Choose a reason for hiding this comment

Uh oh!

agronholm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

agronholm commented Jan 3, 2026

Uh oh!

andreer commented Jan 4, 2026

Uh oh!

agronholm commented Jan 4, 2026

Uh oh!

Uh oh!

agronholm commented Mar 21, 2026

Uh oh!

Uh oh!

agronholm commented Mar 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

andreer commented Jan 2, 2026 •

edited by agronholm

Loading

coveralls commented Jan 2, 2026 •

edited

Loading