Currently, numcodecs assumes that a buffer, a pointer and a size, containing Zstandard compressed only consists of a single frame. However, this is not necessarily the case in that a single buffer may contain multiple frames. A buffer with multiple compressed frames is easily constructed by simply concatenating two encoded buffers. Currently, when decoding this buffer numcodecs.Zstd will report that the destination buffer is too small.

In [1]: import numcodecs

In [2]: codec = numcodecs.Zstd()

In [3]: hello_world = codec.encode(b"Hello world!")

In [4]: codec.decode(hello_world*2)
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[4], line 1
----> 1 codec.decode(hello_world*2)

File numcodecs/zstd.pyx:261, in numcodecs.zstd.Zstd.decode()

File numcodecs/zstd.pyx:221, in numcodecs.zstd.decompress()

RuntimeError: Zstd decompression error: b'Destination buffer is too small'

It will even do this when providing an output buffer of sufficient size.

In [5]: dest_buffer = bytearray(len(b"Hello world!")*2)

In [6]: codec.decode(hello_world*2, out=dest_buffer)
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[6], line 1
----> 1 codec.decode(hello_world*2, out=dest_buffer)

File numcodecs/zstd.pyx:261, in numcodecs.zstd.Zstd.decode()

File numcodecs/zstd.pyx:221, in numcodecs.zstd.decompress()

RuntimeError: Zstd decompression error: b'Destination buffer is too small'

The expectation is that the decompressed output should simply the concatenation of the original uncompressed buffers.

In [7]: import pyzstd

In [8]: pyzstd.decompress(hello_world*2)
Out[8]: b'Hello world!Hello world!'

The flaw is that numcodecs.Zstd currently runs ZSTD_getFrameContentSize only once to get the uncompressed size of the first frame. It does not consider whether there may be other frames in the source buffer.

This pull request introduces a Cython function findTotalContentSize which iterates through all the frames in the buffer, gets the sizes of the uncompressed data from each frame, and then returns the sum. This allows numcodecs.Zstd to decompress the entire buffer.

>>> import numcodecs
>>> codec = numcodecs.Zstd()
>>> hello_world = codec.encode(b"Hello World!")
>>> codec.decode(hello_world*2)
b'Hello World!Hello World!'
>>> out = bytearray(len("Hello World!")*2)
>>> codec.decode(hello_world*2, out=out)
bytearray(b'Hello World!Hello World!')
>>> out
bytearray(b'Hello World!Hello World!')

One application of this is that a buffer containing multiple chunks of a Zarr v3 shard contiguously could be decompressed in a single step.

• Add support for multiple zstd frames in decompression
• Add release notes

[Description of PR]

TODO:

• Unit tests and/or doctests in docstrings
• Tests pass locally
• Docstrings and API docs for any new/modified user-facing classes and functions
• Changes documented in docs/release.rst
• Docs build locally
• GitHub Actions CI passes
• Test coverage to 100% (Codecov passes)