Skip to content

Support arbitrary buffer-like objects #26

@pitrou

Description

@pitrou

The methods in this package (such as ZStdCompressor.compress() and ZStdDecompressor.decompress()) typically don't support bytearray or memoryview inputs. Though for some strange reason they seem to support Numpy arrays:

>>> c = zstd.ZstdCompressor(level=1, write_content_size=True)
>>> b = c.compress(b'12345678')
>>> b == c.compress(np.frombuffer(b'12345678', dtype='int8'))
True
>>> b == c.compress(memoryview(b'12345678'))
Traceback (most recent call last):
  File "<ipython-input-40-9ae8177a1d5c>", line 1, in <module>
    b == c.compress(memoryview(b'12345678'))
TypeError: compress() argument 1 must be read-only bytes-like object, not memoryview

>>> b == c.compress(bytearray(b'12345678'))
Traceback (most recent call last):
  File "<ipython-input-41-ddc68fad54ac>", line 1, in <module>
    b == c.compress(bytearray(b'12345678'))
TypeError: compress() argument 1 must be read-only bytes-like object, not bytearray

Note that other compression libraries typically accept arbitrary buffer-like objects (which allows passing them without copying to bytes before):

>>> zlib.compress(memoryview(b'12345678'))
b'x\x9c3426153\xb7\x00\x00\x07@\x01\xa5'
>>> bz2.compress(memoryview(b'12345678'))
b'BZh91AY&SY\xb6\x1c=\x04\x00\x00\x00\x08\x00?\xc0 \x001\x0c\x08\x19\x1ai\x935s\xf9E\xdc\x91N\x14$-\x87\x0fA\x00'
>>> lzma.compress(memoryview(b'12345678'))
b'\xfd7zXZ\x00\x00\x04\xe6\xd6\xb4F\x02\x00!\x01\x16\x00\x00\x00t/\xe5\xa3\x01\x00\x0712345678\x00\tx\xac+H\x80\x8b\\\x00\x01 \x08\xbb\x19\xd9\xbb\x1f\xb6\xf3}\x01\x00\x00\x00\x00\x04YZ'
>>> blosc.compress(memoryview(b'12345678'))
b'\x02\x01\x13\x08\x08\x00\x00\x00\x08\x00\x00\x00\x18\x00\x00\x0012345678'

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions