gh-134938: Add `set_pledged_input_size` to ZstdCompressor #135010

Merged

serhiy-storchaka merged 17 commits intopython:mainfrom

emmatyping:zstd-set-pledged-input-size

Jun 5, 2025

Member

emmatyping commented Jun 1, 2025 •

edited by github-actions bot

Loading

This PR adds a new method set_pledged_input_size(), which allows users to declare up front how much data will be written to the compressor for a given frame. This combined with CompressionParameter.content_size_flag allows users to ensure that even for streaming compression scenarios the content size is written into the zstd frame header, which is beneficial for reducing decompression memory usage.

Issue: Add set_pledged_input_size to ZstdCompressor #134938

📚 Documentation preview 📚: https://cpython-previews--135010.org.readthedocs.build/

emmatyping added 3 commits

June 1, 2025 10:07


          Add set_pledged_input_size to ZstdCompressor

a8ffed5


          Add tests for set_pledged_input_size

de534ab


          Add documentation for set_pledged_input_size

ee3c6c7

bedevere-app bot added the awaiting review label

bedevere-app bot mentioned this pull request

Add set_pledged_input_size to ZstdCompressor #134938

Closed

emmatyping requested a review from gpshead

June 1, 2025 17:15

emmatyping added skip news needs backport to 3.14 labels

emmatyping requested review from AA-Turner and serhiy-storchaka

June 1, 2025 17:15


          Document last_mode to be able to reference it

b8c9213

Member Author

emmatyping commented Jun 1, 2025

serhiy-storchaka reviewed

View reviewed changes

Doc/library/compression.zstd.rst Outdated Show resolved Hide resolved

Modules/_zstd/compressor.c Outdated Show resolved Hide resolved

Modules/_zstd/compressor.c Outdated Show resolved Hide resolved

Modules/_zstd/compressor.c Outdated Show resolved Hide resolved

Modules/_zstd/compressor.c Outdated Show resolved Hide resolved

ZeroIntensity reviewed

View reviewed changes

Modules/_zstd/compressor.c Outdated Show resolved Hide resolved

Modules/_zstd/compressor.c Outdated Show resolved Hide resolved

Modules/_zstd/compressor.c Outdated Show resolved Hide resolved

DangNhutNguyen approved these changes

View reviewed changes

bedevere-app bot added awaiting core review and removed awaiting review labels

DangNhutNguyen approved these changes

View reviewed changes

emmatyping added 3 commits

June 1, 2025 23:23


          Refactor set_pledged_input_size argument parsing to use CConverter

c1cf65d

Also fixup punctuation usage and don't incref Py_None.


          Remove parens from error message

450c504


          Fix tests

0dafae5

Rogdham reviewed

View reviewed changes

Doc/library/compression.zstd.rst Outdated Show resolved Hide resolved

Modules/_zstd/compressor.c Show resolved Hide resolved

serhiy-storchaka reviewed

View reviewed changes

Modules/_zstd/compressor.c Show resolved Hide resolved

Modules/_zstd/compressor.c Show resolved Hide resolved

emmatyping added 2 commits

June 2, 2025 18:15


          Refactor zstd_contentsize_converter to handle all values

d4b3229


          Correct doc about handling of unknown size and fix refs

f59718e

AA-Turner reviewed

View reviewed changes

Doc/library/compression.zstd.rst

                     The *mode* argument is a :class:`ZstdCompressor` attribute, either
                     :attr:`~.FLUSH_BLOCK`, or :attr:`~.FLUSH_FRAME`.
+                 .. method:: set_pledged_input_size(size)

Member

AA-Turner Jun 3, 2025

This doesn't currently explain why one would want to use the method.

Member Author

emmatyping Jun 4, 2025

I added a bit to describe this, but suggestions welcome.

AA-Turner reviewed

View reviewed changes

Doc/library/compression.zstd.rst Outdated Show resolved Hide resolved

AA-Turner reviewed

View reviewed changes

Modules/_zstd/compressor.c Outdated Show resolved Hide resolved

Modules/_zstd/compressor.c Outdated Show resolved Hide resolved

Modules/_zstd/compressor.c Outdated Show resolved Hide resolved

Modules/_zstd/compressor.c Outdated Show resolved Hide resolved

Modules/_zstd/compressor.c Outdated Show resolved Hide resolved

serhiy-storchaka approved these changes

View reviewed changes

Member

serhiy-storchaka left a comment

I think that mark up should be cleaned up, but there are many occurrences of ~. in existing documentation, so this can be done in a following PR.

Otherwise LGTM. 👍

Doc/library/compression.zstd.rst Outdated Show resolved Hide resolved

Doc/library/compression.zstd.rst Outdated Show resolved Hide resolved

Modules/_zstd/compressor.c Outdated Show resolved Hide resolved

Doc/library/compression.zstd.rst Show resolved Hide resolved

bedevere-app bot removed the awaiting core review label

bedevere-app bot added the awaiting merge label

emmatyping and others added 8 commits

June 3, 2025 18:29


          Apply suggestions from code review

0e40f55

Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>


          Eliminate function for raising a value error

db0ace3


          Switch from RuntimeError to ValueError

3605e11


          Add motivation for pledging size

4d3c005


          s/RuntimeError/ValueError/

6b52d45


          Apply suggestions from code review

fe78bac

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>


          Merge branch 'zstd-set-pledged-input-size' of github.com:emmatyping/c…

d0feb6a

…python into zstd-set-pledged-input-size


          Update clinic files

6c5c087

Member Author

emmatyping commented Jun 4, 2025

@serhiy-storchaka
Thanks for the approval!

I think that mark up should be cleaned up, but there are many occurrences of ~. in existing documentation, so this can be done in a following PR.

I'll make a follow up after this is merged to resolve these.

serhiy-storchaka approved these changes

View reviewed changes

serhiy-storchaka merged commit 4b44b34 into python:main

39 checks passed

bedevere-app bot removed the awaiting merge label

miss-islington-app bot commented Jun 5, 2025

Thanks @emmatyping for the PR, and @serhiy-storchaka for merging it 🌮🎉.. I'm working now to backport this PR to: 3.14.
🐍🍒⛏🤖

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request


          pythongh-134938: Add set_pledged_input_size() to ZstdCompressor (pyth…

ce208e5

…onGH-135010)

(cherry picked from commit 4b44b34)

Co-authored-by: Emma Smith <emma@emmatyping.dev>

bedevere-app bot commented Jun 5, 2025

GH-135173 is a backport of this pull request to the 3.14 branch.

bedevere-app bot removed the needs backport to 3.14 label

serhiy-storchaka pushed a commit that referenced this pull request


          [3.14] gh-134938: Add set_pledged_input_size() to ZstdCompressor (GH-…

5b39741

…135010) (GH-135173)

(cherry picked from commit 4b44b34)

Co-authored-by: Emma Smith <emma@emmatyping.dev>

Pranjal095 pushed a commit to Pranjal095/cpython that referenced this pull request


          pythongh-134938: Add set_pledged_input_size() to ZstdCompressor (pyth…

d01984b

…onGH-135010)

taegyunkim pushed a commit to taegyunkim/cpython that referenced this pull request


          pythongh-134938: Add set_pledged_input_size() to ZstdCompressor (pyth…

8d241b4

…onGH-135010)

Agent-Hellboy pushed a commit to Agent-Hellboy/cpython that referenced this pull request


          pythongh-134938: Add set_pledged_input_size() to ZstdCompressor (pyth…

f2b8dcb

…onGH-135010)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

AA-Turner AA-Turner left review comments

ZeroIntensity ZeroIntensity left review comments

serhiy-storchaka serhiy-storchaka approved these changes

gpshead Awaiting requested review from gpshead

+2 more reviewers

Rogdham Rogdham left review comments

DangNhutNguyen DangNhutNguyen approved these changes

Labels