Skip to content

Auto-port 4.1: Add maxWindowLog parameter to ZstdDecoder to bound memory allocation#16894

Merged
normanmaurer merged 3 commits into
4.1from
auto-port-pr-16850-to-4.1
Jun 2, 2026
Merged

Auto-port 4.1: Add maxWindowLog parameter to ZstdDecoder to bound memory allocation#16894
normanmaurer merged 3 commits into
4.1from
auto-port-pr-16850-to-4.1

Conversation

@netty-project-bot

Copy link
Copy Markdown
Contributor

Auto-port of #16850 to 4.1
Cherry-picked commit: 0bd1657


Motivation:
The ZstdDecoder does not currently constrain the memory the underlying zstd decoder may allocate per stream. RFC 8478 ("Security Considerations") states:
An attacker may provide correctly formed compressed frames with unreasonable memory requirements. A decoder must always control memory requirements and enforce some (system-specific) limits in order to protect memory usage from such scenarios.
The most common instance of this attack is a tiny (a few hundred bytes) Zstandard frame whose header declares a very large Window_Size — for example Window_Log = 31, a 2 GiB sliding window. When such a frame is fed to libzstd, the declared window is allocated as native memory before any actual content is decoded, so a handful of concurrent connections is enough to drive a server into OOM. This is exactly the class of "unreasonable memory requirements" calls out.
The libzstd manual exposes a parameter (ZSTD_d_windowLogMax) specifically to bound this, and zstd-jni surfaces it via ZstdInputStreamNoFinalizer.setLongMax(int). The fix is to wire it up in ZstdDecoder with a sensible default. Note that the existing maximumAllocationSize parameter only caps the Netty-side output buffer handed to the next handler; it does not bound the native window memory libzstd allocates, so the attack surface remained open prior to this change.
Modification:
Add a new public constant DEFAULT_MAX_WINDOW_LOG = 27 (128 MiB window), a reasonable default for general-purpose server use that still leaves significant headroom for typical zstd CLI output (whose default Window_Log is ≤ 23).
Add a new constructor ZstdDecoder(int maximumAllocationSize, int maxWindowLog). maxWindowLog is validated to be in [10, 31] per RFC 8478.
The existing ZstdDecoder(int maximumAllocationSize) constructor now delegates to the new one with DEFAULT_MAX_WINDOW_LOG, so the default behavior of existing call sites is preserved at the API level while no longer being vulnerable to the attack above.
In handlerAdded, after the existing setContinuous(true), call zstdIs.setLongMax(maxWindowLog). Frames whose Window_Log exceeds the configured cap are rejected by libzstd with a ZstdIOException("Frame requires too much memory for decoding"), which the existing catch (Exception e) path wraps into a DecompressionException and transitions the handler to CORRUPTED.

Result:
Fixes #.
ZstdDecoder now controls the per-stream native window memory libzstd will allocate, addressing the "unreasonable memory requirements" scenario in RFC 8478. Users who legitimately need to accept frames with very large windows can opt in via the new constructor with a larger maxWindowLog.

…16850)

Motivation:
The ZstdDecoder does not currently constrain the memory the underlying
zstd decoder may allocate per stream. RFC 8478 ("Security
Considerations") states:
An attacker may provide correctly formed compressed frames with
unreasonable memory requirements. A decoder must always control memory
requirements and enforce some (system-specific) limits in order to
protect memory usage from such scenarios.
The most common instance of this attack is a tiny (a few hundred bytes)
Zstandard frame whose header declares a very large Window_Size — for
example Window_Log = 31, a 2 GiB sliding window. When such a frame is
fed to libzstd, the declared window is allocated as native memory before
any actual content is decoded, so a handful of concurrent connections is
enough to drive a server into OOM. This is exactly the class of
"unreasonable memory requirements" calls out.
The libzstd manual exposes a parameter (ZSTD_d_windowLogMax)
specifically to bound this, and zstd-jni surfaces it via
ZstdInputStreamNoFinalizer.setLongMax(int). The fix is to wire it up in
ZstdDecoder with a sensible default. Note that the existing
maximumAllocationSize parameter only caps the Netty-side output buffer
handed to the next handler; it does not bound the native window memory
libzstd allocates, so the attack surface remained open prior to this
change.
Modification:
Add a new public constant DEFAULT_MAX_WINDOW_LOG = 27 (128 MiB window),
a reasonable default for general-purpose server use that still leaves
significant headroom for typical zstd CLI output (whose default
Window_Log is ≤ 23).
Add a new constructor ZstdDecoder(int maximumAllocationSize, int
maxWindowLog). maxWindowLog is validated to be in [10, 31] per RFC 8478.
The existing ZstdDecoder(int maximumAllocationSize) constructor now
delegates to the new one with DEFAULT_MAX_WINDOW_LOG, so the default
behavior of existing call sites is preserved at the API level while no
longer being vulnerable to the attack above.
In handlerAdded, after the existing setContinuous(true), call
zstdIs.setLongMax(maxWindowLog). Frames whose Window_Log exceeds the
configured cap are rejected by libzstd with a ZstdIOException("Frame
requires too much memory for decoding"), which the existing catch
(Exception e) path wraps into a DecompressionException and transitions
the handler to CORRUPTED.

Result:
Fixes #.
ZstdDecoder now controls the per-stream native window memory libzstd
will allocate, addressing the "unreasonable memory requirements"
scenario in RFC 8478. Users who legitimately need to accept frames with
very large windows can opt in via the new constructor with a larger
maxWindowLog.

---------

Co-authored-by: Chris Vest <christianvest_hansen@apple.com>
(cherry picked from commit 0bd1657)
@normanmaurer normanmaurer merged commit d7f9069 into 4.1 Jun 2, 2026
17 of 18 checks passed
@normanmaurer normanmaurer deleted the auto-port-pr-16850-to-4.1 branch June 2, 2026 13:26
@normanmaurer normanmaurer added this to the 4.1.135.Final milestone Jun 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants