Enforce io.netty.maxDirectMemory accounting on all Java versions (#16489)#16531
Merged
normanmaurer merged 1 commit intonetty:5.0from Mar 24, 2026
Merged
Enforce io.netty.maxDirectMemory accounting on all Java versions (#16489)#16531normanmaurer merged 1 commit intonetty:5.0from
normanmaurer merged 1 commit intonetty:5.0from
Conversation
…ty#16489) **Motivation**: Netty selects a Cleaner implementation based on the Java version and whether `sun.misc.Unsafe` is available. The selection matrix is: ``` Java version | Unsafe available | Cleaner selected ───────────────|──────────────────|───────────────────── 6-8 | Yes | DirectCleaner 9-23 | Yes | DirectCleaner 24 | Yes (warnings) | DirectCleaner 24 | No, native access| CleanerJava24Linker 25+ | No, native access| CleanerJava24Linker 25+ | No | CleanerJava25 ``` Before this change, direct memory accounting (`incrementMemoryCounter` / `decrementMemoryCounter`) was coupled to `USE_DIRECT_BUFFER_NO_CLEANER`, which was only true when Unsafe was available. This had two consequences: 1. `DIRECT_MEMORY_COUNTER` was only initialized inside the `USE_DIRECT_BUFFER_NO_CLEANER=true` branch, so on any Java version without Unsafe the counter was always null even if the user explicitly set `io.netty.maxDirectMemory`. 2. The accounting calls themselves lived only in PlatformDependent's legacy NoCleaner methods (`allocateDirectNoCleaner`, `reallocateDirectNoCleaner`, `freeDirectNoCleaner`), which were only called by DirectCleaner. The other Cleaner implementations (`CleanerJava9`, `CleanerJava6`, `CleanerJava24Linker`, `CleanerJava25`) never called these methods and performed no accounting. The combined effect was that the configured limit was silently ignored on every path that didn't go through DirectCleaner: ``` Java version | Unsafe | Cleaner | Counter init | Accounting ───────────────|────────|─────────────────────|──────────────|─────────── 6-8 | Yes | DirectCleaner | Yes | Yes ✓ 9-23 | Yes | DirectCleaner | Yes | Yes ✓ 24 | Yes | DirectCleaner | Yes | Yes ✓ 24 | No | CleanerJava24Linker | No | No ✗ 25+ | No | CleanerJava24Linker | No | No ✗ 25+ | No | CleanerJava25 | No | No ✗ ``` On Java 25+, where Unsafe is disabled by default, this means `io.netty.maxDirectMemory` has no effect at all. **Modifications**: - Decouple `DIRECT_MEMORY_COUNTER` initialization from Unsafe availability. The counter is now created based solely on the value of `io.netty.maxDirectMemory`, independent of `USE_DIRECT_BUFFER_NO_CLEANER`. - Move accounting into each Cleaner's `CleanableDirectBufferImpl` so that every allocation/deallocation pair tracks memory regardless of which Cleaner is active: - `DirectCleaner`: increment in `CleanableDirectBufferImpl(int capacity)` constructor, decrement in `clean()`. - `CleanerJava9`: increment in constructor, decrement in `clean()`. - `CleanerJava6`: increment in constructor, decrement in `clean()`. - `CleanerJava24Linker`: increment before `malloc()`, decrement in `clean()`, with rollback on allocation failure. - `CleanerJava25`: increment in `allocate()` before MethodHandle invoke, decrement in `clean()` via finally block. - Change `incrementMemoryCounter`/`decrementMemoryCounter` from private to package-private so Cleaner implementations (same package) can call them directly. - Add a default `reallocate(CleanableDirectBuffer, int)` method to the Cleaner interface with an allocate-copy-free fallback. DirectCleaner overrides this with in-place `Unsafe.reallocateMemory`, adjusting the counter by the delta. - Add `PlatformDependent.reallocateDirect()` as the unified public entry point for reallocation. - Remove the legacy NoCleaner API surface from PlatformDependent: `allocateDirectNoCleaner`, `allocateDirectBufferNoCleaner`, `reallocateDirectNoCleaner`, `reallocateDirectBufferNoCleaner`, and `freeDirectNoCleaner`. - Remove `USE_DIRECT_BUFFER_NO_CLEANER` and `DIRECT_CLEANER` fields. `CLEANER` is now the single entry point; `useDirectBufferNoCleaner()` returns whether `CLEANER` is an instance of DirectCleaner. - Update `UnpooledUnsafeNoCleanerDirectByteBuf` to use the new unified API: remove the `allocateDirectBuffer()` override (parent's impl now does the right thing via `PlatformDependent.allocateDirect()`), and delegate `reallocateDirect()` to `PlatformDependent.reallocateDirect()`. - Update `PlatformDependentTest.testAllocateWithCapacity0()` to use the new CleanableDirectBuffer-based API. **Result**: After this change, the accounting matrix becomes: ``` Java version | Unsafe | Cleaner | Counter init | Accounting ───────────────|────────|─────────────────────|──────────────|─────────── 6-8 | Yes | DirectCleaner | Yes | Yes ✓ 9-23 | Yes | DirectCleaner | Yes | Yes ✓ 24 | Yes | DirectCleaner | Yes | Yes ✓ 24 | No | CleanerJava24Linker | Yes | Yes ✓ 25+ | No | CleanerJava24Linker | Yes | Yes ✓ 25+ | No | CleanerJava25 | Yes | Yes ✓ ``` `io.netty.maxDirectMemory` is now enforced on all Java versions and all Cleaner implementations. The legacy raw-ByteBuffer NoCleaner API surface is eliminated, and each `CleanableDirectBuffer` is responsible for its own accounting. --------- Co-authored-by: Chris Vest <mr.chrisvest@gmail.com> (cherry picked from commit 7937553)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation:
Netty selects a Cleaner implementation based on the Java version and whether
sun.misc.Unsafeis available. The selection matrix is:Before this change, direct memory accounting (
incrementMemoryCounter/decrementMemoryCounter) was coupled toUSE_DIRECT_BUFFER_NO_CLEANER, which was only true when Unsafe was available. This had two consequences:DIRECT_MEMORY_COUNTERwas only initialized inside theUSE_DIRECT_BUFFER_NO_CLEANER=truebranch, so on any Java version without Unsafe the counter was always null even if the user explicitly setio.netty.maxDirectMemory.The accounting calls themselves lived only in PlatformDependent's legacy NoCleaner methods (
allocateDirectNoCleaner,reallocateDirectNoCleaner,freeDirectNoCleaner), which were only called by DirectCleaner. The other Cleaner implementations (CleanerJava9,CleanerJava6,CleanerJava24Linker,CleanerJava25) never called these methods and performed no accounting.The combined effect was that the configured limit was silently ignored on every path that didn't go through DirectCleaner:
On Java 25+, where Unsafe is disabled by default, this means
io.netty.maxDirectMemoryhas no effect at all.Modifications:
Decouple
DIRECT_MEMORY_COUNTERinitialization from Unsafe availability. The counter is now created based solely on the value ofio.netty.maxDirectMemory, independent ofUSE_DIRECT_BUFFER_NO_CLEANER.Move accounting into each Cleaner's
CleanableDirectBufferImplso that every allocation/deallocation pair tracks memory regardless of which Cleaner is active:DirectCleaner: increment inCleanableDirectBufferImpl(int capacity)constructor, decrement inclean().CleanerJava9: increment in constructor, decrement inclean().CleanerJava6: increment in constructor, decrement inclean().CleanerJava24Linker: increment beforemalloc(), decrement inclean(), with rollback on allocation failure.CleanerJava25: increment inallocate()before MethodHandle invoke, decrement inclean()via finally block.Change
incrementMemoryCounter/decrementMemoryCounterfrom private to package-private so Cleaner implementations (same package) can call them directly.Add a default
reallocate(CleanableDirectBuffer, int)method to the Cleaner interface with an allocate-copy-free fallback. DirectCleaner overrides this with in-placeUnsafe.reallocateMemory, adjusting the counter by the delta.Add
PlatformDependent.reallocateDirect()as the unified public entry point for reallocation.Remove the legacy NoCleaner API surface from PlatformDependent:
allocateDirectNoCleaner,allocateDirectBufferNoCleaner,reallocateDirectNoCleaner,reallocateDirectBufferNoCleaner, andfreeDirectNoCleaner.Remove
USE_DIRECT_BUFFER_NO_CLEANERandDIRECT_CLEANERfields.CLEANERis now the single entry point;useDirectBufferNoCleaner()returns whetherCLEANERis an instance of DirectCleaner.Update
UnpooledUnsafeNoCleanerDirectByteBufto use the new unified API: remove theallocateDirectBuffer()override (parent's impl now does the right thing viaPlatformDependent.allocateDirect()), and delegatereallocateDirect()toPlatformDependent.reallocateDirect().Update
PlatformDependentTest.testAllocateWithCapacity0()to use the new CleanableDirectBuffer-based API.Result:
After this change, the accounting matrix becomes:
io.netty.maxDirectMemoryis now enforced on all Java versions and all Cleaner implementations. The legacy raw-ByteBuffer NoCleaner API surface is eliminated, and eachCleanableDirectBufferis responsible for its own accounting.Co-authored-by: Chris Vest mr.chrisvest@gmail.com
(cherry picked from commit 7937553)