Enforce io.netty.maxDirectMemory accounting on all Java versions#16489
Enforce io.netty.maxDirectMemory accounting on all Java versions#16489
Conversation
|
@normanmaurer, I took a stab at fixing the issue outlined in the PR above. I am new to the allocator APIs so I would appreciate a through review and would be happy to accept feedback or suggestions on how to improve this PR. I did create a little test harness to help validate the changes across java versions. The simple java program can be found here DirectMemoryAccountingTest.java, which can then be run with shell script here run.sh. Based on the raw output from the script (netty-common-4.2.10.Final.jar and netty-common-4.2.11.Final-SNAPSHOT.jar), you can see that the memory accounting fails in a number of cases. After my fix everything passes except in the cases where we don't have native access or unsafe, which results in the NOOP cleaner being used. In these cases the Thanks, I am looking forward to any feedback you might have. |
|
Tbh I like a lot this work @j-bahr ! |
Motivation:
Netty selects a Cleaner implementation based on the Java version and
whether sun.misc.Unsafe is available. The selection matrix is:
Java version | Unsafe available | Cleaner selected
───────────────|──────────────────|─────────────────────
6-8 | Yes | DirectCleaner
9-23 | Yes | DirectCleaner
24 | Yes (warnings) | DirectCleaner
24 | No, native access| CleanerJava24Linker
25+ | No, native access| CleanerJava24Linker
25+ | No | CleanerJava25
Before this change, direct memory accounting (incrementMemoryCounter /
decrementMemoryCounter) was coupled to USE_DIRECT_BUFFER_NO_CLEANER,
which was only true when Unsafe was available. This had two consequences:
1. DIRECT_MEMORY_COUNTER was only initialized inside the
USE_DIRECT_BUFFER_NO_CLEANER=true branch, so on any Java version
without Unsafe the counter was always null — even if the user
explicitly set io.netty.maxDirectMemory.
2. The accounting calls themselves lived only in PlatformDependent's
legacy NoCleaner methods (allocateDirectNoCleaner,
reallocateDirectNoCleaner, freeDirectNoCleaner), which were only
called by DirectCleaner. The other Cleaner implementations
(CleanerJava9, CleanerJava6, CleanerJava24Linker, CleanerJava25)
never called these methods and performed no accounting.
The combined effect was that the configured limit was silently ignored
on every path that didn't go through DirectCleaner:
Java version | Unsafe | Cleaner | Counter init | Accounting
───────────────|────────|─────────────────────|──────────────|───────────
6-8 | Yes | DirectCleaner | Yes | Yes ✓
9-23 | Yes | DirectCleaner | Yes | Yes ✓
24 | Yes | DirectCleaner | Yes | Yes ✓
24 | No | CleanerJava24Linker | No | No ✗
25+ | No | CleanerJava24Linker | No | No ✗
25+ | No | CleanerJava25 | No | No ✗
On Java 25+, where Unsafe is disabled by default, this means
io.netty.maxDirectMemory has no effect at all.
Modifications:
- Decouple DIRECT_MEMORY_COUNTER initialization from Unsafe
availability. The counter is now created based solely on the value
of io.netty.maxDirectMemory, independent of USE_DIRECT_BUFFER_NO_CLEANER.
- Move accounting into each Cleaner's CleanableDirectBufferImpl so that
every allocation/deallocation pair tracks memory regardless of which
Cleaner is active:
- DirectCleaner: increment in CleanableDirectBufferImpl(int capacity)
constructor, decrement in clean().
- CleanerJava9: increment in constructor, decrement in clean().
- CleanerJava6: increment in constructor, decrement in clean().
- CleanerJava24Linker: increment before malloc(), decrement in clean(),
with rollback on allocation failure.
- CleanerJava25: increment in allocate() before MethodHandle invoke,
decrement in clean() via finally block.
- Change incrementMemoryCounter/decrementMemoryCounter from private to
package-private so Cleaner implementations (same package) can call
them directly.
- Add a default reallocate(CleanableDirectBuffer, int) method to the
Cleaner interface with an allocate-copy-free fallback. DirectCleaner
overrides this with in-place Unsafe.reallocateMemory, adjusting the
counter by the delta.
- Add PlatformDependent.reallocateDirect() as the unified public entry
point for reallocation.
- Remove the legacy NoCleaner API surface from PlatformDependent:
allocateDirectNoCleaner, allocateDirectBufferNoCleaner,
reallocateDirectNoCleaner, reallocateDirectBufferNoCleaner, and
freeDirectNoCleaner.
- Remove USE_DIRECT_BUFFER_NO_CLEANER and DIRECT_CLEANER fields. CLEANER
is now the single entry point; useDirectBufferNoCleaner() returns
whether CLEANER is an instance of DirectCleaner.
- Update UnpooledUnsafeNoCleanerDirectByteBuf to use the new unified
API: remove the allocateDirectBuffer() override (parent's impl now
does the right thing via PlatformDependent.allocateDirect()), and
delegate reallocateDirect() to PlatformDependent.reallocateDirect().
- Update PlatformDependentTest.testAllocateWithCapacity0() to use the
new CleanableDirectBuffer-based API.
Result:
After this change, the accounting matrix becomes:
Java version | Unsafe | Cleaner | Counter init | Accounting
───────────────|────────|─────────────────────|──────────────|───────────
6-8 | Yes | DirectCleaner | Yes | Yes ✓
9-23 | Yes | DirectCleaner | Yes | Yes ✓
24 | Yes | DirectCleaner | Yes | Yes ✓
24 | No | CleanerJava24Linker | Yes | Yes ✓
25+ | No | CleanerJava24Linker | Yes | Yes ✓
25+ | No | CleanerJava25 | Yes | Yes ✓
io.netty.maxDirectMemory is now enforced on all Java versions and all
Cleaner implementations. The legacy raw-ByteBuffer NoCleaner API
surface is eliminated, and each CleanableDirectBuffer is responsible
for its own accounting.
|
@chrisvest @yawkat PTAL... @j-bahr I like it! |
chrisvest
left a comment
There was a problem hiding this comment.
Two small comments, otherwise looks good.
Co-authored-by: Chris Vest <mr.chrisvest@gmail.com>
Co-authored-by: Chris Vest <mr.chrisvest@gmail.com>
|
Thanks @chrisvest for the review, I applied your suggested fixes. |
|
Could not create auto-port PR. |
…ty#16489) **Motivation**: Netty selects a Cleaner implementation based on the Java version and whether `sun.misc.Unsafe` is available. The selection matrix is: ``` Java version | Unsafe available | Cleaner selected ───────────────|──────────────────|───────────────────── 6-8 | Yes | DirectCleaner 9-23 | Yes | DirectCleaner 24 | Yes (warnings) | DirectCleaner 24 | No, native access| CleanerJava24Linker 25+ | No, native access| CleanerJava24Linker 25+ | No | CleanerJava25 ``` Before this change, direct memory accounting (`incrementMemoryCounter` / `decrementMemoryCounter`) was coupled to `USE_DIRECT_BUFFER_NO_CLEANER`, which was only true when Unsafe was available. This had two consequences: 1. `DIRECT_MEMORY_COUNTER` was only initialized inside the `USE_DIRECT_BUFFER_NO_CLEANER=true` branch, so on any Java version without Unsafe the counter was always null even if the user explicitly set `io.netty.maxDirectMemory`. 2. The accounting calls themselves lived only in PlatformDependent's legacy NoCleaner methods (`allocateDirectNoCleaner`, `reallocateDirectNoCleaner`, `freeDirectNoCleaner`), which were only called by DirectCleaner. The other Cleaner implementations (`CleanerJava9`, `CleanerJava6`, `CleanerJava24Linker`, `CleanerJava25`) never called these methods and performed no accounting. The combined effect was that the configured limit was silently ignored on every path that didn't go through DirectCleaner: ``` Java version | Unsafe | Cleaner | Counter init | Accounting ───────────────|────────|─────────────────────|──────────────|─────────── 6-8 | Yes | DirectCleaner | Yes | Yes ✓ 9-23 | Yes | DirectCleaner | Yes | Yes ✓ 24 | Yes | DirectCleaner | Yes | Yes ✓ 24 | No | CleanerJava24Linker | No | No ✗ 25+ | No | CleanerJava24Linker | No | No ✗ 25+ | No | CleanerJava25 | No | No ✗ ``` On Java 25+, where Unsafe is disabled by default, this means `io.netty.maxDirectMemory` has no effect at all. **Modifications**: - Decouple `DIRECT_MEMORY_COUNTER` initialization from Unsafe availability. The counter is now created based solely on the value of `io.netty.maxDirectMemory`, independent of `USE_DIRECT_BUFFER_NO_CLEANER`. - Move accounting into each Cleaner's `CleanableDirectBufferImpl` so that every allocation/deallocation pair tracks memory regardless of which Cleaner is active: - `DirectCleaner`: increment in `CleanableDirectBufferImpl(int capacity)` constructor, decrement in `clean()`. - `CleanerJava9`: increment in constructor, decrement in `clean()`. - `CleanerJava6`: increment in constructor, decrement in `clean()`. - `CleanerJava24Linker`: increment before `malloc()`, decrement in `clean()`, with rollback on allocation failure. - `CleanerJava25`: increment in `allocate()` before MethodHandle invoke, decrement in `clean()` via finally block. - Change `incrementMemoryCounter`/`decrementMemoryCounter` from private to package-private so Cleaner implementations (same package) can call them directly. - Add a default `reallocate(CleanableDirectBuffer, int)` method to the Cleaner interface with an allocate-copy-free fallback. DirectCleaner overrides this with in-place `Unsafe.reallocateMemory`, adjusting the counter by the delta. - Add `PlatformDependent.reallocateDirect()` as the unified public entry point for reallocation. - Remove the legacy NoCleaner API surface from PlatformDependent: `allocateDirectNoCleaner`, `allocateDirectBufferNoCleaner`, `reallocateDirectNoCleaner`, `reallocateDirectBufferNoCleaner`, and `freeDirectNoCleaner`. - Remove `USE_DIRECT_BUFFER_NO_CLEANER` and `DIRECT_CLEANER` fields. `CLEANER` is now the single entry point; `useDirectBufferNoCleaner()` returns whether `CLEANER` is an instance of DirectCleaner. - Update `UnpooledUnsafeNoCleanerDirectByteBuf` to use the new unified API: remove the `allocateDirectBuffer()` override (parent's impl now does the right thing via `PlatformDependent.allocateDirect()`), and delegate `reallocateDirect()` to `PlatformDependent.reallocateDirect()`. - Update `PlatformDependentTest.testAllocateWithCapacity0()` to use the new CleanableDirectBuffer-based API. **Result**: After this change, the accounting matrix becomes: ``` Java version | Unsafe | Cleaner | Counter init | Accounting ───────────────|────────|─────────────────────|──────────────|─────────── 6-8 | Yes | DirectCleaner | Yes | Yes ✓ 9-23 | Yes | DirectCleaner | Yes | Yes ✓ 24 | Yes | DirectCleaner | Yes | Yes ✓ 24 | No | CleanerJava24Linker | Yes | Yes ✓ 25+ | No | CleanerJava24Linker | Yes | Yes ✓ 25+ | No | CleanerJava25 | Yes | Yes ✓ ``` `io.netty.maxDirectMemory` is now enforced on all Java versions and all Cleaner implementations. The legacy raw-ByteBuffer NoCleaner API surface is eliminated, and each `CleanableDirectBuffer` is responsible for its own accounting. --------- Co-authored-by: Chris Vest <mr.chrisvest@gmail.com> (cherry picked from commit 7937553)
|
The 5.0 port: #16531 |
) (#16531) **Motivation**: Netty selects a Cleaner implementation based on the Java version and whether `sun.misc.Unsafe` is available. The selection matrix is: ``` Java version | Unsafe available | Cleaner selected ───────────────|──────────────────|───────────────────── 6-8 | Yes | DirectCleaner 9-23 | Yes | DirectCleaner 24 | Yes (warnings) | DirectCleaner 24 | No, native access| CleanerJava24Linker 25+ | No, native access| CleanerJava24Linker 25+ | No | CleanerJava25 ``` Before this change, direct memory accounting (`incrementMemoryCounter` / `decrementMemoryCounter`) was coupled to `USE_DIRECT_BUFFER_NO_CLEANER`, which was only true when Unsafe was available. This had two consequences: 1. `DIRECT_MEMORY_COUNTER` was only initialized inside the `USE_DIRECT_BUFFER_NO_CLEANER=true` branch, so on any Java version without Unsafe the counter was always null even if the user explicitly set `io.netty.maxDirectMemory`. 2. The accounting calls themselves lived only in PlatformDependent's legacy NoCleaner methods (`allocateDirectNoCleaner`, `reallocateDirectNoCleaner`, `freeDirectNoCleaner`), which were only called by DirectCleaner. The other Cleaner implementations (`CleanerJava9`, `CleanerJava6`, `CleanerJava24Linker`, `CleanerJava25`) never called these methods and performed no accounting. The combined effect was that the configured limit was silently ignored on every path that didn't go through DirectCleaner: ``` Java version | Unsafe | Cleaner | Counter init | Accounting ───────────────|────────|─────────────────────|──────────────|─────────── 6-8 | Yes | DirectCleaner | Yes | Yes ✓ 9-23 | Yes | DirectCleaner | Yes | Yes ✓ 24 | Yes | DirectCleaner | Yes | Yes ✓ 24 | No | CleanerJava24Linker | No | No ✗ 25+ | No | CleanerJava24Linker | No | No ✗ 25+ | No | CleanerJava25 | No | No ✗ ``` On Java 25+, where Unsafe is disabled by default, this means `io.netty.maxDirectMemory` has no effect at all. **Modifications**: - Decouple `DIRECT_MEMORY_COUNTER` initialization from Unsafe availability. The counter is now created based solely on the value of `io.netty.maxDirectMemory`, independent of `USE_DIRECT_BUFFER_NO_CLEANER`. - Move accounting into each Cleaner's `CleanableDirectBufferImpl` so that every allocation/deallocation pair tracks memory regardless of which Cleaner is active: - `DirectCleaner`: increment in `CleanableDirectBufferImpl(int capacity)` constructor, decrement in `clean()`. - `CleanerJava9`: increment in constructor, decrement in `clean()`. - `CleanerJava6`: increment in constructor, decrement in `clean()`. - `CleanerJava24Linker`: increment before `malloc()`, decrement in `clean()`, with rollback on allocation failure. - `CleanerJava25`: increment in `allocate()` before MethodHandle invoke, decrement in `clean()` via finally block. - Change `incrementMemoryCounter`/`decrementMemoryCounter` from private to package-private so Cleaner implementations (same package) can call them directly. - Add a default `reallocate(CleanableDirectBuffer, int)` method to the Cleaner interface with an allocate-copy-free fallback. DirectCleaner overrides this with in-place `Unsafe.reallocateMemory`, adjusting the counter by the delta. - Add `PlatformDependent.reallocateDirect()` as the unified public entry point for reallocation. - Remove the legacy NoCleaner API surface from PlatformDependent: `allocateDirectNoCleaner`, `allocateDirectBufferNoCleaner`, `reallocateDirectNoCleaner`, `reallocateDirectBufferNoCleaner`, and `freeDirectNoCleaner`. - Remove `USE_DIRECT_BUFFER_NO_CLEANER` and `DIRECT_CLEANER` fields. `CLEANER` is now the single entry point; `useDirectBufferNoCleaner()` returns whether `CLEANER` is an instance of DirectCleaner. - Update `UnpooledUnsafeNoCleanerDirectByteBuf` to use the new unified API: remove the `allocateDirectBuffer()` override (parent's impl now does the right thing via `PlatformDependent.allocateDirect()`), and delegate `reallocateDirect()` to `PlatformDependent.reallocateDirect()`. - Update `PlatformDependentTest.testAllocateWithCapacity0()` to use the new CleanableDirectBuffer-based API. **Result**: After this change, the accounting matrix becomes: ``` Java version | Unsafe | Cleaner | Counter init | Accounting ───────────────|────────|─────────────────────|──────────────|─────────── 6-8 | Yes | DirectCleaner | Yes | Yes ✓ 9-23 | Yes | DirectCleaner | Yes | Yes ✓ 24 | Yes | DirectCleaner | Yes | Yes ✓ 24 | No | CleanerJava24Linker | Yes | Yes ✓ 25+ | No | CleanerJava24Linker | Yes | Yes ✓ 25+ | No | CleanerJava25 | Yes | Yes ✓ ``` `io.netty.maxDirectMemory` is now enforced on all Java versions and all Cleaner implementations. The legacy raw-ByteBuffer NoCleaner API surface is eliminated, and each `CleanableDirectBuffer` is responsible for its own accounting. --------- Co-authored-by: Chris Vest <mr.chrisvest@gmail.com> (cherry picked from commit 7937553) Co-authored-by: Jeff Bahr <jbahr@fb.com>
### What changes were proposed in this pull request? This PR upgrades `Netty` to 4.2.12.Final. ### Why are the changes needed? To bring in the latest bug fixes. - https://netty.io/news/2026/03/24/4-2-12-Final.html - netty/netty#16550 - https://netty.io/news/2026/03/24/4-2-11-Final.html - netty/netty#16489 - netty/netty#16536 - netty/netty#16412 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? CI should pass with the existing tests. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 4.6) Closes #589 from dongjoon-hyun/SPARK-56213. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
### What changes were proposed in this pull request? This PR upgrades `Netty` to 4.2.12.Final. ### Why are the changes needed? To bring in the latest bug fixes. - https://netty.io/news/2026/03/24/4-2-12-Final.html - netty/netty#16550 - https://netty.io/news/2026/03/24/4-2-11-Final.html - netty/netty#16489 - netty/netty#16536 - netty/netty#16412 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? CI should pass with the existing tests. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 4.6) Closes #55016 from dongjoon-hyun/SPARK-56214. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
Motivation:
Netty selects a Cleaner implementation based on the Java version and whether
sun.misc.Unsafeis available. The selection matrix is:Before this change, direct memory accounting (
incrementMemoryCounter/decrementMemoryCounter) was coupled toUSE_DIRECT_BUFFER_NO_CLEANER, which was only true when Unsafe was available. This had two consequences:DIRECT_MEMORY_COUNTERwas only initialized inside theUSE_DIRECT_BUFFER_NO_CLEANER=truebranch, so on any Java version without Unsafe the counter was always null even if the user explicitly setio.netty.maxDirectMemory.The accounting calls themselves lived only in PlatformDependent's legacy NoCleaner methods (
allocateDirectNoCleaner,reallocateDirectNoCleaner,freeDirectNoCleaner), which were only called by DirectCleaner. The other Cleaner implementations (CleanerJava9,CleanerJava6,CleanerJava24Linker,CleanerJava25) never called these methods and performed no accounting.The combined effect was that the configured limit was silently ignored on every path that didn't go through DirectCleaner:
On Java 25+, where Unsafe is disabled by default, this means
io.netty.maxDirectMemoryhas no effect at all.Modifications:
Decouple
DIRECT_MEMORY_COUNTERinitialization from Unsafe availability. The counter is now created based solely on the value ofio.netty.maxDirectMemory, independent ofUSE_DIRECT_BUFFER_NO_CLEANER.Move accounting into each Cleaner's
CleanableDirectBufferImplso that every allocation/deallocation pair tracks memory regardless of which Cleaner is active:DirectCleaner: increment inCleanableDirectBufferImpl(int capacity)constructor, decrement inclean().CleanerJava9: increment in constructor, decrement inclean().CleanerJava6: increment in constructor, decrement inclean().CleanerJava24Linker: increment beforemalloc(), decrement inclean(), with rollback on allocation failure.CleanerJava25: increment inallocate()before MethodHandle invoke, decrement inclean()via finally block.Change
incrementMemoryCounter/decrementMemoryCounterfrom private to package-private so Cleaner implementations (same package) can call them directly.Add a default
reallocate(CleanableDirectBuffer, int)method to the Cleaner interface with an allocate-copy-free fallback. DirectCleaner overrides this with in-placeUnsafe.reallocateMemory, adjusting the counter by the delta.Add
PlatformDependent.reallocateDirect()as the unified public entry point for reallocation.Remove the legacy NoCleaner API surface from PlatformDependent:
allocateDirectNoCleaner,allocateDirectBufferNoCleaner,reallocateDirectNoCleaner,reallocateDirectBufferNoCleaner, andfreeDirectNoCleaner.Remove
USE_DIRECT_BUFFER_NO_CLEANERandDIRECT_CLEANERfields.CLEANERis now the single entry point;useDirectBufferNoCleaner()returns whetherCLEANERis an instance of DirectCleaner.Update
UnpooledUnsafeNoCleanerDirectByteBufto use the new unified API: remove theallocateDirectBuffer()override (parent's impl now does the right thing viaPlatformDependent.allocateDirect()), and delegatereallocateDirect()toPlatformDependent.reallocateDirect().Update
PlatformDependentTest.testAllocateWithCapacity0()to use the new CleanableDirectBuffer-based API.Result:
After this change, the accounting matrix becomes:
io.netty.maxDirectMemoryis now enforced on all Java versions and all Cleaner implementations. The legacy raw-ByteBuffer NoCleaner API surface is eliminated, and eachCleanableDirectBufferis responsible for its own accounting.