Skip to content

Enforce io.netty.maxDirectMemory accounting on all Java versions (#16489)#16531

Merged
normanmaurer merged 1 commit intonetty:5.0from
chrisvest:5.0-memory-tracking
Mar 24, 2026
Merged

Enforce io.netty.maxDirectMemory accounting on all Java versions (#16489)#16531
normanmaurer merged 1 commit intonetty:5.0from
chrisvest:5.0-memory-tracking

Conversation

@chrisvest
Copy link
Copy Markdown
Member

Motivation:

Netty selects a Cleaner implementation based on the Java version and whether sun.misc.Unsafe is available. The selection matrix is:

  Java version   | Unsafe available | Cleaner selected
  ───────────────|──────────────────|─────────────────────
  6-8            | Yes              | DirectCleaner
  9-23           | Yes              | DirectCleaner
  24             | Yes (warnings)   | DirectCleaner
  24             | No, native access| CleanerJava24Linker
  25+            | No, native access| CleanerJava24Linker
  25+            | No               | CleanerJava25

Before this change, direct memory accounting (incrementMemoryCounter / decrementMemoryCounter) was coupled to USE_DIRECT_BUFFER_NO_CLEANER, which was only true when Unsafe was available. This had two consequences:

  1. DIRECT_MEMORY_COUNTER was only initialized inside the USE_DIRECT_BUFFER_NO_CLEANER=true branch, so on any Java version without Unsafe the counter was always null even if the user explicitly set io.netty.maxDirectMemory.

  2. The accounting calls themselves lived only in PlatformDependent's legacy NoCleaner methods (allocateDirectNoCleaner, reallocateDirectNoCleaner, freeDirectNoCleaner), which were only called by DirectCleaner. The other Cleaner implementations (CleanerJava9, CleanerJava6, CleanerJava24Linker, CleanerJava25) never called these methods and performed no accounting.

The combined effect was that the configured limit was silently ignored on every path that didn't go through DirectCleaner:

  Java version   | Unsafe | Cleaner             | Counter init | Accounting
  ───────────────|────────|─────────────────────|──────────────|───────────
  6-8            | Yes    | DirectCleaner       | Yes          | Yes ✓
  9-23           | Yes    | DirectCleaner       | Yes          | Yes ✓
  24             | Yes    | DirectCleaner       | Yes          | Yes ✓
  24             | No     | CleanerJava24Linker | No           | No  ✗
  25+            | No     | CleanerJava24Linker | No           | No  ✗
  25+            | No     | CleanerJava25       | No           | No  ✗

On Java 25+, where Unsafe is disabled by default, this means io.netty.maxDirectMemory has no effect at all.

Modifications:

  • Decouple DIRECT_MEMORY_COUNTER initialization from Unsafe availability. The counter is now created based solely on the value of io.netty.maxDirectMemory, independent of
    USE_DIRECT_BUFFER_NO_CLEANER.

  • Move accounting into each Cleaner's CleanableDirectBufferImpl so that every allocation/deallocation pair tracks memory regardless of which Cleaner is active:

  • DirectCleaner: increment in CleanableDirectBufferImpl(int capacity) constructor, decrement in clean().

    • CleanerJava9: increment in constructor, decrement in clean().
    • CleanerJava6: increment in constructor, decrement in clean().
  • CleanerJava24Linker: increment before malloc(), decrement in clean(), with rollback on allocation failure.

  • CleanerJava25: increment in allocate() before MethodHandle invoke, decrement in clean() via finally block.

  • Change incrementMemoryCounter/decrementMemoryCounter from private to package-private so Cleaner implementations (same package) can call them directly.

  • Add a default reallocate(CleanableDirectBuffer, int) method to the Cleaner interface with an allocate-copy-free fallback. DirectCleaner overrides this with in-place Unsafe.reallocateMemory, adjusting the counter by the delta.

  • Add PlatformDependent.reallocateDirect() as the unified public entry point for reallocation.

  • Remove the legacy NoCleaner API surface from PlatformDependent: allocateDirectNoCleaner, allocateDirectBufferNoCleaner, reallocateDirectNoCleaner, reallocateDirectBufferNoCleaner, and freeDirectNoCleaner.

  • Remove USE_DIRECT_BUFFER_NO_CLEANER and DIRECT_CLEANER fields. CLEANER is now the single entry point; useDirectBufferNoCleaner() returns whether CLEANER is an instance of DirectCleaner.

  • Update UnpooledUnsafeNoCleanerDirectByteBuf to use the new unified API: remove the allocateDirectBuffer() override (parent's impl now does the right thing via PlatformDependent.allocateDirect()), and delegate reallocateDirect() to PlatformDependent.reallocateDirect().

  • Update PlatformDependentTest.testAllocateWithCapacity0() to use the new CleanableDirectBuffer-based API.

Result:

After this change, the accounting matrix becomes:

  Java version   | Unsafe | Cleaner             | Counter init | Accounting
  ───────────────|────────|─────────────────────|──────────────|───────────
  6-8            | Yes    | DirectCleaner       | Yes          | Yes ✓
  9-23           | Yes    | DirectCleaner       | Yes          | Yes ✓
  24             | Yes    | DirectCleaner       | Yes          | Yes ✓
  24             | No     | CleanerJava24Linker | Yes          | Yes ✓
  25+            | No     | CleanerJava24Linker | Yes          | Yes ✓
  25+            | No     | CleanerJava25       | Yes          | Yes ✓

io.netty.maxDirectMemory is now enforced on all Java versions and all Cleaner implementations. The legacy raw-ByteBuffer NoCleaner API surface is eliminated, and each CleanableDirectBuffer is responsible for its own accounting.


Co-authored-by: Chris Vest mr.chrisvest@gmail.com

(cherry picked from commit 7937553)

…ty#16489)

**Motivation**:

Netty selects a Cleaner implementation based on the Java version and
whether `sun.misc.Unsafe` is available. The selection matrix is:
```
  Java version   | Unsafe available | Cleaner selected
  ───────────────|──────────────────|─────────────────────
  6-8            | Yes              | DirectCleaner
  9-23           | Yes              | DirectCleaner
  24             | Yes (warnings)   | DirectCleaner
  24             | No, native access| CleanerJava24Linker
  25+            | No, native access| CleanerJava24Linker
  25+            | No               | CleanerJava25
```
Before this change, direct memory accounting (`incrementMemoryCounter` /
`decrementMemoryCounter`) was coupled to `USE_DIRECT_BUFFER_NO_CLEANER`,
which was only true when Unsafe was available. This had two
consequences:

1. `DIRECT_MEMORY_COUNTER` was only initialized inside the
`USE_DIRECT_BUFFER_NO_CLEANER=true` branch, so on any Java version
without Unsafe the counter was always null even if the user explicitly
set `io.netty.maxDirectMemory`.

2. The accounting calls themselves lived only in PlatformDependent's
legacy NoCleaner methods (`allocateDirectNoCleaner`,
`reallocateDirectNoCleaner`, `freeDirectNoCleaner`), which were only
called by DirectCleaner. The other Cleaner implementations
(`CleanerJava9`, `CleanerJava6`, `CleanerJava24Linker`, `CleanerJava25`)
never called these methods and performed no accounting.

The combined effect was that the configured limit was silently ignored
on every path that didn't go through DirectCleaner:
```
  Java version   | Unsafe | Cleaner             | Counter init | Accounting
  ───────────────|────────|─────────────────────|──────────────|───────────
  6-8            | Yes    | DirectCleaner       | Yes          | Yes ✓
  9-23           | Yes    | DirectCleaner       | Yes          | Yes ✓
  24             | Yes    | DirectCleaner       | Yes          | Yes ✓
  24             | No     | CleanerJava24Linker | No           | No  ✗
  25+            | No     | CleanerJava24Linker | No           | No  ✗
  25+            | No     | CleanerJava25       | No           | No  ✗
```
On Java 25+, where Unsafe is disabled by default, this means
`io.netty.maxDirectMemory` has no effect at all.

**Modifications**:

- Decouple `DIRECT_MEMORY_COUNTER` initialization from Unsafe
availability. The counter is now created based solely on the value of
`io.netty.maxDirectMemory`, independent of
`USE_DIRECT_BUFFER_NO_CLEANER`.

- Move accounting into each Cleaner's `CleanableDirectBufferImpl` so
that every allocation/deallocation pair tracks memory regardless of
which Cleaner is active:
- `DirectCleaner`: increment in `CleanableDirectBufferImpl(int
capacity)` constructor, decrement in `clean()`.
  - `CleanerJava9`: increment in constructor, decrement in `clean()`.
  - `CleanerJava6`: increment in constructor, decrement in `clean()`.
- `CleanerJava24Linker`: increment before `malloc()`, decrement in
`clean()`, with rollback on allocation failure.
- `CleanerJava25`: increment in `allocate()` before MethodHandle invoke,
decrement in `clean()` via finally block.

- Change `incrementMemoryCounter`/`decrementMemoryCounter` from private
to package-private so Cleaner implementations (same package) can call
them directly.

- Add a default `reallocate(CleanableDirectBuffer, int)` method to the
Cleaner interface with an allocate-copy-free fallback. DirectCleaner
overrides this with in-place `Unsafe.reallocateMemory`, adjusting the
counter by the delta.

- Add `PlatformDependent.reallocateDirect()` as the unified public entry
point for reallocation.

- Remove the legacy NoCleaner API surface from PlatformDependent:
`allocateDirectNoCleaner`, `allocateDirectBufferNoCleaner`,
`reallocateDirectNoCleaner`, `reallocateDirectBufferNoCleaner`, and
`freeDirectNoCleaner`.

- Remove `USE_DIRECT_BUFFER_NO_CLEANER` and `DIRECT_CLEANER` fields.
`CLEANER` is now the single entry point; `useDirectBufferNoCleaner()`
returns whether `CLEANER` is an instance of DirectCleaner.

- Update `UnpooledUnsafeNoCleanerDirectByteBuf` to use the new unified
API: remove the `allocateDirectBuffer()` override (parent's impl now
does the right thing via `PlatformDependent.allocateDirect()`), and
delegate `reallocateDirect()` to `PlatformDependent.reallocateDirect()`.

- Update `PlatformDependentTest.testAllocateWithCapacity0()` to use the
new CleanableDirectBuffer-based API.

**Result**:

After this change, the accounting matrix becomes:
```
  Java version   | Unsafe | Cleaner             | Counter init | Accounting
  ───────────────|────────|─────────────────────|──────────────|───────────
  6-8            | Yes    | DirectCleaner       | Yes          | Yes ✓
  9-23           | Yes    | DirectCleaner       | Yes          | Yes ✓
  24             | Yes    | DirectCleaner       | Yes          | Yes ✓
  24             | No     | CleanerJava24Linker | Yes          | Yes ✓
  25+            | No     | CleanerJava24Linker | Yes          | Yes ✓
  25+            | No     | CleanerJava25       | Yes          | Yes ✓
```
`io.netty.maxDirectMemory` is now enforced on all Java versions and all
Cleaner implementations. The legacy raw-ByteBuffer NoCleaner API surface
is eliminated, and each `CleanableDirectBuffer` is responsible for its
own accounting.

---------

Co-authored-by: Chris Vest <mr.chrisvest@gmail.com>

(cherry picked from commit 7937553)
@chrisvest chrisvest enabled auto-merge (squash) March 23, 2026 22:10
@normanmaurer normanmaurer added this to the 5.0.0.Final milestone Mar 24, 2026
@normanmaurer normanmaurer disabled auto-merge March 24, 2026 07:06
@normanmaurer normanmaurer merged commit 43125d9 into netty:5.0 Mar 24, 2026
12 of 13 checks passed
@chrisvest chrisvest deleted the 5.0-memory-tracking branch March 24, 2026 17:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants