This is a change coming from #15741 @ 8953bbe

FYI @laosijikaichele this should deliver quite a decent boost 🗡️
in pretty much ALL cases, non thread locals and thread locals in particular ^^

@chrisvest sadly I have left the ref cnt to still be allocated per chunk, due to other optimizations, but it shouldn't be used

This is ready to go @normanmaurer in term of perf numbers I have already performed tests a lot for other prs on this, and the numbers were showing 1.5X or 2.X better perf for the thread local allocations

@laosijikaichele I will check if using your perf test on mimalloc, but sadly we don't have the same HW so I will likely get some different numbers 😢
But last time I tried this, it was speeding things up a lot

@franz1981 great work, no worries about the hardware, we can mainly compare the number change on adaptive, before and after this PR, to see the improvement.

@chrisvest @laosijikaichele @yawkat PTAL

PTAL @normanmaurer I have simplified the capacity method to make it inlineable, but TBH it's the logic there which concern me, since mpsc::size is not "trivial" (is a loop!)

@laosijikaichele I was wrong, it seems to improve, on my x86 by a mere ~7-8% atm, maybe you will have better luck with aarch64?
Now the most relevant cost is the single buffer release :"( (due to the new RefCnt indirection and VarHandle usage)

When i see things like these my heart is bleeding 😢

   0.10%    0x00007f1ddc2103cb:   jg     0x00007f1ddc21118c           ;*if_icmpgt {reexecute=0 rethrow=0 return_oop=0}
                                                                      ; - io.netty.buffer.AdaptivePoolingAllocator::allocate@7 (line 260)
            0x00007f1ddc2103d1:   lea    0x1f(%rdx),%r8d
   4.09%    0x00007f1ddc2103d5:   sar    $0x5,%r8d                    ;*ishr {reexecute=0 rethrow=0 return_oop=0}
                                                                      ; - io.netty.buffer.AdaptivePoolingAllocator::sizeIndexOf@5 (line 282)
                                                                      ; - io.netty.buffer.AdaptivePoolingAllocator::sizeClassIndexOf@1 (line 286)
                                                                      ; - io.netty.buffer.AdaptivePoolingAllocator::allocate@11 (line 261)

I know that is a data dep but, c'mon...on Intel this the lea is faster...

@laosijikaichele i would be curious how the numbers now look vs mimalloc to understand the gap left (assuming proper sized chunk q 😢)

Results on my machine
without thread local accesses

  ┌────────────────────────┬──────────────────────┬──────────────────────┬───────┐
  │        Scenario        │  Before (77e2a683)   │   After (6975d4e7)   │ Delta │
  ├────────────────────────┼──────────────────────┼──────────────────────┼───────┤
  │ Default                │ 63.368 ± 1.487 ns/op │ 60.635 ± 0.900 ns/op │ -4.3% │
  ├────────────────────────┼──────────────────────┼──────────────────────┼───────┤
  │ --enable-native-access │ 62.687 ± 0.828 ns/op │ 59.861 ± 0.240 ns/op │ -4.5% │
  └────────────────────────┴──────────────────────┴──────────────────────┴───────┘

and thread local one

  ┌───────────────────────────────────────────┬──────────────────────┬──────────────────────┬────────┐
  │                 Scenario                  │  Before (77e2a683)   │   After (6975d4e7)   │ Delta  │
  ├───────────────────────────────────────────┼──────────────────────┼──────────────────────┼────────┤
  │ Custom executor                           │ 37.993 ± 0.762 ns/op │ 35.543 ± 0.936 ns/op │ -6.4%  │
  ├───────────────────────────────────────────┼──────────────────────┼──────────────────────┼────────┤
  │ Custom executor + native-access           │ 39.102 ± 0.503 ns/op │ 34.715 ± 0.704 ns/op │ -11.2% │
  └───────────────────────────────────────────┴──────────────────────┴──────────────────────┴────────┘

Not wow, but not terrible.
Let's say the most of the improvements were already rolled out in the previous PR i made@ #15741
Would be nice to see how much aarch64 benefit from removing atomic ops in the hot path (twice atomic ops)

And the other hidden performance benefit is that we now have scalable allocate/release from different threads - because it will scale as much as the underlying mpsc queue, while before it was "limited" by the shared "chunk" ref cnt. It's a pathological case, but still, worth mentioning.

PTAL @normanmaurer if you're happy with the first numbers at #16306 (comment)

@chrisvest in relation of your request to turn some release into mark to deallocate I'll gently push back: these three release() calls are all refcount-related operations (undoing a prior retain()), not "I'm done with this chunk" signals. Changing them to markToDeallocate() would be semantically misleading IMHO 🙏

Fair enough 👍

Revert the io_uring changes. We can rebase on #16359 when it goes in.

The builds got cancelled after 6 hours and didn't produce any logs, which is unfortunate as we don't know if it's the allocator changes or the io_uring changes that made them get stuck.

PTAL @normanmaurer

@chrisvest done bud ! Let's see if the CI is happy ;)
Right after this @laosijikaichele if you are interested we can take a look to fix the chunk reuse q, which is not good ATM (the offer/poll storm is not good at all), maybe with @chrisvest ? wdyt?

CI is green @normanmaurer @chrisvest so, ready to go?

Very nice. Thanks!

Could not create auto-port PR.
Got conflicts when cherry-picking onto 4.1.

Auto-port PR for 5.0: #16378

Port PR for 4.1: #16379