Pass size hint to jemalloc for faster deallocation#15071
Conversation
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Reviewed by Cursor Bugbot for commit 2b05c14. Configure here.
🤖 Augment PR SummarySummary: Adds a sized-free API to speed up jemalloc deallocation by passing a size hint to Changes:
Technical Notes: Correctness depends on callers passing a size that matches the allocation’s size class; CI now validates this via jemalloc’s opt size checks. 🤖 Was this summary useful? React with 👍 or 👎 |
|
augment review |
CE Performance Automation : step 1 of 2 (build) DONE.This comment was automatically generated given a benchmark was triggered.
You can check a comparison in detail via the grafana link |
CE Performance Automation : step 2 of 2 (benchmark) RUNNING...This comment was automatically generated given a benchmark was triggered. Started benchmark suite at 2026-04-18 19:27:34.388513 and took 168.772082 seconds up until now. In total will run 331 benchmarks. |
|
what zfree_with_size want to save is zmalloc_size(), but we already have free_with_usize(). |
|
@sundb it is faster "in jemalloc". Avoids emap/rtree lookup:
|
While profiling command execution, I noticed that command argv object alloc/free overhead is quite high for workloads with many small arguments (e.g. `HSET` with many fields). The effect is much more visible with pipelining when Redis becomes CPU bound. I experimented with replacing argv object alloc/free with a simple object pool and saw significant speedups. (Note: related effort around this topic: #13726) In this PR, I tried to improve the main hotspots in the memory allocation path (focusing on command arg allocations) to close the gap with custom pool performance, so we can avoid having a dedicated memory pools and let the whole codebase benefit from these optimizations. ## Changes ### 1) Faster dealloc via passing size hint to jemalloc (separate PR #15071) Jemalloc does more work than an object pool on free (a lookup on a tree to find the allocation's size class). For some deallocations, we can reduce free path overhead by passing a size hint to jemalloc (i.e. `sdallocx()`) which can skip metadata lookup in the common case. This PR introduces `zfree_with_size()` and uses it where we can know the allocation size i.e. `OBJ_ENCODING_EMBSTR` objects in `decrRefCount()` and SDS free path. ### 2) Reduce atomic operation cost for stat updates `update_zmalloc_stat_alloc()` / `update_zmalloc_stat_free()` previously used atomic read-modify-write (RMW) operations (`atomicIncrGet` / `atomicDecr`) which can emit expensive locked instructions on x86. When we can guarantee a single writer to a counter, we can use a cheaper load+add+store sequence instead of a locked RMW. This PR gives the first 16 threads dedicated slots for used_memory stats (intended to cover the main thread/ I/O threads) so they can use this single writer fast path. Threads beyond that fall back to a shared pool and continue to use full atomic RMW. ### 3) Improve jemalloc tcache hit rate With the default `lookahead=16` config, a pipelined HSET with ~20 fields does ~40 small allocations per command (fields + values), so you can get 16 x 40 = ~640 allocations. When args are small, many of these land in the 32 byte size class (often `EMBSTR`). Jemalloc’s default per-bin tcache cap is 200, so this kind of burst overflows the cache and it does frequent flushes. I raised the small-bin tcache limits (lg_tcache_nslots_mul:3, tcache_nslots_small_max:1000) to handle these bursts better. In the worst case, tcache may have a higher memory usage due to this change. Perhaps, another option was lowering `lookahead` to tune it differently. ### 4) Inlining When you have a simple pool, it has a few small functions and it is easy for compiler to inline them. Compared to that, jemalloc alloc/free path has a deeper call stack. Also, jemalloc was not compiled with `-flto` which was preventing inlining jemalloc functions. As part of this PR, I added `-flto` flag to jemalloc when it is enabled for Redis. Compiler also chooses not to inline some hot path functions in Redis. This suggests PGO (profile-guided optimization) could provide additional wins and perhaps we can start experimenting with it sometime. We could try to force inlining with attributes like `always_inline` but it is hard to apply across a deep call stack and misuse can cause code bloat. So, rather than going in this direction, I added `inline` keyword to some functions for now. This doesn't make compiler to inline all hot path functions but at least it is a step ahead. (If we can further improve this in future, performance gets very close to custom memory pool implementation). ## Benchmark results Commands were like: ``` memtier_benchmark --command="HSET __key__ username john_doe email john@example.com password hashed_pwd_123 created_at 1709125200 updated_at 1709125200 first_name John last_name Doe phone_number +1234567890 address 123_Main_St city NewYork country USA postal_code 10001 company Acme_Corp job_title Engineer bio Loves_coding" --command-ratio=1 --command-key-pattern=P --key-prefix="hsetkey" --key-minimum=1 --key-maximum=100000 -n 1000000 -c 50 -t 2 --hide-histogram --pipeline 50 ``` | Benchmark | Improvement | | --- | ---: | | SET | +0% | | SET (pipeline) | +8% | | HSET 15 fields | +2% | | HSET 15 fields (pipeline) | +17% | | ZADD 15 elements| +3% | | ZADD 15 elements (pipeline) | +15% |



This PR is based on valkey-io/valkey#453 and valkey-io/valkey#694
When jemalloc frees memory, it performs a lookup to find the allocation's size class.
sdallocx()lets us skip this lookup by passing the size we already know. Introduced a new free function wrapper for this:zfree_with_size().Note: Impact of this optimization is only visible on hot paths e.g. on repeated memory deallocations.
For the initial phase, I integrated this at
sdsfree()only. Over time, we may expand the usage of this new API for other performance sensitive paths.For testing, added jemalloc config
--enable-opt-size-checksto the daily fortify build. This makes jemalloc validate that the size passed tosdallocx()matches the actual allocation's size class, aborting on mismatch.Signed-off-by: Vadym Khoptynets vadymkh@amazon.com
Signed-off-by: Madelyn Olson madelyneolson@gmail.com
Co-authored-by: Madelyn Olson madelyneolson@gmail.com
Signed-off-by: ranshid ranshid@amazon.com
Note
Medium Risk
Touches low-level memory deallocation paths (
zmalloc/sdsfree) and introduces sized frees, where an incorrect size hint could crash under jemalloc or skew memory accounting (mitigated by added jemalloc size-check CI coverage).Overview
Speeds up jemalloc deallocation by adding
zfree_with_size()(wrappingje_sdallocx()with a usable-size hint) and wiring it through SDS via a news_free_with_sizemacro.Updates
sdsfree()to use sized free for all SDS types exceptSDS_TYPE_5(which lacks reliable allocation metadata), and removes the unusedsdsfreeusable()API.Strengthens test coverage by enabling jemalloc
--enable-opt-size-checksin the daily fortify workflow and treating jemalloc "size mismatch" output as a sanitizer failure in the test harness.Reviewed by Cursor Bugbot for commit 22168fe. Bugbot is set up for automated code reviews on this repo. Configure here.