Upgrade jemalloc 5.2.1 #9623
Upgrade jemalloc 5.2.1 #9623
Conversation
8e47347 to
aa73d76
Compare
git-subtree-dir: deps/jemalloc git-subtree-split: 886e40bb339ec1358a5ff2a52fdb782ca66461cb
./autogen.sh --with-version=5.2.1-0-g0
* overflow in jemalloc fragmentation hint to the defragger
aa73d76 to
1cea18d
Compare
oranagra
left a comment
There was a problem hiding this comment.
i've reviewed all commits except for the first 3, and all modifications outside the jemalloc folder.
i've reviewed the jemalloc ChangeLog, and the only thing that i want to look into is this:
- jemalloc/jemalloc#1421 - maybe we can now undo #6145 ref: jemalloc/jemalloc#521 (change that was added in jemalloc 5.0)
other than that, @yoav-steinberg please confirm that following the procedure you described you are able to upgrade jemalloc in the future without going though these cherry picks again.
There's a rare case which leads to stagnation in the defragger, causing it to keep scanning the keyspace and do nothing (not moving any allocation), this happens when all the allocator slabs of a certain bin have the same % utilization, but the slab from which new allocations are made have a lower utilization. this commit fixes it by removing the current slab from the overall average utilization of the bin, and also eliminate any precision loss in the utilization calculation and move the decision about the defrag to reside inside jemalloc. and also add a test that consistently reproduce this issue.
- The argument `u` in for `ar` is ignored (and generates warnings since `D` became the default. All it does is avoid updating unchanged objects (shouldn't have any impact on our build) - Enable `LUA_USE_MKSTEMP` to force the use of `mkstemp()` instead of `tmpname()` (which is dead code in redis anyway). - Remove unused variable `c` in `f_parser()` - Removed misleadingly indented space in `luaL_loadfile()` and ``addfield()` Co-authored-by: Oran Agra <oran@redislabs.com>
6685dab to
85737e6
Compare
The background jemalloc (#6145) thread does two things with unused pages:
The new jemalloc essentially disables the muzzy state. But there might still be pages that are unused but no one called
Confirming. |
oranagra
left a comment
There was a problem hiding this comment.
@redis/core-team please approve
|
triggered daily CI: |
|
looks like we have an issue with active-defrag on 32 bit |
|
i've tested it locally and i can reproduce the problem. |
Test started failing consistently in 32bit builds after upgrading to jemalloc 5.2.1 (redis#9623).
Test started failing consistently in 32bit builds after upgrading to jemalloc 5.2.1 (#9623).
Upgraded to jemalloc 5.2.1 from 5.1.0. Cherry picked all relevant fixes (by diffing our 5.1.0 to upstream 5.10 and finding relevant commits). Details of what was done:
[cherry-picked] fd7d51c 2021-05-03 Resolve nonsense static analysis warnings (Oran Agra)
[cherry-picked] 448c435 2020-09-29 Fix compilation warnings in Lua and jemalloc dependencies (#7785) (YoongHM)
[skipped - already in upstream] 9216b96 2020-09-21 Fix compilation warning in jemalloc's malloc_vsnprintf (#7789) (YoongHM)
[cherry-picked] 88d71f4 2020-05-20 fix a rare active defrag edge case bug leading to stagnation (Oran Agra)
[skipped - already in upstream] 2fec7d9 2019-05-30 Jemalloc: Avoid blocking on background thread lock for stats.
[cherry-picked] 920158e 2018-07-11 Active defrag fixes for 32bit builds (again) (Oran Agra)
[cherry-picked] e8099ca 2018-06-26 add defrag hint support into jemalloc 5 (Oran Agra)
[re-done] 4e729fc 2018-05-24 Generate configure for Jemalloc. (antirez)
Additionally had to do this:
7727cc248 2021-10-10 Fix defrag to support sharded bins in arena (added in v5.2.1) (Yoav Steinberg)
When reviewing please look at all except the first commit which is just replacing 5.1.0 with 5.2.1 sources.
Also I think we should merge this without squashing to preserve the changes we did to to jemalloc.