Introduce DEBUG_DEFRAG compilation option to allow run test with activedefrag when allocator is not jemalloc#14326
Conversation
…hen allocator is not jemalloc
🎉 Snyk checks have passed. No issues have been found so far.✅ security/snyk check is complete. No issues have been found. (View Details) ✅ license/snyk check is complete. No issues have been found. (View Details) |
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
|
defrag CI(failed) with commit defrag CI(all tests passed except a known issue) with commit |
oranagra
left a comment
There was a problem hiding this comment.
this causes the server to always detect fragmentation, and when it reaches a pointer, it always relocates it, but it still runs only on server cron, so there's a good chance many tests will reach certain states, and then clear these states before the defragger had a chance to mess them up.
maybe add another mode in which we run full defrag cycle in each beforeSleep (then we don't even need to fake high fragmentation metric)
@oranagra in the commit, I changed the defrag duty cycle to 30 seconds to avoid failures caused by a large number of slow tests. The effect is roughly the same as manually forcing a full cycle in beforeSleep(). Through this change, I also discovered another issue related to defragmentation. |
|
Ok. I missed that. That's indeed probably sufficient to scan the whole thing in one cycle. |
We are no longer doing defragmentation in cron now, but in ae timer. So after the commit: 2432cff, As long as the fragment is complete and the cycle does not exceed 30 seconds (if one completed cycle ends, we will immediately start the next one), each ae timer will be triggered.
I tried to comment out the tests that are consuming too much time, but there might be quite a few tests that need to be commented out. Below is the test duration consumption when I used fully defrag. The total test time was one and a half hours. |
|
ok. so essentially it'll run on each event loop. regarding duration, that's not too bad.. |
|
from my perspective, you can merge it. |
|
Intentional failures(commit c8876fc) can be detected by this PR. |
This reverts commit c8876fc.
There was a problem hiding this comment.
Pull Request Overview
This PR introduces a DEBUG_DEFRAG compilation option that enables active defragmentation functionality for testing purposes even when jemalloc is not available. The changes allow tests to run with defragmentation enabled using alternative allocators like AddressSanitizer.
- Adds DEBUG_DEFRAG compilation flags (force/fully modes) to enable defrag testing without jemalloc
- Implements alternative defrag functions for non-jemalloc allocators that always perform defragmentation
- Updates test framework to support --debug-defrag flag and skip incompatible tests
Reviewed Changes
Copilot reviewed 20 out of 20 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| src/Makefile | Adds DEBUG_DEFRAG compilation flags for force and fully modes |
| src/zmalloc.h | Extends HAVE_DEFRAG macro to include DEBUG_DEFRAG_FORCE |
| src/zmalloc.c | Conditionally compiles jemalloc-specific functions |
| src/defrag.c | Implements alternative defrag functions and forces high fragmentation reporting |
| tests/test_helper.tcl | Adds --debug-defrag command line option |
| tests/support/server.tcl | Adds debug_defrag:skip tag handling and defrag config |
| .github/workflows/daily.yml | Adds CI job for testing with DEBUG_DEFRAG |
| tests/unit/*.tcl | Adds debug_defrag:skip tags to tests incompatible with debug defrag mode |
| tests/integration/*.tcl | Adds debug_defrag:skip tags to replication tests |
| tests/unit/moduleapi/*.tcl | Adds debug_defrag:skip tags to module tests |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
|
Run the CI ten times and all passed, i think we can merge it now. |
This PR is based on valkey-io/valkey#1303
This PR introduces a DEBUG_DEFRAG compilation option that enables activedefrag functionality even when the allocator is not jemalloc, and always forces defragmentation regardless of the amount or ratio of fragmentation.
Using
DEBUG_DEFRAG=force
DEBUG_DEFRAG=fully
force.Co-authored-by: Ran Shidlansik ranshid@amazon.com