Fix rate-limit induced hanging at test completion#340
Merged
fcostaoliveira merged 9 commits intoredis:masterfrom Feb 19, 2026
Merged
Fix rate-limit induced hanging at test completion#340fcostaoliveira merged 9 commits intoredis:masterfrom
fcostaoliveira merged 9 commits intoredis:masterfrom
Conversation
|
Hi, I’m Jit, a friendly security platform designed to help developers build secure applications from day zero with an MVS (Minimal viable security) mindset. In case there are security findings, they will be communicated to you as a comment inside the PR. Hope you’ll enjoy using Jit. Questions? Comments? Want to learn more? Get in touch with us. |
Collaborator
|
@tschneiter-figma I've confirmed on master with your test we get the expected failure: and after the fix we're good to go: I'm adding this 99 shards scenario to CI and if all passes we're good to merge. |
fcostaoliveira
approved these changes
Feb 19, 2026
Collaborator
fcostaoliveira
left a comment
There was a problem hiding this comment.
Thank you Tristan!
filipecosta90
pushed a commit
that referenced
this pull request
Feb 26, 2026
* Fix rate-limit induced hanging at test completion * include a null check for conns in all_connections_idle * Remove crufty ctx reference * Extend CI with higher shard count scenario. * Extra logging on 99 shards scenario * Verbose was already defined on CI. Using 49 shards to expedite CI * Using RLTEST_DEBUG to avoid overriding old behaviour * Shard count 99 in rate-limiting test --------- Co-authored-by: fcostaoliveira <filipe@redis.com>
2 tasks
fcostaoliveira
added a commit
that referenced
this pull request
Feb 26, 2026
* Add clang-format for code style enforcement (#336) Cherry-picked and re-applied clang-format configuration, CI workflow, Makefile targets, and DEVELOPMENT.md docs from master. Source files reformatted against the 2.2.x branch codebase. * Concurrent ubuntu test jobs for faster CI (#337) * Concurrent ubuntu test jobs for faster CI * Prevent duplicate workflow runs on push+PR by filtering branches * Fixed coverage workflow * Concurrent ASAN, TSAN, and UBSAN test jobs using matrix strategy * Updated org from redislabs to redis (#339) * Fix rate-limit induced hanging at test completion (#340) * Fix rate-limit induced hanging at test completion * include a null check for conns in all_connections_idle * Remove crufty ctx reference * Extend CI with higher shard count scenario. * Extra logging on 99 shards scenario * Verbose was already defined on CI. Using 49 shards to expedite CI * Using RLTEST_DEBUG to avoid overriding old behaviour * Shard count 99 in rate-limiting test --------- Co-authored-by: fcostaoliveira <filipe@redis.com> * configure: Respect user-supplied CXXFLAGS (#342) * Add AGENTS.md and CLAUDE.md for AI assistant guidelines (#343) Add documentation to help AI assistants work effectively with the memtier_benchmark codebase, following the https://agents.md/ conventions. AGENTS.md includes: - Project overview and repository structure - Build system and commands (autotools) - Code style (clang-format) - Testing with RLTest (standalone, cluster, TLS, sanitizers) - Key technical details - Common development tasks - Debugging guide (GDB, crash handler, core dumps, sanitizers) - License header requirements CLAUDE.md points to AGENTS.md for shared guidelines. References: - https://agents.md/ - Standard for AI agent documentation - https://docs.anthropic.com/en/docs/agents - Anthropic agent guidelines * Use latest rltest and set cluster-start-timeout to accomodate large shard count (#345) * CI: trigger workflows on semver release branches Add branch pattern '[0-9]+.[0-9]+' to push/pull_request triggers for ci, code-style, asan, tsan, and ubsan workflows so CI runs on PRs targeting release branches like 2.2. * Fix hang when using --reconnect-interval with --rate-limiting (#348) The rate-limiting timer was only created on the first connection (when get_reqs_processed() == 0). After a reconnect triggered by --reconnect-interval, disconnect() properly freed the timer, but handle_event() never recreated it because requests had already been processed. This left m_request_per_cur_interval permanently at 0, causing fill_pipeline() to return immediately on every call. Move timer creation outside the first-connection guard so it is recreated on every successful connect/reconnect when m_event_timer is NULL. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * Bumping version to 2.2.2 --------- Co-authored-by: Tristan Schneiter <tschneiter@figma.com> Co-authored-by: LINKIWI <LINKIWI@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix Rate-Limited Benchmark Hang on Completion
Problem
When running benchmarks with
--rate-limiting, the program hangs indefinitely after reaching 100% completion. This is particularly prevalent in cluster mode with TLS connections.Ctrl+Calso fails to terminate the hung process.Root Cause
Rate-limited mode uses persistent timers (
EV_PERSIST) to control request pacing. When a benchmark completes:finished()returnstrueevent_del()on the timer andbufferevent_disable()on connectionsevent_del()only removes the event from pending — it does not free the resourceSolution
disconnect_all()method to properly disconnect all shard connections when a client finishesall_connections_idle()to check that all connections have no pending responses (required for cluster mode)finished() && all_connections_idle(), calldisconnect_all(), which properly frees all bufferevents and timers via:bufferevent_free()event_free()Testing
test_rate_limited_completion_no_hangthat runs multiple iterations with multiple threads/connections to catch race conditionsFiles Changed
shard_connection.cppCleanup logic in
handle_timer_event()andfill_pipeline()client.cppAdded
disconnect_all()andall_connections_idle()implementationsclient.hAdded declarations
connections_manager.hInterface additions
tests_oss_simple_flow.pyAdded regression test
Note
Medium Risk
Touches connection lifecycle and event-loop shutdown behavior across clustered shards, so mistakes could cause premature disconnects or missed responses under load. Changes are mitigated by a targeted non-hanging regression test and dedicated CI coverage in the previously problematic cluster+TLS setup.
Overview
Fixes a hang at the end of rate-limited runs by changing completion/cleanup logic to wait for all shard connections to become idle and then forcibly tear down every connection (new
all_connections_idle()+disconnect_all()added to theconnections_manager/clientAPI and invoked fromshard_connectiontimer/pipeline paths).Adds a regression test
test_rate_limited_completion_no_hangand extends the test harness/CI to run it in an OSS Cluster TLS configuration with high shard count (plus new RLTest tuning knobs like--cluster_node_timeout, configurable log level, and longer per-test timeout).Written by Cursor Bugbot for commit 716cca4. This will update automatically on new commits. Configure here.