Skip to content

Batch delete keys#98

Merged
niemyjski merged 1 commit intomainfrom
bugfix/timeouts
Apr 22, 2025
Merged

Batch delete keys#98
niemyjski merged 1 commit intomainfrom
bugfix/timeouts

Conversation

@niemyjski
Copy link
Copy Markdown
Member

@niemyjski niemyjski commented Apr 22, 2025

Noticed some tests were flakey on the ci. Ran locally using rider until they failed with logging. Noticed the following:

30:49.118 E: Retry error: Timeout awaiting response (outbound=1KiB, inbound=0KiB, 8618ms elapsed, timeout is 5000ms)
...
31:22.363 E: Retry error: Timeout awaiting response (outbound=1KiB, inbound=0KiB, 5014ms elapsed, timeout is 5000ms)
...
31:29.382 E: Retry error: Timeout awaiting response (outbound=2KiB, inbound=0KiB, 5015ms elapsed, timeout is 5000ms)
...
31:40.407 E: Retry error: Timeout awaiting response (outbound=1KiB, inbound=0KiB, 5012ms elapsed, timeout is 5000ms)

Retry error: Timeout awaiting response (outbound=1KiB, inbound=0KiB, 5014ms elapsed, timeout is 5000ms), command=UNLINK, next: LLEN q:SimpleWorkItem:work, inst: 0, qu: 0, qs: 46, aw: False, bw: SpinningDown, rs: ReadAsync, ws: Idle, in: 187, in-pipe: 0, out-pipe: 0, last-in: 1, cur-in: 0, sync-ops: 1309, async-ops: 2465, serverEndpoint: 127.0.0.1:7006, conn-sec: 77.47, aoc: 1, mc: 1/1/0, mgr: 10 of 10 available, clientName: MacBookPro(SE.Redis-v2.8.31.52602), IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=111,Free=32656,Min=10,Max=32767), POOL: (Threads=111,QueuedItems=10,CompletedItems=7299,Timers=5), v: 2.8.31.52602 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts) - RedisQueue

Per https://stackexchange.github.io/StackExchange.Redis/Timeouts#was-there-a-big-request-preceding-several-small-requests-to-the-redis-that-timed-out there is a lot of in work yet to be processed / awaiting response. Also if one of these commands fail while going wide on tasks it will retry them all again.

https://redis.io/docs/latest/commands/unlink/ supports specifying multiple keys at once, thus our payload and commands shouldn't stack up as much in the client.

RemoveAllAsync in the cache client could probably do a much better job at batching these keys as it's not doing that now...

@niemyjski niemyjski requested review from Copilot and ejsmith April 22, 2025 20:40
@niemyjski niemyjski assigned niemyjski and unassigned ejsmith Apr 22, 2025
@niemyjski niemyjski added the bug label Apr 22, 2025
Copilot AI reviewed Apr 22, 2025
View reviewed changes
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR updates Redis key deletion operations to use batched multi-key deletion, reducing the number of individual Redis commands and mitigating timeouts under heavy load.

  • Batch deletes in CompleteAsync by combining multiple key deletion calls into a single call
  • Batch deletes in AbandonAsync, DeleteListAsync, and TrimDeadletterItemsAsync to minimize command stacking

@niemyjski niemyjski merged commit 3c29efc into main Apr 22, 2025
2 of 3 checks passed
@niemyjski niemyjski deleted the bugfix/timeouts branch April 22, 2025 20:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

@ejsmith ejsmith ejsmith approved these changes

Assignees

@niemyjski niemyjski

Labels

bug

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@niemyjski @ejsmith