Add XDELEX and XACKDEL commands for stream by sundb · Pull Request #14130 · redis/redis

sundb · 2025-06-15T15:23:01Z

Summary and detailed design for new stream command

XDELEX

Syntax

XDELEX key [KEEPREF | DELREF | ACKED] IDS numids id [id ...]

Description

The XDELEX command extends the Redis Streams XDEL command, offering enhanced control over message entry deletion with respect to consumer groups. It accepts optional DELREF or ACKED parameters to modify its behavior:

KEEPREF: Deletes the specified entries from the stream, but preserves existing references to these entries in all consumer groups' PEL. This behavior is similar to XDEL.
DELREF: Deletes the specified entries from the stream and also removes all references to these entries from all consumer groups' pending entry lists, effectively cleaning up all traces of the messages.
ACKED: Only trims entries that were read and acknowledged by all consumer groups.

Note: The IDS block can appear at any position in the command, consistent with other commands.

Reply

Array reply, for each id:

-1: No such id exists in the provided stream key.
1: Entry was deleted from the stream.
2: Entry was not deleted, but there are still dangling references. (ACKED option)

XACKDEL

Syntax

XACKDEL key group [KEEPREF | DELREF | ACKED] IDS numids id [id ...]

Description

The XACKDEL command combines XACK and XDEL functionalities in Redis Streams. It acknowledges specified message IDs in the given consumer group and attempts to delete corresponding stream entries. It accepts optional DELREF or ACKED parameters:

KEEPREF: Acknowledges the messages in the specified consumer group and deletes the entries from the stream, but preserves existing references to these entries in all consumer groups' PEL.
DELREF: Acknowledges the messages in the specified consumer group, deletes the entries from the stream, and also removes all references to these entries from all consumer groups' pending entry lists, effectively cleaning up all traces of the messages.
ACKED: Acknowledges the messages in the specified consumer group and only trims entries that were read and acknowledged by all consumer groups.

Reply

Array reply, for each id:

-1: No such id exists in the provided stream key.
1: Entry was acknowledged and deleted from the stream.
2: Entry was acknowledged but not deleted, but there are still dangling references. (ACKED option)

Redis Streams Commands Extension

XTRIM

Syntax

XTRIM key <MAXLEN | MINID> [= | ~] threshold [LIMIT count] [KEEPREF | DELREF | ACKED]

Description

The XTRIM command trims a stream by removing entries based on specified criteria, extended to include optional DELREF or ACKED parameters for consumer group handling:

KEEPREF: Trims the stream according to the specified strategy (MAXLEN or MINID) regardless of whether entries are referenced by any consumer groups, but preserves existing references to these entries in all consumer groups' PEL.
DELREF: Trims the stream according to the specified strategy and also removes all references to the trimmed entries from all consumer groups' PEL.
ACKED: Only trims entries that were read and acknowledged by all consumer groups.

Reply

No change.

XADD

Syntax

XADD key [NOMKSTREAM] [<MAXLEN | MINID> [= | ~] threshold [LIMIT count]] [KEEPREF | DELREF | ACKED] <* | id> field value [field value ...]

Description

The XADD command appends a new entry to a stream and optionally trims it in the same operation, extended to include optional DELREF or ACKED parameters for trimming behavior:

KEEPREF: When trimming, removes entries from the stream according to the specified strategy (MAXLEN or MINID), regardless of whether they are referenced by any consumer groups, but preserves existing references to these entries in all consumer groups' PEL.
DELREF: When trimming, removes entries from the stream according to the specified strategy and also removes all references to these entries from all consumer groups' PEL.
ACKED: When trimming, only removes entries that were read and acknowledged by all consumer groups. Note that if the number of referenced entries is bigger than MAXLEN, we will still stop.

Reply

No change.

Key implementation

Since we currently have no simple way to track the association between an entry and consumer groups without iterating over all groups, we introduce two mechanisms to establish this link. This allows us to determine whether an entry has been seen by all consumer groups, and to identify which groups are referencing it. With this links, we can break the association when the entry is either acknowledged or deleted.

Added reference tracking between stream messages and consumer groups using cgroups_ref
The cgroups_ref is implemented as a rax that maps stream message IDs to lists of consumer groups that reference those messages, and streamNACK stores the corresponding nodes of this list, so that the corresponding groups can be deleted during ACK.
In this way, we can determine whether an entry has been seen but not ack.
Store a cache minimum last_id in the stream structure.
The reason for doing this is that there is a situation where an entry has never been seen by the consume group. In this case, we think this entry has not been consumed either. If there is an "ACKED" option, we cannot directly delete this entry either.
When a consumer group updates its last_id, we don’t immediately update the cached minimum last_id. Instead, we check whether the group’s previous last_id was equal to the current minimum, or whether the new last_id is smaller than the current minimum (when using XGROUP SETID). If either is true, we mark the cached minimum last_id as invalid, and defer the actual update until the next time it’s needed.

snyk-io · 2025-06-15T15:23:12Z

🎉 Snyk checks have passed. No issues have been found so far.

✅ security/snyk check is complete. No issues have been found. (View Details)

✅ license/snyk check is complete. No issues have been found. (View Details)

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

@oranagra

…fter reload (#14276) This bug was introduced by #14130 found by @oranagra ### Summary Because `s->cgroup_ref` is created at runtime the first time a consumer group is linked with a message, but it is not released when all references are removed. However, after `debug reload` or restart, if the PEL is empty (meaning no consumer group is referencing any message), `s->cgroup_ref` will not be recreated. As a result, when executing XADD or XTRIM with `ACKED` option and checking whether a message that is being read but has not been ACKed can be deleted, the cgroup_ref being NULL will cause a crash. ### Code Path ``` xaddCommand -> streamTrim -> streamEntryIsReferenced ``` ### Solution Check if `s->cgroup_ref` is NULL in streamEntryIsReferenced().

@oranagra

…fter reload (redis#14276) This bug was introduced by redis#14130 found by @oranagra ### Summary Because `s->cgroup_ref` is created at runtime the first time a consumer group is linked with a message, but it is not released when all references are removed. However, after `debug reload` or restart, if the PEL is empty (meaning no consumer group is referencing any message), `s->cgroup_ref` will not be recreated. As a result, when executing XADD or XTRIM with `ACKED` option and checking whether a message that is being read but has not been ACKed can be deleted, the cgroup_ref being NULL will cause a crash. ### Code Path ``` xaddCommand -> streamTrim -> streamEntryIsReferenced ``` ### Solution Check if `s->cgroup_ref` is NULL in streamEntryIsReferenced().

@oranagra

…fter reload (#14276) This bug was introduced by #14130 found by @oranagra ### Summary Because `s->cgroup_ref` is created at runtime the first time a consumer group is linked with a message, but it is not released when all references are removed. However, after `debug reload` or restart, if the PEL is empty (meaning no consumer group is referencing any message), `s->cgroup_ref` will not be recreated. As a result, when executing XADD or XTRIM with `ACKED` option and checking whether a message that is being read but has not been ACKed can be deleted, the cgroup_ref being NULL will cause a crash. ### Code Path ``` xaddCommand -> streamTrim -> streamEntryIsReferenced ``` ### Solution Check if `s->cgroup_ref` is NULL in streamEntryIsReferenced().

This PR fixes three defrag issues. 1. Fix the issue that forget to update cgroup_ref_node when the consume group was reallocated. This crash was introduced by #14130 In this PR, when performing defragmentation on `s->cgroups` using `defragRadixTree()`, we no longer rely on the automatic data defragmentation of `defragRadixTree()`. Instead, we manually defragment the consumer group and then update its reference in `s->cgroups`. 2. Fix a use-after-free issue caused by updating dictionary keys after HFE key is reallocated. This issue was introduced by #13842 3. Fix the issue that forgot to be updated NextSegHdr->firstSeg when the first segment was reallocated. This issue was introduced by #13842 --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

This PR fixes three defrag issues. 1. Fix the issue that forget to update cgroup_ref_node when the consume group was reallocated. This crash was introduced by redis#14130 In this PR, when performing defragmentation on `s->cgroups` using `defragRadixTree()`, we no longer rely on the automatic data defragmentation of `defragRadixTree()`. Instead, we manually defragment the consumer group and then update its reference in `s->cgroups`. 2. Fix a use-after-free issue caused by updating dictionary keys after HFE key is reallocated. This issue was introduced by redis#13842 3. Fix the issue that forgot to be updated NextSegHdr->firstSeg when the first segment was reallocated. This issue was introduced by redis#13842 --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

… (CVE-2025-62507) This issue was introduced by #14130. The problem is that when the number of IDs exceeds STREAMID_STATIC_VECTOR_LEN (8), the code forgot to reallocate memory for the IDs array, which causes a stack overflow.

… (CVE-2025-62507) This issue was introduced by redis#14130. The problem is that when the number of IDs exceeds STREAMID_STATIC_VECTOR_LEN (8), the code forgot to reallocate memory for the IDs array, which causes a stack overflow.

… (CVE-2025-62507) This issue was introduced by #14130. The problem is that when the number of IDs exceeds STREAMID_STATIC_VECTOR_LEN (8), the code forgot to reallocate memory for the IDs array, which causes a stack overflow.

…egies (#14623) This bug was introduced by #14130 and found by guybe7 When using XTRIM/XADD with approx mode (~) and DELREF/ACKED delete strategies, if a node was eligible for removal but couldn't be removed directly (because consumer group references need to be checked), the code would incorrectly break out of the loop instead of continuing to process entries within the node. This fix allows the per-entry deletion logic to execute for eligible nodes when using non-KEEPREF strategies.

@oranagra

…fter reload (redis#14276) This bug was introduced by redis#14130 found by @oranagra ### Summary Because `s->cgroup_ref` is created at runtime the first time a consumer group is linked with a message, but it is not released when all references are removed. However, after `debug reload` or restart, if the PEL is empty (meaning no consumer group is referencing any message), `s->cgroup_ref` will not be recreated. As a result, when executing XADD or XTRIM with `ACKED` option and checking whether a message that is being read but has not been ACKed can be deleted, the cgroup_ref being NULL will cause a crash. ### Code Path ``` xaddCommand -> streamTrim -> streamEntryIsReferenced ``` ### Solution Check if `s->cgroup_ref` is NULL in streamEntryIsReferenced().

@oranagra

…fter reload (redis#14276) This bug was introduced by redis#14130 found by @oranagra ### Summary Because `s->cgroup_ref` is created at runtime the first time a consumer group is linked with a message, but it is not released when all references are removed. However, after `debug reload` or restart, if the PEL is empty (meaning no consumer group is referencing any message), `s->cgroup_ref` will not be recreated. As a result, when executing XADD or XTRIM with `ACKED` option and checking whether a message that is being read but has not been ACKed can be deleted, the cgroup_ref being NULL will cause a crash. ### Code Path ``` xaddCommand -> streamTrim -> streamEntryIsReferenced ``` ### Solution Check if `s->cgroup_ref` is NULL in streamEntryIsReferenced().

@oranagra

…fter reload (redis#14276) This bug was introduced by redis#14130 found by @oranagra ### Summary Because `s->cgroup_ref` is created at runtime the first time a consumer group is linked with a message, but it is not released when all references are removed. However, after `debug reload` or restart, if the PEL is empty (meaning no consumer group is referencing any message), `s->cgroup_ref` will not be recreated. As a result, when executing XADD or XTRIM with `ACKED` option and checking whether a message that is being read but has not been ACKed can be deleted, the cgroup_ref being NULL will cause a crash. ### Code Path ``` xaddCommand -> streamTrim -> streamEntryIsReferenced ``` ### Solution Check if `s->cgroup_ref` is NULL in streamEntryIsReferenced().

@oranagra

…fter reload (redis#14276) This bug was introduced by redis#14130 found by @oranagra ### Summary Because `s->cgroup_ref` is created at runtime the first time a consumer group is linked with a message, but it is not released when all references are removed. However, after `debug reload` or restart, if the PEL is empty (meaning no consumer group is referencing any message), `s->cgroup_ref` will not be recreated. As a result, when executing XADD or XTRIM with `ACKED` option and checking whether a message that is being read but has not been ACKed can be deleted, the cgroup_ref being NULL will cause a crash. ### Code Path ``` xaddCommand -> streamTrim -> streamEntryIsReferenced ``` ### Solution Check if `s->cgroup_ref` is NULL in streamEntryIsReferenced().

@oranagra

…fter reload (redis#14276) This bug was introduced by redis#14130 found by @oranagra ### Summary Because `s->cgroup_ref` is created at runtime the first time a consumer group is linked with a message, but it is not released when all references are removed. However, after `debug reload` or restart, if the PEL is empty (meaning no consumer group is referencing any message), `s->cgroup_ref` will not be recreated. As a result, when executing XADD or XTRIM with `ACKED` option and checking whether a message that is being read but has not been ACKed can be deleted, the cgroup_ref being NULL will cause a crash. ### Code Path ``` xaddCommand -> streamTrim -> streamEntryIsReferenced ``` ### Solution Check if `s->cgroup_ref` is NULL in streamEntryIsReferenced().

@oranagra

…fter reload (redis#14276) This bug was introduced by redis#14130 found by @oranagra ### Summary Because `s->cgroup_ref` is created at runtime the first time a consumer group is linked with a message, but it is not released when all references are removed. However, after `debug reload` or restart, if the PEL is empty (meaning no consumer group is referencing any message), `s->cgroup_ref` will not be recreated. As a result, when executing XADD or XTRIM with `ACKED` option and checking whether a message that is being read but has not been ACKed can be deleted, the cgroup_ref being NULL will cause a crash. ### Code Path ``` xaddCommand -> streamTrim -> streamEntryIsReferenced ``` ### Solution Check if `s->cgroup_ref` is NULL in streamEntryIsReferenced().

@oranagra

…fter reload (redis#14276) This bug was introduced by redis#14130 found by @oranagra ### Summary Because `s->cgroup_ref` is created at runtime the first time a consumer group is linked with a message, but it is not released when all references are removed. However, after `debug reload` or restart, if the PEL is empty (meaning no consumer group is referencing any message), `s->cgroup_ref` will not be recreated. As a result, when executing XADD or XTRIM with `ACKED` option and checking whether a message that is being read but has not been ACKed can be deleted, the cgroup_ref being NULL will cause a crash. ### Code Path ``` xaddCommand -> streamTrim -> streamEntryIsReferenced ``` ### Solution Check if `s->cgroup_ref` is NULL in streamEntryIsReferenced().

…egies (redis#14623) This bug was introduced by redis#14130 and found by guybe7 When using XTRIM/XADD with approx mode (~) and DELREF/ACKED delete strategies, if a node was eligible for removal but couldn't be removed directly (because consumer group references need to be checked), the code would incorrectly break out of the loop instead of continuing to process entries within the node. This fix allows the per-entry deletion logic to execute for eligible nodes when using non-KEEPREF strategies.

sundb added 2 commits June 15, 2025 18:47

new steam command

0240e9b

Add XDELEX and XACKDEL commands for stream

9d6304d

sundb added 8 commits June 17, 2025 15:48

Refine xackdelCommand()

3eb2eb5

Add missing Propagate

8633bbd

Refine xdelex command

12adeaa

Refine naming

95858cb

Refine naming

a4b5521

Naming

9dc1e04

Naming

aa83496

Naming

3f248b6

sundb requested a review from Copilot June 17, 2025 09:28

This comment was marked as outdated.

Sign in to view

sundb and others added 5 commits June 17, 2025 17:38

Update tests/unit/type/stream.tcl

6e39dc1

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Refine error msg

97630c4

Refine code

42fb495

Add new option KEEPREF and rename DELPEL to DELREF

cbcfd49

Add new option for XADD and XTRIM commands and update schame

05f7a14

sundb force-pushed the new_stream_command branch from 79d3438 to 05f7a14 Compare June 19, 2025 08:26

sundb added 10 commits June 19, 2025 16:55

Fix schema

b771325

Update schema

313720e

Add streamAckDelArgs

ef6b535

Cleanup comment

db1a15a

Create entry_cgroups_index in runtime

28fd35e

Remove dup code and refine comment

ba1a6bf

Refine code

5478702

Fix module test failed

df0fa05

Refine method names

6a03335

Fix default delete strategy

f2e8167

sundb added this to Redis 8.2 Jun 21, 2025

This was referenced Jul 5, 2025

Add new XACKDEL commands to commands.json redis/rueidis#878

Closed

Add new XDELEX commands to commands.json redis/rueidis#879

Closed

Ryan2327 mentioned this pull request Jul 5, 2025

Add DELREF and ACKED support to XADD redis/rueidis#881

Merged

sundb mentioned this pull request Jul 7, 2025

Feature request: allow remove message with XACK #6399

Closed

uglide mentioned this pull request Jul 24, 2025

Add support for new stream commands redis/jedis#4211

Merged

This was referenced Aug 1, 2025

Streams new commands StefanPalashev/redis-rs#7

Closed

Support Redis 8.2 stream command extensions and new deletion commands redis-rs/redis-rs#1743

Merged

sundb deleted the new_stream_command branch August 7, 2025 02:07

sundb mentioned this pull request Aug 14, 2025

Prevent crash when cgroups_ref is null in streamEntryIsReferenced() after reload #14276

Merged

sundb mentioned this pull request Sep 1, 2025

Fix defrag issues for stream defrag and HFE #14323

Merged

sundb mentioned this pull request Nov 21, 2025

Fix min_cgroup_last_id cache not updated when destroying consumer group #14552

Merged

sundb mentioned this pull request Dec 17, 2025

Fix XTRIM/XADD with approx not deletes entries for DELREF/ACKED strategies #14623

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add XDELEX and XACKDEL commands for stream#14130

Add XDELEX and XACKDEL commands for stream#14130
sundb merged 83 commits intoredis:unstablefrom
sundb:new_stream_command

sundb commented Jun 15, 2025 •

edited

Loading

Uh oh!

snyk-io bot commented Jun 15, 2025 •

edited

Loading

Uh oh!

This comment was marked as outdated.

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Conversation

sundb commented Jun 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary and detailed design for new stream command

XDELEX

Syntax

Description

Reply

XACKDEL

Syntax

Description

Reply

Redis Streams Commands Extension

XTRIM

Syntax

Description

Reply

XADD

Syntax

Description

Reply

Key implementation

Uh oh!

snyk-io bot commented Jun 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🎉 Snyk checks have passed. No issues have been found so far.

Uh oh!

This comment was marked as outdated.

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

sundb commented Jun 15, 2025 •

edited

Loading

snyk-io bot commented Jun 15, 2025 •

edited

Loading