Introduce copy-free version streamPropagateXCLAIM to reduce allocation overhead in stream propagation#14516
Merged
sundb merged 5 commits intoredis:unstablefrom Nov 7, 2025
Merged
Conversation
…n overhead in stream propagation
c6d6d45 to
72a9410
Compare
sggeorgiev
reviewed
Nov 7, 2025
sundb
reviewed
Nov 7, 2025
sundb
approved these changes
Nov 7, 2025
Merged
sundb
added a commit
that referenced
this pull request
Nov 18, 2025
This is the General Availability release of Redis 8.4 in Redis Open Source. ### Major changes compared to 8.2 - `DIGEST`, `DELEX`; `SET` extensions - atomic compare-and-set and compare-and-delete for string keys - `MSETEX` - atomically set multiple string keys and update their expiration - `XREADGROUP` - new `CLAIM` option for reading both idle pending and incoming stream entries - `CLUSTER MIGRATION` - atomic slot migration - `CLUSTER SLOT-STATS` - per-slot usage metrics: key count, CPU time, and network I/O - Redis query engine: `FT.HYBRID` - hybrid search and fused scoring - Redis query engine: I/O threading with performance boost for search and query commands (FT.*) - I/O threading: substantial throughput increase (e.g. >30% for caching use cases (10% `SET`, 90% `GET`), 4 cores) - JSON: substantial memory reduction for homogenous arrays (up to 91%) ### Binary distributions - Alpine and Debian Docker images - https://hub.docker.com/_/redis - Install using snap - see https://github.com/redis/redis-snap - Install using brew - see https://github.com/redis/homebrew-redis - Install using RPM - see https://github.com/redis/redis-rpm - Install using Debian APT - see https://github.com/redis/redis-debian ### Operating systems we test Redis 8.4 on - Ubuntu 22.04 (Jammy Jellyfish), 24.04 (Noble Numbat) - Rocky Linux 8.10, 9.5 - AlmaLinux 8.10, 9.5 - Debian 12 (Bookworm), Debian 13 (Trixie) - macOS 13 (Ventura), 14 (Sonoma), 15 (Sequoia) ### Bug fixes (compared to 8.4-RC1) - #14524 `XREADGROUP CLAIM` returns strings instead of integers - #14529 Add variable key-spec flags to SET IF* and DELEX - #P928 Potential memory leak (MOD-11484) - #T1801, #T1805 macOS build failures (MOD-12293) - #J1438 `JSON.NUMINCRBY` - wrong result on integer array with non-integer increment (MOD-12282) - #J1437 Thread safety issue related to ASM and shared strings (MOD-12013) ### Performance and resource utilization improvements (compared to 8.4-RC1) - #14480, #14516 Optimize `XREADGROUP` ### known bugs and limitations - When executing `FT.SEARCH`, `FT.AGGREGATE`, `FT.CURSOR`, `FT.HYBRID`, `TS.MGET`, `TS.MRANGE`, `TS.MREVRANGE` and `TS.QUERYINDEX` while an atomic slot migration process is in progress, the results may be partial or contain duplicates - `FT.PROFILE`, `FT.EXPLAIN` and `FT.EXPLACINCLI` doesn’t contain the `FT.HYBRID` option - Metrics from `FT.HYBRID` command aren’t displayed on `FT.INFO` and `INFO` - Option `EXPLAINSCORE`, `SHARD_K_RATIO`, `YIELD_DISTANCE_AS` and `WITHCURSOR` with `FT.HYBRID` are not available - Post-filtering (after `COMBINE` step) using FILTER is not available - Currently the default response format considers only `key_id` and `score`, this may change for delivering entire document content
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
As seen in the following flamegraph, even after PR #14480, there a lot of redundant work when propagating multiple XCLAIMs withing a XREADGROUP.
This PR refactors streamPropagateXCLAIM to add a new static inline variant,
streamPropagateXCLAIMCopyFree(), which accepts pre-createdrobj*arguments.This enables reusing argument objects across multiple XCLAIM propagations, reducing repeated creation and destruction costs during high-throughput consumer group operations.
Functional validation:
Performance boost
Compared to unstable