Skip to content

Introduce copy-free version streamPropagateXCLAIM to reduce allocation overhead in stream propagation#14516

Merged
sundb merged 5 commits intoredis:unstablefrom
filipecosta90:intrinsics.bswap.itt2
Nov 7, 2025
Merged

Introduce copy-free version streamPropagateXCLAIM to reduce allocation overhead in stream propagation#14516
sundb merged 5 commits intoredis:unstablefrom
filipecosta90:intrinsics.bswap.itt2

Conversation

@fcostaoliveira
Copy link
Collaborator

@fcostaoliveira fcostaoliveira commented Nov 7, 2025

As seen in the following flamegraph, even after PR #14480, there a lot of redundant work when propagating multiple XCLAIMs withing a XREADGROUP.

image

This PR refactors streamPropagateXCLAIM to add a new static inline variant, streamPropagateXCLAIMCopyFree(), which accepts pre-created robj* arguments.
This enables reusing argument objects across multiple XCLAIM propagations, reducing repeated creation and destruction costs during high-throughput consumer group operations.

Functional validation:

tclsh tests/test_helper.tcl --dump-logs --single unit/type/stream
tclsh tests/test_helper.tcl --dump-logs --single unit/type/stream-cgroups

Performance boost

Compared to unstable

Test Case Baseline redis b9ad4f6 (median obs. +- std.dev) Comparison filipecosta90/redis intrinsics.bswap.itt2 (median obs. +- std.dev) % change (higher-better) Note
memtier_benchmark-stream-10M-entries-xreadgroup-count-100 6456 +- 3.2% (3 datapoints) 7194 11.4% Comparison is Better

@sundb sundb added this to Redis 8.4 Nov 7, 2025
@github-project-automation github-project-automation bot moved this to Todo in Redis 8.4 Nov 7, 2025
Copy link
Collaborator

@sggeorgiev sggeorgiev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good.

@fcostaoliveira fcostaoliveira requested a review from sundb November 7, 2025 08:51
@sundb sundb merged commit a64e725 into redis:unstable Nov 7, 2025
19 checks passed
@github-project-automation github-project-automation bot moved this from Todo to Done in Redis 8.4 Nov 7, 2025
@sundb sundb mentioned this pull request Nov 18, 2025
sundb added a commit that referenced this pull request Nov 18, 2025
This is the General Availability release of Redis 8.4 in Redis Open
Source.

### Major changes compared to 8.2

- `DIGEST`, `DELEX`; `SET` extensions - atomic compare-and-set and
compare-and-delete for string keys
- `MSETEX` - atomically set multiple string keys and update their
expiration
- `XREADGROUP` - new `CLAIM` option for reading both idle pending and
incoming stream entries
- `CLUSTER MIGRATION` - atomic slot migration
- `CLUSTER SLOT-STATS` - per-slot usage metrics: key count, CPU time,
and network I/O
- Redis query engine: `FT.HYBRID` - hybrid search and fused scoring
- Redis query engine: I/O threading with performance boost for search
and query commands (FT.*)
- I/O threading: substantial throughput increase (e.g. >30% for caching
use cases (10% `SET`, 90% `GET`), 4 cores)
- JSON: substantial memory reduction for homogenous arrays (up to 91%)

### Binary distributions

- Alpine and Debian Docker images - https://hub.docker.com/_/redis
- Install using snap - see https://github.com/redis/redis-snap
- Install using brew - see https://github.com/redis/homebrew-redis
- Install using RPM - see https://github.com/redis/redis-rpm
- Install using Debian APT - see https://github.com/redis/redis-debian


### Operating systems we test Redis 8.4 on

- Ubuntu 22.04 (Jammy Jellyfish), 24.04 (Noble Numbat)
- Rocky Linux 8.10, 9.5
- AlmaLinux 8.10, 9.5
- Debian 12 (Bookworm), Debian 13 (Trixie)
- macOS 13 (Ventura), 14 (Sonoma), 15 (Sequoia)

### Bug fixes (compared to 8.4-RC1)

- #14524 `XREADGROUP CLAIM` returns strings instead of integers
- #14529 Add variable key-spec flags to SET IF* and DELEX
- #P928 Potential memory leak (MOD-11484)
- #T1801, #T1805 macOS build failures (MOD-12293)
- #J1438 `JSON.NUMINCRBY` - wrong result on integer array with
non-integer increment (MOD-12282)
- #J1437 Thread safety issue related to ASM and shared strings
(MOD-12013)


### Performance and resource utilization improvements (compared to
8.4-RC1)

- #14480, #14516 Optimize `XREADGROUP`

### known bugs and limitations

- When executing `FT.SEARCH`, `FT.AGGREGATE`, `FT.CURSOR`, `FT.HYBRID`,
`TS.MGET`, `TS.MRANGE`, `TS.MREVRANGE` and `TS.QUERYINDEX` while an
atomic slot migration process is in progress, the results may be partial
or contain duplicates
- `FT.PROFILE`, `FT.EXPLAIN` and `FT.EXPLACINCLI` doesn’t contain the
`FT.HYBRID` option
- Metrics from `FT.HYBRID` command aren’t displayed on `FT.INFO` and
`INFO`
- Option `EXPLAINSCORE`, `SHARD_K_RATIO`, `YIELD_DISTANCE_AS` and
`WITHCURSOR` with `FT.HYBRID` are not available
- Post-filtering (after `COMBINE` step) using FILTER is not available
- Currently the default response format considers only `key_id` and
`score`, this may change for delivering entire document content
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants