Skip to content

Conversation

@judeng
Copy link
Contributor

@judeng judeng commented Jul 8, 2023

Notice: this change was reverted in 8afb72a
we concluded we can't accept it as we should avoid breaking changes for non-critical bug fixes.

Move the TYPE filtering to the scan callback so that avoided the lookupKey operation. This is the follow-up to #12209 . In this thread we introduced two breaking changes:

  1. we will not attempt to do lazy expire (delete) a key that was filtered by not matching the TYPE (like we already do for MATCH pattern).
  2. when the specified key TYPE filter is an unknown type, server will reply a error immediately instead of doing a full scan that comes back empty handed.

Benchmark result:
For different scenarios, we obtained about 48% more performance improvement:

  • scene 1: All keys are volatile keys
scan type match 50% match 100% unmatched
unstable 2665 2924
pr 3954 7463
imporovement 48.37% 155.23%
  • scene 2: All keys are persistent keys
scan type 50% unmatched 100% unmatched
unstable 3229 4117
pr 5175 10985
imporovement 60.27% 166.82%

@oranagra oranagra added the breaking-change This change can potentially break existing application label Jul 9, 2023
@CLAassistant
Copy link

CLAassistant commented Mar 24, 2024

CLA assistant check
All committers have signed the CLA.

@sundb
Copy link
Collaborator

sundb commented Jun 4, 2024

@judeng hi, let's continue with it? thanks.

@judeng
Copy link
Contributor Author

judeng commented Jun 27, 2024

@sundb Thank you, long time no see. I haven't been online in a long time, I'd like fix conflicts in this week.

@judeng
Copy link
Contributor Author

judeng commented Jun 27, 2024

I failed to sign the CLA, the page always return 403 error, I will try it later

@sundb
Copy link
Collaborator

sundb commented Jun 27, 2024

@judeng glad to see you again, thanks.

@sundb
Copy link
Collaborator

sundb commented Jul 16, 2024

@sundb sundb merged commit 7f0a7f0 into redis:unstable Aug 20, 2024
@sundb
Copy link
Collaborator

sundb commented Aug 20, 2024

@judeng it's merged, thanks.

@YaacovHazan YaacovHazan mentioned this pull request Sep 11, 2024
YaacovHazan added a commit that referenced this pull request Sep 12, 2024
### New Features in binary distributions

- 7 new data structures: JSON, Time series, Bloom filter, Cuckoo filter,
Count-min sketch, Top-k, t-digest
- Redis scalable query engine (including vector search)

### Potentially breaking changes

- #12272 `GETRANGE` returns an empty bulk when the negative end index is
out of range
- #12395 Optimize `SCAN` command when matching data type

### Bug fixes

- #13510 Fix `RM_RdbLoad` to enable AOF after RDB loading is completed
- #13489 `ACL CAT` - return module commands
- #13476 Fix a race condition in the `cache_memory` of `functionsLibCtx`
- #13473 Fix incorrect lag due to trimming stream via `XTRIM` command
- #13338 Fix incorrect lag field in `XINFO` when tombstone is after the
`last_id` of the consume group
- #13470 On `HDEL` of last field - update the global hash field
expiration data structure
- #13465 Cluster: Pass extensions to node if extension processing is
handled by it
- #13443 Cluster: Ensure validity of myself when loading cluster config
- #13422 Cluster: Fix `CLUSTER SHARDS` command returns empty array

### Modules API

- #13509 New API calls: `RM_DefragAllocRaw`, `RM_DefragFreeRaw`, and
`RM_RegisterDefragCallbacks` - defrag API to allocate and free raw
memory

### Performance and resource utilization improvements

- #13503 Avoid overhead of comparison function pointer calls in listpack
`lpFind`
- #13505 Optimize `STRING` datatype write commands
- #13499 Optimize `SMEMBERS` command
- #13494 Optimize `GEO*` commands reply
- #13490 Optimize `HELLO` command
- #13488 Optimize client query buffer
- #12395 Optimize `SCAN` command when matching data type
- #13529 Optimize `LREM`, `LPOS`, `LINSERT`, and `LINDEX` commands
- #13516 Optimize `LRANGE` and other commands that perform several
writes to client buffers per call
- #13431 Avoid `used_memory` contention when updating from multiple
threads

### Other general improvements

- #13495 Reply `-LOADING` on replica while flushing the db

### CLI tools

- #13411 redis-cli: Fix wrong `dbnum` showed after the client
reconnected

### Notes

- No backward compatibility for replication or persistence.
- Additional distributions, upgrade paths, features, and improvements
will be introduced in upcoming pre-releases.
- With the GA release of 8.0 we will deprecate Redis Stack.
YaacovHazan pushed a commit to YaacovHazan/redis that referenced this pull request Jan 13, 2025
…edis#12395)"

Although the commit #7f0a7f0a6 improves the performance of the SCAN command,
we can't accept it as we should avoid breaking changes for non-critical bug fixes.

This reverts commit 7f0a7f0.
YaacovHazan pushed a commit that referenced this pull request Feb 5, 2025
…12395)"

Although the commit #7f0a7f0a6 improves the performance of the SCAN command,
we can't accept it as we should avoid breaking changes for non-critical bug fixes.

This reverts commit 7f0a7f0.
@sundb sundb removed this from Redis 8.2 Aug 15, 2025
@sundb sundb added this to Redis 8.0 Aug 15, 2025
@sundb sundb moved this from Todo to Done in Redis 8.0 Aug 15, 2025
funny-dog pushed a commit to funny-dog/redis that referenced this pull request Sep 17, 2025
…2395)

Move the TYPE filtering to the scan callback so that avoided the
`lookupKey` operation. This is the follow-up to redis#12209 . In this thread
we introduced two breaking changes:
1. we will not attempt to do lazy expire (delete) a key that was
filtered by not matching the TYPE (like we already do for MATCH
pattern).
2. when the specified key TYPE filter is an unknown type, server will
reply a error immediately instead of doing a full scan that comes back
empty handed.
funny-dog pushed a commit to funny-dog/redis that referenced this pull request Sep 17, 2025
### New Features in binary distributions

- 7 new data structures: JSON, Time series, Bloom filter, Cuckoo filter,
Count-min sketch, Top-k, t-digest
- Redis scalable query engine (including vector search)

### Potentially breaking changes

- redis#12272 `GETRANGE` returns an empty bulk when the negative end index is
out of range
- redis#12395 Optimize `SCAN` command when matching data type

### Bug fixes

- redis#13510 Fix `RM_RdbLoad` to enable AOF after RDB loading is completed
- redis#13489 `ACL CAT` - return module commands
- redis#13476 Fix a race condition in the `cache_memory` of `functionsLibCtx`
- redis#13473 Fix incorrect lag due to trimming stream via `XTRIM` command
- redis#13338 Fix incorrect lag field in `XINFO` when tombstone is after the
`last_id` of the consume group
- redis#13470 On `HDEL` of last field - update the global hash field
expiration data structure
- redis#13465 Cluster: Pass extensions to node if extension processing is
handled by it
- redis#13443 Cluster: Ensure validity of myself when loading cluster config
- redis#13422 Cluster: Fix `CLUSTER SHARDS` command returns empty array

### Modules API

- redis#13509 New API calls: `RM_DefragAllocRaw`, `RM_DefragFreeRaw`, and
`RM_RegisterDefragCallbacks` - defrag API to allocate and free raw
memory

### Performance and resource utilization improvements

- redis#13503 Avoid overhead of comparison function pointer calls in listpack
`lpFind`
- redis#13505 Optimize `STRING` datatype write commands
- redis#13499 Optimize `SMEMBERS` command
- redis#13494 Optimize `GEO*` commands reply
- redis#13490 Optimize `HELLO` command
- redis#13488 Optimize client query buffer
- redis#12395 Optimize `SCAN` command when matching data type
- redis#13529 Optimize `LREM`, `LPOS`, `LINSERT`, and `LINDEX` commands
- redis#13516 Optimize `LRANGE` and other commands that perform several
writes to client buffers per call
- redis#13431 Avoid `used_memory` contention when updating from multiple
threads

### Other general improvements

- redis#13495 Reply `-LOADING` on replica while flushing the db

### CLI tools

- redis#13411 redis-cli: Fix wrong `dbnum` showed after the client
reconnected

### Notes

- No backward compatibility for replication or persistence.
- Additional distributions, upgrade paths, features, and improvements
will be introduced in upcoming pre-releases.
- With the GA release of 8.0 we will deprecate Redis Stack.
funny-dog pushed a commit to funny-dog/redis that referenced this pull request Sep 17, 2025
…edis#12395)"

Although the commit #779a3ccdb improves the performance of the SCAN command,
we can't accept it as we should avoid breaking changes for non-critical bug fixes.

This reverts commit 779a3cc.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

breaking-change This change can potentially break existing application

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

4 participants