Add new KSN types - overwritten and type_changed #14141

minchopaskal · 2025-06-20T10:19:07Z

What

Add new keyspace notification event types

OVERWRITTEN - emitted when the value of a key is completely overwritten
TYPE_CHANGED - when the value of a key's type changes

Used in Pub/Sub KSN mechanism. Also added module hooks for the new types.

Motivation

Many commands overwrite the value of a key. F.e SET completely overwrites the value of any key, even its type. Other commands that have the REPLACE parameter also do so.

This commit gives more granularity over following such events.
Specific use-case at hand was module that is subscribed to string events for the sole purpose of checking if hash keys get converted to strings via the SET command. Subscribing to type_changed event not only removes the need to subscribe to string events but is also more correct as not only SET can change the type of a key.

List of commands emitting the new events

SET
MSET
COPY
RESTORE
RENAME
BITOP

Each type with STORE operation:

SORT
S*STORE
Z*STORE
GEORADIUS
GEOSEARCHSTORE

Usage example

pub-sub

Emit overwritten and type-changed events...

$ redis-server --notify-keyspace-events KEoc

Generate an overwritten event that also changes the type of a key...

$ redis-cli
127.0.0.1:6379> lpush l 1 2 3
(integer) 3
127.0.0.1:6379> set l x
OK

Subscribe to the events...

$ ./src/redis-cli
127.0.0.1:6379> psubscribe *
1) "psubscribe"
2) "*"
3) (integer) 1
1) "pmessage"
2) "*"
3) "__keyspace@0__:l"
4) "overwritten"
1) "pmessage"
2) "*"
3) "__keyevent@0__:overwritten"
4) "l"
1) "pmessage"
2) "*"
3) "__keyspace@0__:l"
4) "type_changed"
1) "pmessage"
2) "*"
3) "__keyevent@0__:type_changed"
4) "l"

Modules

As with any other KSN type subscribe to the appropriate events

RedisModule_SubscribeToKeyspaceEvents(
      ctx,
      REDISMODULE_NOTIFY_OVERWRITTEN | REDISMODULE_NOTIFY_TYPE_CHANGED | ...
      notificationCallback
);

Implementation notes

Most of the cases are handled in setKeyByLink but for some commands overwriting had to be manually checked - specifically RESTORE, COPY and RENAME manually call dbAddInternal

fix slop in ai generated tests

snyk-io · 2025-06-20T10:19:20Z

🎉 Snyk checks have passed. No issues have been found so far.

✅ security/snyk check is complete. No issues have been found. (View Details)

✅ license/snyk check is complete. No issues have been found. (View Details)

sundb · 2025-06-20T13:53:51Z

I saw that some *store command (similar to SUNIONSTORE) were not mentioned, which all use setKey() to override keys. We'd better list all the involved commands to avoid omissions.

oranagra · 2025-06-25T09:44:37Z

i can't afford to review the code at the moment, commenting on the description.
feel free to tag me on specific topics in the code though.

Is this PR only for module KSN hooks? or also for pubsub? i think it makes sense to cover both (unless we wanna keep the scope small due to lack of time). let's make it clear in the description, and if it covers pubsub, we need to document the syntax.

Most of the cases are handled in dbSetKeyByLink but for some commands overwritting had to be manually checked - RESTORE, COPY and RENAME manually call dbAddInternal

maybe list other commands like GEORADIUS and SUNIONSTORE? just for completeness.

A subject to change is the decision to emit overwrite event when SETRANGE is used in the edge case SETRANGE x 0 <value_bigger_than_x's_value> which is functionally equivalent to SET.

does SETRANGE in such case resets the TTL or LFU? i don't think so, in which case it's an update not an overwrite.

minchopaskal · 2025-06-27T13:05:02Z

I saw that some *store command (similar to SUNIONSTORE) were not mentioned, which all use setKey() to override keys. We'd better list all the involved commands to avoid omissions.

Should we add tests for each such command, or did u only mean to add in the description as oran said ?

sundb · 2025-06-27T13:12:57Z

I saw that some *store command (similar to SUNIONSTORE) were not mentioned, which all use setKey() to override keys. We'd better list all the involved commands to avoid omissions.

Should we add tests for each such command, or did u only mean to add in the description as oran said ?

only add them in the top comment, maybe it would be better if there were tests.

oranagra · 2025-06-29T13:56:41Z

please document a usage example of both module API and pubsub in the description.
also, it looks like documentation in module.c, and in redis.conf is missing.
i'm assuming the new events have to be explicitly enabled via the notify-keyspace-events config?

minchopaskal · 2025-06-30T10:47:46Z

@oranagra Yes, we're also going to need to update the docs. Let me know if the new description with examples is satisfactory.

minchopaskal · 2025-06-30T11:09:22Z

I saw that some *store command (similar to SUNIONSTORE) were not mentioned, which all use setKey() to override keys. We'd better list all the involved commands to avoid omissions.

Should we add tests for each such command, or did u only mean to add in the description as oran said ?

only add them in the top comment, maybe it would be better if there were tests.

added with 1fd74f0

oranagra · 2025-06-30T18:54:21Z

LGTM. do we also generate a keyspace event, in addition to the key event?
i.e. in one the channel is the event type and the data is the key name, and in the other the channel is the key name, and the data is the event.

minchopaskal · 2025-07-01T07:22:31Z

@oranagra yes, of course, since notifyKeyspaceEvent emits both keyspace and keyevent. I'll update the top-comment

minchopaskal · 2025-07-01T09:36:14Z

Docs PR ~~redis/docs#1781~~ redis/docs#1789

redis.conf

sundb

LGTM

oranagra · 2025-07-06T05:09:19Z

LGTM (based on skimming over the interfaces and docs).

This is the General Availability release of Redis Open Source 8.2. ### Major changes compared to 8.0 - Streams - new commands: `XDELEX` and `XACKDEL`; extension to `XADD` and `XTRIM` - Bitmap - `BITOP`: new operators: `DIFF`, `DIFF1`, `ANDOR`, and `ONE` - Query Engine - new SVS-VAMANA vector index type which supports vector compression - More than 15 performance and resource utilization improvements - New metrics: per-slot usage metrics, key size distributions for basic data types, and more ### Binary distributions - Alpine and Debian Docker images - https://hub.docker.com/_/redis - Install using snap - see https://github.com/redis/redis-snap - Install using brew - see https://github.com/redis/homebrew-redis - Install using RPM - see https://github.com/redis/redis-rpm - Install using Debian APT - see https://github.com/redis/redis-debian ### Operating systems we test Redis 8.2 on - Ubuntu 22.04 (Jammy Jellyfish), 24.04 (Noble Numbat) - Rocky Linux 8.10, 9.5 - AlmaLinux 8.10, 9.5 - Debian 12 (Bookworm) - macOS 13 (Ventura), 14 (Sonoma), 15 (Sequoia) ### Security fixes (compared to 8.2-RC1) - (CVE-2025-32023) Fix out-of-bounds write in `HyperLogLog` commands - (CVE-2025-48367) Retry accepting other connections even if the accepted connection reports an error ### New Features (compared to 8.2-RC1) - #14141 Keyspace notifications - new event types: - `OVERWRITTEN` - the value of a key is completely overwritten - `TYPE_CHANGED` - key type change ### Bug fixes (compared to 8.2-RC1) - #14162 Crash when using evport with I/O threads - #14163 `EVAL` crash when error table is empty - #14144 Vector sets - RDB format is not compatible with big endian machines - #14165 Endless client blocking for blocking commands - #14164 Prevent `CLIENT UNBLOCK` from unblocking `CLIENT PAUSE` - #14216 TTL was not removed by the `SET` command - #14224 `HINCRBYFLOAT` removes field expiration on replica ### Performance and resource utilization improvements (compared to 8.2-RC1) - #14200 Store iterators on stack instead of on heap - #14144 Vector set - improve RDB loading / RESTORE speed by storing the worst link info - #Q6430 More compression variants for the SVS-VAMANA vector index - #Q6535 `SHARD_K_RATIO` parameter - favor network latency over accuracy for KNN vector query in a Redis cluster (unstable feature) (MOD-10359) ### Modules API - #14051 `RedisModule_Get*`, `RedisModule_Set*` - allow modules to access Redis configurations - #14114 `RM_UnsubscribeFromKeyspaceEvents` - unregister a module from specific keyspace notifications

minchopaskal added 4 commits June 18, 2025 20:05

Add overwrite and type_changed KSN types

f258c15

Add restore tests

8e10173

Add overwritten and type_changed to copy and rename

faa8f40

fix slop in ai generated tests

Add overwritten event for SETRANGE when it overwrites the whole value

39e94c8

Fix wrong check in renameGenericCommand

c83bbeb

minchopaskal requested a review from sundb June 20, 2025 11:04

minchopaskal added 2 commits June 20, 2025 15:55

Same hashslot for sets of sunionstore test

bff74dc

Unify key slots for test keys

0e88bab

minchopaskal requested a review from oranagra June 23, 2025 10:04

Revert SETRANGE overwritten event

a7656b0

minchopaskal added the state:needs-doc-pr requires a PR to redis-doc repository label Jun 30, 2025

Add more tests and docs

1fd74f0

minchopaskal mentioned this pull request Jul 1, 2025

Add overwritten and type_changed KSN types to docs redis/docs#1781

Closed

oranagra reviewed Jul 3, 2025

View reviewed changes

redis.conf Outdated Show resolved Hide resolved

Fix comment in redis.conf about all KSN events

9031cff

sundb added the release-notes indication that this issue needs to be mentioned in the release notes label Jul 3, 2025

Exclude overwritten and type_changed from NOTIFY_ALL

2a42a9e

sundb reviewed Jul 4, 2025

View reviewed changes

sundb approved these changes Jul 6, 2025

View reviewed changes

minchopaskal merged commit 86c8be6 into redis:unstable Jul 7, 2025
18 checks passed

minchopaskal mentioned this pull request Jul 7, 2025

Fix docs about the A param of KSN config redis/docs#1803

Merged

sundb mentioned this pull request Aug 4, 2025

Redis 8.2.0 GA #14250

Merged

Add new KSN types - overwritten and type_changed #14141

Add new KSN types - overwritten and type_changed #14141

Uh oh!

Conversation

minchopaskal commented Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Motivation

List of commands emitting the new events

Usage example

pub-sub

Modules

Implementation notes

Uh oh!

snyk-io bot commented Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🎉 Snyk checks have passed. No issues have been found so far.

Uh oh!

sundb commented Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

oranagra commented Jun 25, 2025

Uh oh!

minchopaskal commented Jun 27, 2025

Uh oh!

sundb commented Jun 27, 2025

Uh oh!

oranagra commented Jun 29, 2025

Uh oh!

minchopaskal commented Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

minchopaskal commented Jun 30, 2025

Uh oh!

oranagra commented Jun 30, 2025

Uh oh!

minchopaskal commented Jul 1, 2025

Uh oh!

minchopaskal commented Jul 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

sundb left a comment

Choose a reason for hiding this comment

Uh oh!

oranagra commented Jul 6, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

minchopaskal commented Jun 20, 2025 •

edited

Loading

snyk-io bot commented Jun 20, 2025 •

edited

Loading

sundb commented Jun 20, 2025 •

edited

Loading

minchopaskal commented Jun 30, 2025 •

edited

Loading

minchopaskal commented Jul 1, 2025 •

edited

Loading