Skip to content

Fix engine crash on module client blocking during keyspace events#1819

Merged
madolson merged 13 commits into
valkey-io:unstablefrom
yairgott:client_block
Apr 18, 2025
Merged

Fix engine crash on module client blocking during keyspace events#1819
madolson merged 13 commits into
valkey-io:unstablefrom
yairgott:client_block

Conversation

@yairgott

@yairgott yairgott commented Mar 5, 2025

Copy link
Copy Markdown
Contributor

This change enhances user experience and consistency by allowing a module to block a client on keyspace event notifications. Consistency is improved by allowing that reads after writes on the same connection yield expected results. For example, in ValkeySearch, mutations processed earlier on the same connection will be available for search.

The implementation extends VM_BlockClient to support blocking clients on keyspace event notifications. Internal clients, LUA clients, clients issueing multi exec and those with the deny_blocking flag set are not blocked. Once blocked, a client’s reply is withheld until it is explicitly unblocked.

@yairgott yairgott changed the title Enable Blocking a Client on Keyspace Event Notification Support Blocking a Client on Keyspace Event Notification Mar 5, 2025
@yairgott yairgott force-pushed the client_block branch 4 times, most recently from a064f39 to 74cb5e4 Compare March 5, 2025 07:08
@codecov

codecov Bot commented Mar 5, 2025

Copy link
Copy Markdown

Codecov Report

Attention: Patch coverage is 81.64557% with 29 lines in your changes missing coverage. Please review.

Project coverage is 71.03%. Comparing base (3f6581b) to head (95b4ca8).
Report is 59 commits behind head on unstable.

Files with missing lines Patch % Lines
src/module.c 10.00% 27 Missing ⚠️
src/networking.c 95.00% 2 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##           unstable    #1819      +/-   ##
============================================
+ Coverage     70.99%   71.03%   +0.04%     
============================================
  Files           123      123              
  Lines         65651    66033     +382     
============================================
+ Hits          46609    46909     +300     
- Misses        19042    19124      +82     
Files with missing lines Coverage Δ
src/adlist.c 77.50% <ø> (+0.57%) ⬆️
src/bitops.c 94.07% <100.00%> (+0.01%) ⬆️
src/expire.c 96.59% <100.00%> (ø)
src/module.h 0.00% <ø> (ø)
src/notify.c 97.22% <100.00%> (+0.20%) ⬆️
src/server.c 87.58% <100.00%> (+0.01%) ⬆️
src/server.h 100.00% <ø> (ø)
src/t_hash.c 96.19% <100.00%> (-0.04%) ⬇️
src/t_list.c 92.88% <100.00%> (+0.32%) ⬆️
src/t_set.c 97.50% <100.00%> (-0.35%) ⬇️
... and 5 more

... and 28 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@PingXie PingXie requested a review from enjoy-binbin March 6, 2025 00:44
Comment thread tests/modules/block_keyspace_notification.c
Comment thread tests/unit/moduleapi/block_keyspace_notification.tcl Outdated
Comment thread src/t_zset.c Outdated
Comment thread src/module.c Outdated
Comment thread src/networking.c Outdated
Comment thread tests/unit/moduleapi/block_keyspace_notification.tcl
Comment thread src/db.c Outdated
Comment thread src/module.c Outdated
Comment thread src/module.c Outdated
Comment thread src/module.c
@yairgott yairgott force-pushed the client_block branch 3 times, most recently from 2428ae0 to 7c9cff2 Compare March 6, 2025 23:06
Comment thread tests/unit/moduleapi/block_keyspace_notification.tcl Outdated
@yairgott yairgott force-pushed the client_block branch 9 times, most recently from dd60c6a to 0445d46 Compare March 7, 2025 22:00
zuiderkwast pushed a commit that referenced this pull request Apr 23, 2025
See
#1819 (comment)

Verified the fix by:
1. Build: `make SERVER_CFLAGS='-Werror -DLOG_REQ_RES' -j10`
2. Running module api tests: `CFLAGS='-Werror' ./runtest-moduleapi
--log-req-res --no-latency --dont-clean --force-resp3 --dont-pre-clean
--verbose --dump-logs`

---------

Signed-off-by: yairgott <yairgott@gmail.com>
Signed-off-by: Jacob Murphy <jkmurphy@google.com>
hwware pushed a commit to wuranxx/valkey that referenced this pull request Apr 24, 2025
…lkey-io#1819)

This change enhances user experience and consistency by allowing a
module to block a client on keyspace event notifications. Consistency is
improved by allowing that reads after writes on the same connection
yield expected results. For example, in ValkeySearch, mutations
processed earlier on the same connection will be available for search.

The implementation extends `VM_BlockClient` to support blocking clients
on keyspace event notifications. Internal clients, LUA clients, clients
issueing multi exec and those with the `deny_blocking` flag set are not
blocked. Once blocked, a client’s reply is withheld until it is
explicitly unblocked.

---------

Signed-off-by: yairgott <yairgott@gmail.com>
Signed-off-by: hwware <wen.hui.ware@gmail.com>
hwware pushed a commit to wuranxx/valkey that referenced this pull request Apr 24, 2025
See
valkey-io#1819 (comment)

Verified the fix by:
1. Build: `make SERVER_CFLAGS='-Werror -DLOG_REQ_RES' -j10`
2. Running module api tests: `CFLAGS='-Werror' ./runtest-moduleapi
--log-req-res --no-latency --dont-clean --force-resp3 --dont-pre-clean
--verbose --dump-logs`

---------

Signed-off-by: yairgott <yairgott@gmail.com>
Signed-off-by: hwware <wen.hui.ware@gmail.com>
rainsupreme pushed a commit to rainsupreme/valkey that referenced this pull request May 14, 2025
See
valkey-io#1819 (comment)

Verified the fix by:
1. Build: `make SERVER_CFLAGS='-Werror -DLOG_REQ_RES' -j10`
2. Running module api tests: `CFLAGS='-Werror' ./runtest-moduleapi
--log-req-res --no-latency --dont-clean --force-resp3 --dont-pre-clean
--verbose --dump-logs`

---------

Signed-off-by: yairgott <yairgott@gmail.com>
Comment thread src/module.c
Comment thread src/module.c
enjoy-binbin added a commit to enjoy-binbin/valkey that referenced this pull request Jun 15, 2025
This flag.deny_blocking check causes some code to becomde dead code.
Some of the code below checks islua or ismulti, but they are marked
with flag.deny_blocking and will return early.

In addition, cleanup some documents, some of them are inaccurate,
and restore the code of blocking_auth_cb in tests/modules/auth.c,
this reply should be returned from core. The reply used to from the
core and was changed to from the module in valkey-io#1819, and now it is from
the core again.

Cleanup some dead code around valkey-io#1819.

Signed-off-by: Binbin <binloveplay1314@qq.com>
enjoy-binbin added a commit to enjoy-binbin/valkey that referenced this pull request Jun 30, 2025
When we add f8a5a4f, a module auth
can add reply in its callback, or not add reply and just return an
error robj in its callback, and we have to figure a way to determine
whether it has added the reply.

Before we are using `clientHasPendingReplies` and it had a issue,
the auth might be used inside a transaction or pipeline which already
has some pending replies in the client reply list. It causes us to
not add the error reply to auth.

After valkey-io#1819, now we have a buffered_reply client flag, it indicates
the reply for the current command was buffered, either in client::reply
or in client::buf. We can use this flag to check.

Fixes valkey-io#2106.

Signed-off-by: Binbin <binloveplay1314@qq.com>
ranshid pushed a commit that referenced this pull request Jul 1, 2025
When we add f8a5a4f, a module auth
can add reply in its callback, or not add reply and just return an
error robj in its callback, and we have to figure a way to determine
whether it has added the reply.

Before we are using `clientHasPendingReplies` and it had a issue,
the auth might be used inside a transaction or pipeline which already
has some pending replies in the client reply list. It causes us to
not add the error reply to auth.

After #1819, now we have a buffered_reply client flag, it indicates
the reply for the current command was buffered, either in client::reply
or in client::buf. We can use this flag to check.

Fixes #2106.

---------

Signed-off-by: Binbin <binloveplay1314@qq.com>
enjoy-binbin added a commit to enjoy-binbin/valkey that referenced this pull request Jul 2, 2025
…y-io#2287)

When we add f8a5a4f, a module auth
can add reply in its callback, or not add reply and just return an
error robj in its callback, and we have to figure a way to determine
whether it has added the reply.

Before we are using `clientHasPendingReplies` and it had a issue,
the auth might be used inside a transaction or pipeline which already
has some pending replies in the client reply list. It causes us to
not add the error reply to auth.

After valkey-io#1819, now we have a buffered_reply client flag, it indicates
the reply for the current command was buffered, either in client::reply
or in client::buf. We can use this flag to check.

Fixes valkey-io#2106.

---------

Signed-off-by: Binbin <binloveplay1314@qq.com>
enjoy-binbin added a commit that referenced this pull request Sep 2, 2025
…#2215)

This flag.deny_blocking check causes some code to becomde dead code.
Some of the code below checks islua or ismulti, but they are marked
with flag.deny_blocking and will return early.

In addition, cleanup some documents, some of them are inaccurate,
and restore the code of blocking_auth_cb in tests/modules/auth.c,
this reply should be returned from core. The reply used to from the
core and was changed to from the module in #1819, and now it is from
the core again.

Cleanup some dead code around #1819.

Signed-off-by: Binbin <binloveplay1314@qq.com>
rjd15372 pushed a commit to rjd15372/valkey that referenced this pull request Sep 19, 2025
…valkey-io#2215)

This flag.deny_blocking check causes some code to becomde dead code.
Some of the code below checks islua or ismulti, but they are marked
with flag.deny_blocking and will return early.

In addition, cleanup some documents, some of them are inaccurate,
and restore the code of blocking_auth_cb in tests/modules/auth.c,
this reply should be returned from core. The reply used to from the
core and was changed to from the module in valkey-io#1819, and now it is from
the core again.

Cleanup some dead code around valkey-io#1819.

Signed-off-by: Binbin <binloveplay1314@qq.com>
rjd15372 pushed a commit that referenced this pull request Sep 23, 2025
…#2215)

This flag.deny_blocking check causes some code to becomde dead code.
Some of the code below checks islua or ismulti, but they are marked
with flag.deny_blocking and will return early.

In addition, cleanup some documents, some of them are inaccurate,
and restore the code of blocking_auth_cb in tests/modules/auth.c,
this reply should be returned from core. The reply used to from the
core and was changed to from the module in #1819, and now it is from
the core again.

Cleanup some dead code around #1819.

Signed-off-by: Binbin <binloveplay1314@qq.com>
hpatro pushed a commit to hpatro/valkey that referenced this pull request Oct 3, 2025
…valkey-io#2215)

This flag.deny_blocking check causes some code to becomde dead code.
Some of the code below checks islua or ismulti, but they are marked
with flag.deny_blocking and will return early.

In addition, cleanup some documents, some of them are inaccurate,
and restore the code of blocking_auth_cb in tests/modules/auth.c,
this reply should be returned from core. The reply used to from the
core and was changed to from the module in valkey-io#1819, and now it is from
the core again.

Cleanup some dead code around valkey-io#1819.

Signed-off-by: Binbin <binloveplay1314@qq.com>
Signed-off-by: Harkrishn Patro <harkrisp@amazon.com>
ranshid pushed a commit that referenced this pull request May 18, 2026
…cations (#3743)

When a module subscribes to NOTIFY_HASH keyspace events and blocks the
client in the notification callback, hexpireGenericCommand,
hgetdelCommand,
and hpersistCommand would trigger debugServerAssert in
notifyKeyspaceEvent()
because addReplyArrayLen() sets buffered_reply=1 before the notification
fires. This debug assert was added in #1819.

Affected commands: HEXPIRE, HPEXPIRE, HEXPIREAT, HPEXPIREAT, HGETDEL,
HPERSIST.

---------

Signed-off-by: Binbin <binloveplay1314@qq.com>
valkeyrie-ops Bot pushed a commit that referenced this pull request May 18, 2026
…cations (#3743)

When a module subscribes to NOTIFY_HASH keyspace events and blocks the
client in the notification callback, hexpireGenericCommand,
hgetdelCommand,
and hpersistCommand would trigger debugServerAssert in
notifyKeyspaceEvent()
because addReplyArrayLen() sets buffered_reply=1 before the notification
fires. This debug assert was added in #1819.

Affected commands: HEXPIRE, HPEXPIRE, HEXPIREAT, HPEXPIREAT, HGETDEL,
HPERSIST.

---------

Signed-off-by: Binbin <binloveplay1314@qq.com>
valkeyrie-ops Bot pushed a commit that referenced this pull request May 20, 2026
…cations (#3743)

When a module subscribes to NOTIFY_HASH keyspace events and blocks the
client in the notification callback, hexpireGenericCommand,
hgetdelCommand,
and hpersistCommand would trigger debugServerAssert in
notifyKeyspaceEvent()
because addReplyArrayLen() sets buffered_reply=1 before the notification
fires. This debug assert was added in #1819.

Affected commands: HEXPIRE, HPEXPIRE, HEXPIREAT, HPEXPIREAT, HGETDEL,
HPERSIST.

---------

Signed-off-by: Binbin <binloveplay1314@qq.com>
valkeyrie-ops Bot pushed a commit that referenced this pull request May 29, 2026
…cations (#3743)

When a module subscribes to NOTIFY_HASH keyspace events and blocks the
client in the notification callback, hexpireGenericCommand,
hgetdelCommand,
and hpersistCommand would trigger debugServerAssert in
notifyKeyspaceEvent()
because addReplyArrayLen() sets buffered_reply=1 before the notification
fires. This debug assert was added in #1819.

Affected commands: HEXPIRE, HPEXPIRE, HEXPIREAT, HPEXPIREAT, HGETDEL,
HPERSIST.

---------

Signed-off-by: Binbin <binloveplay1314@qq.com>
valkeyrie-ops Bot pushed a commit that referenced this pull request Jun 10, 2026
…cations (#3743)

When a module subscribes to NOTIFY_HASH keyspace events and blocks the
client in the notification callback, hexpireGenericCommand,
hgetdelCommand,
and hpersistCommand would trigger debugServerAssert in
notifyKeyspaceEvent()
because addReplyArrayLen() sets buffered_reply=1 before the notification
fires. This debug assert was added in #1819.

Affected commands: HEXPIRE, HPEXPIRE, HEXPIREAT, HPEXPIREAT, HGETDEL,
HPERSIST.

---------

Signed-off-by: Binbin <binloveplay1314@qq.com>
valkeyrie-ops Bot pushed a commit that referenced this pull request Jun 17, 2026
…cations (#3743)

When a module subscribes to NOTIFY_HASH keyspace events and blocks the
client in the notification callback, hexpireGenericCommand,
hgetdelCommand,
and hpersistCommand would trigger debugServerAssert in
notifyKeyspaceEvent()
because addReplyArrayLen() sets buffered_reply=1 before the notification
fires. This debug assert was added in #1819.

Affected commands: HEXPIRE, HPEXPIRE, HEXPIREAT, HPEXPIREAT, HGETDEL,
HPERSIST.

---------

Signed-off-by: Binbin <binloveplay1314@qq.com>
enjoy-binbin added a commit to enjoy-binbin/valkey that referenced this pull request Jun 23, 2026
moduleNotifyKeyspaceEvent() runs subscriber callbacks on the executing
client and calls selectDb(dbid) on it, but never restores the DB. When
dbid differs from the client's current DB (e.g. MOVE/COPY notify on the
destination DB), the client is left on the wrong DB, breaking subsequent
commands.

This was introduced by valkey-io#1819, which started reusing the executing client
instead of a temporary one. Save the executing client's DB before the
subscriber loop and restore it afterwards. Covers both MOVE and COPY.

Signed-off-by: Binbin <binloveplay1314@qq.com>
CPUmaker added a commit to jjuleslasarte/valkey that referenced this pull request Jun 23, 2026
…#3381)

Under reply-blocking, module keyspace notifications were delivered only
via the deferred post-commit task, which is registered only when the
client's notify-keyspace-events matches the event. With the default
empty config, or for background jobs (expiry/eviction) that bypass
certification, module subscribers were dropped entirely -- yet modules
filter by their own event mask and should see every event. Notify
modules inline (as upstream does) and defer only the client pub/sub;
certification no longer re-notifies modules.

Inline notification also lets a module block the client from its
callback (valkey-io#1819) under reply-blocking. To keep that durable, re-apply
the reply-blocking boundary when the parked reply is committed on
unblock, unless the write is already acked. The boundary is snapshotted
at commit time, not reused from command start, to avoid a use-after-free
on a reply buffer node freed during the block.

Add tests/unit/moduleapi/block_keyspace_notification_durability.tcl.

Signed-off-by: Chris Li <1070743423@qq.com>
CPUmaker added a commit to jjuleslasarte/valkey that referenced this pull request Jun 23, 2026
…#3381)

Under reply-blocking, module keyspace notifications were delivered only
via the deferred post-commit task, which is registered only when the
client's notify-keyspace-events matches the event. With the default
empty config, or for background jobs (expiry/eviction) that bypass
certification, module subscribers were dropped entirely -- yet modules
filter by their own event mask and should see every event. Notify
modules inline (as upstream does) and defer only the client pub/sub;
certification no longer re-notifies modules.

Inline notification also lets a module block the client from its
callback (valkey-io#1819) under reply-blocking. To keep that durable, re-apply
the reply-blocking boundary when the parked reply is committed on
unblock, unless the write is already acked. The boundary is snapshotted
at commit time, not reused from command start, to avoid a use-after-free
on a reply buffer node freed during the block.

Add tests/unit/moduleapi/block_keyspace_notification_durability.tcl.

Signed-off-by: Chris Li <1070743423@qq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release-notes This issue should get a line item in the release notes

Projects

Status: 8.1.1

Development

Successfully merging this pull request may close these issues.

5 participants