I’m trying to test a regression in Valkey server functionality by loading the Valkey-Search module.
To do this, I added the "loadmodule libsearch.so" in tests/assets/default.conf and ran ./runtest. I expected the tests to pass (aside from known flaky tests), but instead, I observed a consistent crash due to an assertion failure in blocked.c, specifically related to the wait/expire test cases.
I tried manually recreate the scenario but had no luck on the issue. Please correct me if I'm using the test setup in an unsupported way.
This issue occurs reliably with runtest, and I'm not sure if it's expected behavior to load the module via default.conf, especially since we have separate tests for modules.
Crash report
$ ./runtest --verbose --dump-logs
Cleanup: may take some time... OK
Starting test server at port 21079
[ready]: 1361940
Testing unit/acl-v2
[ready]: 1361939
Testing unit/limits
[ready]: 1361941
Testing unit/commandlog
[ready]: 1361942
Testing unit/latency-monitor
[ready]: 1361943
-----
-----
1364596:M 22 May 2025 18:36:18.666 - Client closed connection id=7 addr=127.0.0.1:39319 laddr=127.0.0.1:26614 fd=11 name= age=0 idle=0 flags=N capa= db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37504 events=r cmd=ping user=default redir=-1 resp=2 lib-name= lib-ver= tot-net-in=7 tot-net-out=7 tot-cmds=1
1364596:M 22 May 2025 18:36:18.696 - Accepted 127.0.0.1:33991
### Starting test WAITAOF local copy before fsync in tests/unit/wait.tcl
=== VALKEY BUG REPORT START: Cut & paste starting from here ===
1364596:M 22 May 2025 18:36:18.697 # === ASSERTION FAILED ===
1364596:M 22 May 2025 18:36:18.697 # ==> blocked.c:357 'server.also_propagate.numops == 0' is not true
------ STACK TRACE ------
1364623 bio_close_file
/lib/x86_64-linux-gnu/libc.so.6(+0x91117)[0x7f2bf8904117]
/lib/x86_64-linux-gnu/libc.so.6(pthread_cond_wait+0x211)[0x7f2bf8906a41]
src/valkey-server 127.0.0.1:26614(bioProcessBackgroundJobs+0x1af)[0x556cd453530f]
/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7f2bf8907ac3]
/lib/x86_64-linux-gnu/libc.so.6(+0x126850)[0x7f2bf8999850]
1364605 read-worker-0
/lib/x86_64-linux-gnu/libc.so.6(syscall+0x1d)[0x7f2bf899188d]
./libsearch.so(+0x632f6a)[0x7f2bf6e60f6a]
./libsearch.so(+0x632fe3)[0x7f2bf6e60fe3]
./libsearch.so(+0x632e75)[0x7f2bf6e60e75]
./libsearch.so(+0xbc664)[0x7f2bf68ea664]
./libsearch.so(+0x418fad)[0x7f2bf6c46fad]
./libsearch.so(+0x41a598)[0x7f2bf6c48598]
./libsearch.so(+0x419751)[0x7f2bf6c47751]
/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7f2bf8907ac3]
/lib/x86_64-linux-gnu/libc.so.6(+0x126850)[0x7f2bf8999850]
1364606 read-worker-1
/lib/x86_64-linux-gnu/libc.so.6(syscall+0x1d)[0x7f2bf899188d]
./libsearch.so(+0x632f6a)[0x7f2bf6e60f6a]
./libsearch.so(+0x632fe3)[0x7f2bf6e60fe3]
./libsearch.so(+0x632e75)[0x7f2bf6e60e75]
./libsearch.so(+0xbc664)[0x7f2bf68ea664]
./libsearch.so(+0x418fad)[0x7f2bf6c46fad]
./libsearch.so(+0x41a598)[0x7f2bf6c48598]
./libsearch.so(+0x419751)[0x7f2bf6c47751]
/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7f2bf8907ac3]
/lib/x86_64-linux-gnu/libc.so.6(+0x126850)[0x7f2bf8999850]
1364607 read-worker-2
/lib/x86_64-linux-gnu/libc.so.6(syscall+0x1d)[0x7f2bf899188d]
./libsearch.so(+0x632f6a)[0x7f2bf6e60f6a]
./libsearch.so(+0x632fe3)[0x7f2bf6e60fe3]
./libsearch.so(+0x632e75)[0x7f2bf6e60e75]
./libsearch.so(+0xbc664)[0x7f2bf68ea664]
./libsearch.so(+0x418fad)[0x7f2bf6c46fad]
./libsearch.so(+0x41a598)[0x7f2bf6c48598]
./libsearch.so(+0x419751)[0x7f2bf6c47751]
/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7f2bf8907ac3]
/lib/x86_64-linux-gnu/libc.so.6(+0x126850)[0x7f2bf8999850]
1364596 valkey-server *
src/valkey-server 127.0.0.1:26614(+0x1a25f8)[0x556cd45585f8]
src/valkey-server 127.0.0.1:26614(beforeSleep+0xcb)[0x556cd4459e9b]
src/valkey-server 127.0.0.1:26614(aeMain+0x3f)[0x556cd444ba6f]
src/valkey-server 127.0.0.1:26614(main+0x5b2)[0x556cd443fa92]
/lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f2bf889cd90]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f2bf889ce40]
src/valkey-server 127.0.0.1:26614(_start+0x25)[0x556cd4441275]
1364608 read-worker-3
/lib/x86_64-linux-gnu/libc.so.6(syscall+0x1d)[0x7f2bf899188d]
./libsearch.so(+0x632f6a)[0x7f2bf6e60f6a]
./libsearch.so(+0x632fe3)[0x7f2bf6e60fe3]
./libsearch.so(+0x632e75)[0x7f2bf6e60e75]
./libsearch.so(+0xbc664)[0x7f2bf68ea664]
./libsearch.so(+0x418fad)[0x7f2bf6c46fad]
./libsearch.so(+0x41a598)[0x7f2bf6c48598]
./libsearch.so(+0x419751)[0x7f2bf6c47751]
/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7f2bf8907ac3]
/lib/x86_64-linux-gnu/libc.so.6(+0x126850)[0x7f2bf8999850]
1364609 read-worker-4
/lib/x86_64-linux-gnu/libc.so.6(syscall+0x1d)[0x7f2bf899188d]
./libsearch.so(+0x632f6a)[0x7f2bf6e60f6a]
./libsearch.so(+0x632fe3)[0x7f2bf6e60fe3]
./libsearch.so(+0x632e75)[0x7f2bf6e60e75]
./libsearch.so(+0xbc664)[0x7f2bf68ea664]
./libsearch.so(+0x418fad)[0x7f2bf6c46fad]
./libsearch.so(+0x41a598)[0x7f2bf6c48598]
./libsearch.so(+0x419751)[0x7f2bf6c47751]
/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7f2bf8907ac3]
/lib/x86_64-linux-gnu/libc.so.6(+0x126850)[0x7f2bf8999850]
1364611 read-worker-5
/lib/x86_64-linux-gnu/libc.so.6(syscall+0x1d)[0x7f2bf899188d]
./libsearch.so(+0x632f6a)[0x7f2bf6e60f6a]
./libsearch.so(+0x632fe3)[0x7f2bf6e60fe3]
./libsearch.so(+0x632e75)[0x7f2bf6e60e75]
./libsearch.so(+0xbc664)[0x7f2bf68ea664]
./libsearch.so(+0x418fad)[0x7f2bf6c46fad]
./libsearch.so(+0x41a598)[0x7f2bf6c48598]
./libsearch.so(+0x419751)[0x7f2bf6c47751]
/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7f2bf8907ac3]
/lib/x86_64-linux-gnu/libc.so.6(+0x126850)[0x7f2bf8999850]
1364612 read-worker-6
/lib/x86_64-linux-gnu/libc.so.6(syscall+0x1d)[0x7f2bf899188d]
./libsearch.so(+0x632f6a)[0x7f2bf6e60f6a]
./libsearch.so(+0x632fe3)[0x7f2bf6e60fe3]
./libsearch.so(+0x632e75)[0x7f2bf6e60e75]
./libsearch.so(+0xbc664)[0x7f2bf68ea664]
./libsearch.so(+0x418fad)[0x7f2bf6c46fad]
./libsearch.so(+0x41a598)[0x7f2bf6c48598]
./libsearch.so(+0x419751)[0x7f2bf6c47751]
/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7f2bf8907ac3]
/lib/x86_64-linux-gnu/libc.so.6(+0x126850)[0x7f2bf8999850]
1364613 read-worker-7
/lib/x86_64-linux-gnu/libc.so.6(syscall+0x1d)[0x7f2bf899188d]
./libsearch.so(+0x632f6a)[0x7f2bf6e60f6a]
./libsearch.so(+0x632fe3)[0x7f2bf6e60fe3]
./libsearch.so(+0x632e75)[0x7f2bf6e60e75]
./libsearch.so(+0xbc664)[0x7f2bf68ea664]
./libsearch.so(+0x418fad)[0x7f2bf6c46fad]
./libsearch.so(+0x41a598)[0x7f2bf6c48598]
./libsearch.so(+0x419751)[0x7f2bf6c47751]
/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7f2bf8907ac3]
/lib/x86_64-linux-gnu/libc.so.6(+0x126850)[0x7f2bf8999850]
1364614 write-worker-0
/lib/x86_64-linux-gnu/libc.so.6(syscall+0x1d)[0x7f2bf899188d]
./libsearch.so(+0x632f6a)[0x7f2bf6e60f6a]
./libsearch.so(+0x632fe3)[0x7f2bf6e60fe3]
./libsearch.so(+0x632e75)[0x7f2bf6e60e75]
./libsearch.so(+0xbc664)[0x7f2bf68ea664]
./libsearch.so(+0x418fad)[0x7f2bf6c46fad]
./libsearch.so(+0x41a598)[0x7f2bf6c48598]
./libsearch.so(+0x419751)[0x7f2bf6c47751]
/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7f2bf8907ac3]
/lib/x86_64-linux-gnu/libc.so.6(+0x126850)[0x7f2bf8999850]
1364615 write-worker-1
/lib/x86_64-linux-gnu/libc.so.6(syscall+0x1d)[0x7f2bf899188d]
./libsearch.so(+0x632f6a)[0x7f2bf6e60f6a]
./libsearch.so(+0x632fe3)[0x7f2bf6e60fe3]
./libsearch.so(+0x632e75)[0x7f2bf6e60e75]
./libsearch.so(+0xbc664)[0x7f2bf68ea664]
./libsearch.so(+0x418fad)[0x7f2bf6c46fad]
./libsearch.so(+0x41a598)[0x7f2bf6c48598]
./libsearch.so(+0x419751)[0x7f2bf6c47751]
/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7f2bf8907ac3]
/lib/x86_64-linux-gnu/libc.so.6(+0x126850)[0x7f2bf8999850]
1364616 write-worker-2
/lib/x86_64-linux-gnu/libc.so.6(syscall+0x1d)[0x7f2bf899188d]
./libsearch.so(+0x632f6a)[0x7f2bf6e60f6a]
./libsearch.so(+0x632fe3)[0x7f2bf6e60fe3]
./libsearch.so(+0x632e75)[0x7f2bf6e60e75]
./libsearch.so(+0xbc664)[0x7f2bf68ea664]
./libsearch.so(+0x418fad)[0x7f2bf6c46fad]
./libsearch.so(+0x41a598)[0x7f2bf6c48598]
./libsearch.so(+0x419751)[0x7f2bf6c47751]
/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7f2bf8907ac3]
/lib/x86_64-linux-gnu/libc.so.6(+0x126850)[0x7f2bf8999850]
1364617 write-worker-3
/lib/x86_64-linux-gnu/libc.so.6(syscall+0x1d)[0x7f2bf899188d]
./libsearch.so(+0x632f6a)[0x7f2bf6e60f6a]
./libsearch.so(+0x632fe3)[0x7f2bf6e60fe3]
./libsearch.so(+0x632e75)[0x7f2bf6e60e75]
./libsearch.so(+0xbc664)[0x7f2bf68ea664]
./libsearch.so(+0x418fad)[0x7f2bf6c46fad]
./libsearch.so(+0x41a598)[0x7f2bf6c48598]
./libsearch.so(+0x419751)[0x7f2bf6c47751]
/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7f2bf8907ac3]
/lib/x86_64-linux-gnu/libc.so.6(+0x126850)[0x7f2bf8999850]
1364619 write-worker-4
/lib/x86_64-linux-gnu/libc.so.6(syscall+0x1d)[0x7f2bf899188d]
./libsearch.so(+0x632f6a)[0x7f2bf6e60f6a]
./libsearch.so(+0x632fe3)[0x7f2bf6e60fe3]
./libsearch.so(+0x632e75)[0x7f2bf6e60e75]
./libsearch.so(+0xbc664)[0x7f2bf68ea664]
./libsearch.so(+0x418fad)[0x7f2bf6c46fad]
./libsearch.so(+0x41a598)[0x7f2bf6c48598]
./libsearch.so(+0x419751)[0x7f2bf6c47751]
/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7f2bf8907ac3]
/lib/x86_64-linux-gnu/libc.so.6(+0x126850)[0x7f2bf8999850]
1364620 write-worker-5
/lib/x86_64-linux-gnu/libc.so.6(syscall+0x1d)[0x7f2bf899188d]
./libsearch.so(+0x632f6a)[0x7f2bf6e60f6a]
./libsearch.so(+0x632fe3)[0x7f2bf6e60fe3]
./libsearch.so(+0x632e75)[0x7f2bf6e60e75]
./libsearch.so(+0xbc664)[0x7f2bf68ea664]
./libsearch.so(+0x418fad)[0x7f2bf6c46fad]
./libsearch.so(+0x41a598)[0x7f2bf6c48598]
./libsearch.so(+0x419751)[0x7f2bf6c47751]
/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7f2bf8907ac3]
/lib/x86_64-linux-gnu/libc.so.6(+0x126850)[0x7f2bf8999850]
1364621 write-worker-6
/lib/x86_64-linux-gnu/libc.so.6(syscall+0x1d)[0x7f2bf899188d]
./libsearch.so(+0x632f6a)[0x7f2bf6e60f6a]
./libsearch.so(+0x632fe3)[0x7f2bf6e60fe3]
./libsearch.so(+0x632e75)[0x7f2bf6e60e75]
./libsearch.so(+0xbc664)[0x7f2bf68ea664]
./libsearch.so(+0x418fad)[0x7f2bf6c46fad]
./libsearch.so(+0x41a598)[0x7f2bf6c48598]
./libsearch.so(+0x419751)[0x7f2bf6c47751]
/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7f2bf8907ac3]
/lib/x86_64-linux-gnu/libc.so.6(+0x126850)[0x7f2bf8999850]
1364622 write-worker-7
/lib/x86_64-linux-gnu/libc.so.6(syscall+0x1d)[0x7f2bf899188d]
./libsearch.so(+0x632f6a)[0x7f2bf6e60f6a]
./libsearch.so(+0x632fe3)[0x7f2bf6e60fe3]
./libsearch.so(+0x632e75)[0x7f2bf6e60e75]
./libsearch.so(+0xbc664)[0x7f2bf68ea664]
./libsearch.so(+0x418fad)[0x7f2bf6c46fad]
./libsearch.so(+0x41a598)[0x7f2bf6c48598]
./libsearch.so(+0x419751)[0x7f2bf6c47751]
/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7f2bf8907ac3]
/lib/x86_64-linux-gnu/libc.so.6(+0x126850)[0x7f2bf8999850]
1364624 bio_aof
/lib/x86_64-linux-gnu/libc.so.6(+0x91117)[0x7f2bf8904117]
/lib/x86_64-linux-gnu/libc.so.6(pthread_cond_wait+0x211)[0x7f2bf8906a41]
src/valkey-server 127.0.0.1:26614(bioProcessBackgroundJobs+0x1af)[0x556cd453530f]
/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7f2bf8907ac3]
/lib/x86_64-linux-gnu/libc.so.6(+0x126850)[0x7f2bf8999850]
1364625 bio_lazy_free
/lib/x86_64-linux-gnu/libc.so.6(+0x91117)[0x7f2bf8904117]
/lib/x86_64-linux-gnu/libc.so.6(pthread_cond_wait+0x211)[0x7f2bf8906a41]
src/valkey-server 127.0.0.1:26614(bioProcessBackgroundJobs+0x1af)[0x556cd453530f]
/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7f2bf8907ac3]
/lib/x86_64-linux-gnu/libc.so.6(+0x126850)[0x7f2bf8999850]
20/20 expected stacktraces.
Additional information
valkey-search$ git log -2
commit 41c4ff082a584c599c2c4b6155e2ea61bf7e0c42 (HEAD -> main, origin/main, origin/HEAD)
Author: Yair Gottdenker <yairgott@gmail.com>
Date: Thu May 22 09:34:26 2025 -0700
Fixing empty vector query clause ft.search cmd parsing (#138)
Signed-off-by: yairgott <yairgott@gmail.com>
commit 5001de38f1a39b1c10bf371a9746b5529ba901f1
Author: Yair Gottdenker <yairgott@gmail.com>
Date: Wed May 21 17:44:42 2025 -0700
renaming total_indexed_hash_keys to total_indexed_documents (#137)
* renaming total_indexed_hash_keys to total_indexed_documents
Signed-off-by: yairgott <yairgott@gmail.com>
Config details:
$ cat tests/assets/default.conf
# Server configuration for testing.
always-show-logo yes
notify-keyspace-events KEA
daemonize no
pidfile /var/run/valkey.pid
port 6379
timeout 0
bind 127.0.0.1
loglevel verbose
logfile ''
databases 16
latency-monitor-threshold 1
repl-diskless-sync-delay 0
# Turn off RDB by default (to speedup tests)
# Note the infrastructure in server.tcl uses a dict, we can't provide several save directives
save ''
rdbcompression yes
dbfilename dump.rdb
dir ./
loadmodule <path>/libsearch.so
slave-serve-stale-data yes
appendonly no
appendfsync everysec
no-appendfsync-on-rewrite no
activerehashing yes
enable-protected-configs yes
enable-debug-command yes
enable-module-command yes
propagation-error-behavior panic
# Make sure shutdown doesn't fail if there's an initial AOFRW
shutdown-on-sigterm force
enable-debug-assert yes
hide-user-data-from-log no
I’m trying to test a regression in Valkey server functionality by loading the Valkey-Search module.
To do this, I added the "loadmodule libsearch.so" in tests/assets/default.conf and ran ./runtest. I expected the tests to pass (aside from known flaky tests), but instead, I observed a consistent crash due to an assertion failure in blocked.c, specifically related to the wait/expire test cases.
I tried manually recreate the scenario but had no luck on the issue. Please correct me if I'm using the test setup in an unsupported way.
This issue occurs reliably with runtest, and I'm not sure if it's expected behavior to load the module via default.conf, especially since we have separate tests for modules.
Crash report
Additional information
valkey-server - 8.1.1 release branch
valkey-search - latest main barnch commit id: 41c4ff08
Config details: