Lazy loading of RDMA libs in CLI/Benchmark when building as module by quanyeyang · Pull Request #3072 · valkey-io/valkey

quanyeyang · 2026-01-18T04:46:04Z

Summary

This PR decouples the mandatory RDMA library dependency from valkey-cli and valkey-benchmark by implementing a runtime dynamic loading (lazy loading) mechanism.

When BUILD_RDMA=module is used, the CLI tools are no longer hard-linked to librdmacm and libibverbs. Instead, these libraries are only loaded via dlopen when the --rdma flag is explicitly invoked by the user.

Why this is needed

Currently, enabling RDMA support forces a hard dependency on RDMA libraries for all users of CLI tools, even those who only use TCP/UNIX sockets. This creates significant overhead for Linux distribution packaging (see #3070). This PR ensures CLI tools remain lightweight and dependency-free by default.

What has been done already

Dynamic Symbol Loader: Implemented a symbol loader in libvalkey using dlopen() and dlsym(), here: Implement runtime dynamic loading for RDMA libraries libvalkey#284. That was merged and libvalkey was updated in valkey to include this change.
Build System Propagation:
- Makefile: When building as a module, pass USE_DLOPEN_RDMA flag to libvalkey's makefile.
- CMake: When building as a module, pass ENABLE_DLOPEN_RDMA flag to libvalkey's CMake build.

Verification & Testing (Conducted on Arch Linux)

1. Dependency Analysis

make BUILD_RDMA=module
ldd src/valkey-cli | grep -E "rdma|verbs"
# Output: (empty) - Successfully decoupled.

2. Runtime Behavior (Scenario-based)

Scenario : RDMA libraries present, but hardware/driver missing
The loader successfully maps symbols. The program reaches the RDMA stack, which then reports a standard driver error:
Could not connect to Valkey at 127.0.0.1:6379: RDMA: create event channel failed
This confirms the dlopen/dlsym chain is 100% functional and successfully calls into the driver layer.

3. Integration

Standard TCP/UNIX socket connections remain unaffected and function as expected.

Future Plans

This implementation serves as a Proof of Concept for RDMA. If the maintainers agree with this dynamic loading approach, I will provide a follow-up PR to apply the same pattern to TLS (OpenSSL) support.

codecov · 2026-01-18T05:49:33Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 0.00%. Comparing base (262b371) to head (4f3b3b8).
⚠️ Report is 6 commits behind head on unstable.

Additional details and impacted files

@@       Coverage Diff        @@
##   unstable   #3072   +/-   ##
================================
================================

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

zuiderkwast · 2026-01-19T09:31:22Z

Thanks,

so this PR removes TLS and RDMA support in valkey-cli and valkey-benchmark when building the server with there features as modules. Maybe it can break some user's builds? 🤔

If we can load TLS and RDMA support in cli and benchmark using dynamic linking, it would be even better. Let's think about it first. I don't know if it's possible to load the RDMA libraries dynamically.

Conan-Kudo · 2026-01-19T21:55:53Z

Well, it probably could be done if the code for loading this is turned into a module, and that module links the RDMA libraries properly. That's basically what we do for the server, right?

zuiderkwast · 2026-01-19T23:12:35Z

Well, it probably could be done if the code for loading this is turned into a module, and that module links the RDMA libraries properly. That's basically what we do for the server, right?

Yeah but the module machinery is pretty large. I think we can do it in a simpler way:

If valkey-cli is built without statically linked RDMA libraries, we can instead put some code there that will try to dynamically load these libraries just when it's invoked with the --rdma flag. For this to work, they need to be installed in the system, otherwise valkey-cli would print an error message and exit.

To load an .so file, we just need to call dlopen() for each of the required RDMA libraries and then dlsym() to get the pointer to the RDMA functions that we need to call, as is shown under Examples on this page: https://pubs.opengroup.org/onlinepubs/009604299/functions/dlsym.html

Conan-Kudo · 2026-01-19T23:19:57Z

Yeah, a simple dlopen should be sufficient here, but the soversions will change, so we need to be able to handle that properly. That's why I suggested a wrapper module library that links to those libraries normally.

zuiderkwast · 2026-01-19T23:33:15Z

the soversions will change, so we need to be able to handle that properly

Why can't we build for a specific .so version decided at compile-time? Each package is built for specific versions of their dependencies, isn't it?

Conan-Kudo · 2026-01-19T23:43:12Z

We certainly can. There will need to be packaging notes about this.

natoscott · 2026-01-25T23:13:28Z

+1 for a dlopen solution. Dropping this functionality entirely as in the PR isn't a viable solution as users with the server modules still need this.

quanyeyang · 2026-02-20T11:06:56Z

@zuiderkwast I've implemented the dlopen approach as suggested. It now handles RDMA dynamically and falls back gracefully if libraries are missing. CI is passing. Please take a look when you have time!

zuiderkwast

Yeah, I like this approach!

/deps/libvalkey is vendored code though. Those changes should be made in the libvalkey repo and then we'll lift that into Valkey later. We can keep these modification to libvalkey during review, just because it's easier to review it together in one place, but once that's done, let's move the changes to the libvalkey repo. @bjosv @michael-grunder (libvalkey maintainers) please take a look at this.

Reviews from @pizhenwei and @natoscott would also be much appreciated.

Regarding loading OpenSSL in the same way, I think it's lower priority, because OpenSSL is much more generally available and Valkey is more often built with built-in TLS support. But it's good to know that it's possible.

zuiderkwast

Yeah, this looks good to me.

I think the next step is to go and open a PR for libvalkey at https://github.com/valkey-io/libvalkey. Perhaps minor things like the names of these build options can be discussed.

pizhenwei

LGTM, thanks.

natoscott · 2026-02-23T03:38:14Z

LGTM too, thank you @Ada-Church-Closure

zuiderkwast · 2026-02-23T14:31:25Z

@Ada-Church-Closure The vendored libvalkey is updated to 0.4.0 and includes this functionality now. Please update this PR when you have time.

quanyeyang · 2026-02-24T06:25:56Z

I have updated, is this right?

quanyeyang · 2026-02-24T12:25:42Z

I'm sorry, let me check again.

Signed-off-by: Ada-Church-Closure <nunotabashinobu066@gmail.com>

zuiderkwast

I don't think we have any CI job that builds RDMA as a module using CMake. Can you test it manually? We should check that binary doesn't contain the RDMA libs.

I'm uncertain about the CMake code. We should do similar to how we do in the Makefile, where we statically link libvalkey_rdma.a but not the $(RDMA_LIBS).

Btw, it's easier to review if you just push multiple small commits. Then I can follow what happened since last time I checked. We will squash-merge in the end anyway.

Signed-off-by: Ada-Church-Closure <nunotabashinobu066@gmail.com>

quanyeyang · 2026-02-25T01:34:27Z

I don't think we have any CI job that builds RDMA as a module using CMake. Can you test it manually? We should check that binary doesn't contain the RDMA libs.

I'm uncertain about the CMake code. We should do similar to how we do in the Makefile, where we statically link libvalkey_rdma.a but not the $(RDMA_LIBS).

Btw, it's easier to review if you just push multiple small commits. Then I can follow what happened since last time I checked. We will squash-merge in the end anyway.

1.I have test it manually:

~/Project/valkey/build-cmake-test fix/link-problem-3070* 21s ❯ make valkey-cli valkey-benchmark
[  0%] Building C object deps/fpconv/CMakeFiles/fpconv.dir/fpconv_dtoa.c.o
[  0%] Linking C static library ../../lib/libfpconv.a
[  0%] Built target fpconv
[  5%] Generating commands_def_generated
Processing json files...
Linking container command to subcommands...
Checking all commands...
Generating commands.def...
All done, exiting.
[  5%] Built target generate_commands_def
[  5%] Generating fmtargs_generated
[  5%] Built target generate_fmtargs_h
[  5%] Building C object deps/libvalkey/CMakeFiles/valkey.dir/src/adlist.c.o
[ 11%] Building C object deps/libvalkey/CMakeFiles/valkey.dir/src/alloc.c.o
[ 11%] Building C object deps/libvalkey/CMakeFiles/valkey.dir/src/async.c.o
[ 16%] Building C object deps/libvalkey/CMakeFiles/valkey.dir/src/cluster.c.o
[ 16%] Building C object deps/libvalkey/CMakeFiles/valkey.dir/src/command.c.o
[ 16%] Building C object deps/libvalkey/CMakeFiles/valkey.dir/src/conn.c.o
[ 22%] Building C object deps/libvalkey/CMakeFiles/valkey.dir/src/crc16.c.o
[ 22%] Building C object deps/libvalkey/CMakeFiles/valkey.dir/src/net.c.o
[ 27%] Building C object deps/libvalkey/CMakeFiles/valkey.dir/src/read.c.o
[ 27%] Building C object deps/libvalkey/CMakeFiles/valkey.dir/src/sockcompat.c.o
[ 33%] Building C object deps/libvalkey/CMakeFiles/valkey.dir/src/valkey.c.o
[ 33%] Building C object deps/libvalkey/CMakeFiles/valkey.dir/src/vkutil.c.o
[ 33%] Linking C static library ../../lib/libvalkey.a
[ 33%] Built target valkey
[ 38%] Building C object deps/libvalkey/CMakeFiles/valkey_rdma.dir/src/rdma.c.o
[ 38%] Linking C static library ../../lib/libvalkey_rdma.a
[ 38%] Built target valkey_rdma
[ 44%] Building C object deps/linenoise/CMakeFiles/linenoise.dir/linenoise.c.o
[ 44%] Linking C static library ../../lib/liblinenoise.a
[ 44%] Built target linenoise
[ 50%] Building C object src/CMakeFiles/valkey-cli.dir/anet.c.o
[ 50%] Building C object src/CMakeFiles/valkey-cli.dir/adlist.c.o
[ 55%] Building C object src/CMakeFiles/valkey-cli.dir/dict.c.o
[ 55%] Building C object src/CMakeFiles/valkey-cli.dir/sds.c.o
[ 55%] Building C object src/CMakeFiles/valkey-cli.dir/sha256.c.o
[ 61%] Building C object src/CMakeFiles/valkey-cli.dir/util.c.o
[ 61%] Building C object src/CMakeFiles/valkey-cli.dir/valkey-cli.c.o
[ 66%] Building C object src/CMakeFiles/valkey-cli.dir/zmalloc.c.o
[ 66%] Building C object src/CMakeFiles/valkey-cli.dir/release.c.o
[ 72%] Building C object src/CMakeFiles/valkey-cli.dir/ae.c.o
[ 72%] Building C object src/CMakeFiles/valkey-cli.dir/serverassert.c.o
[ 72%] Building C object src/CMakeFiles/valkey-cli.dir/crcspeed.c.o
[ 77%] Building C object src/CMakeFiles/valkey-cli.dir/crccombine.c.o
[ 77%] Building C object src/CMakeFiles/valkey-cli.dir/crc64.c.o
[ 83%] Building C object src/CMakeFiles/valkey-cli.dir/siphash.c.o
[ 83%] Building C object src/CMakeFiles/valkey-cli.dir/crc16.c.o
[ 83%] Building C object src/CMakeFiles/valkey-cli.dir/monotonic.c.o
[ 88%] Building C object src/CMakeFiles/valkey-cli.dir/cli_common.c.o
[ 88%] Building C object src/CMakeFiles/valkey-cli.dir/mt19937-64.c.o
[ 94%] Building C object src/CMakeFiles/valkey-cli.dir/strl.c.o
[ 94%] Building C object src/CMakeFiles/valkey-cli.dir/cli_commands.c.o
[100%] Linking C executable ../bin/valkey-cli
[100%] Built target valkey-cli
[  0%] Building C object deps/hdr_histogram/CMakeFiles/hdr_histogram.dir/hdr_histogram.c.o
[  0%] Linking C static library ../../lib/libhdr_histogram.a
[  0%] Built target hdr_histogram
[  5%] Built target generate_commands_def
[  5%] Built target generate_fmtargs_h
[ 35%] Built target valkey
[ 41%] Built target valkey_rdma
[ 41%] Built target fpconv
[ 47%] Building C object src/CMakeFiles/valkey-benchmark.dir/ae.c.o
[ 47%] Building C object src/CMakeFiles/valkey-benchmark.dir/anet.c.o
[ 52%] Building C object src/CMakeFiles/valkey-benchmark.dir/sds.c.o
[ 52%] Building C object src/CMakeFiles/valkey-benchmark.dir/sha256.c.o
[ 58%] Building C object src/CMakeFiles/valkey-benchmark.dir/util.c.o
[ 58%] Building C object src/CMakeFiles/valkey-benchmark.dir/valkey-benchmark.c.o
[ 58%] Building C object src/CMakeFiles/valkey-benchmark.dir/adlist.c.o
[ 64%] Building C object src/CMakeFiles/valkey-benchmark.dir/dict.c.o
[ 64%] Building C object src/CMakeFiles/valkey-benchmark.dir/zmalloc.c.o
[ 70%] Building C object src/CMakeFiles/valkey-benchmark.dir/serverassert.c.o
[ 70%] Building C object src/CMakeFiles/valkey-benchmark.dir/release.c.o
[ 76%] Building C object src/CMakeFiles/valkey-benchmark.dir/crcspeed.c.o
[ 76%] Building C object src/CMakeFiles/valkey-benchmark.dir/crccombine.c.o
[ 76%] Building C object src/CMakeFiles/valkey-benchmark.dir/crc64.c.o
[ 82%] Building C object src/CMakeFiles/valkey-benchmark.dir/siphash.c.o
[ 82%] Building C object src/CMakeFiles/valkey-benchmark.dir/crc16.c.o
[ 88%] Building C object src/CMakeFiles/valkey-benchmark.dir/monotonic.c.o
[ 88%] Building C object src/CMakeFiles/valkey-benchmark.dir/cli_common.c.o
[ 94%] Building C object src/CMakeFiles/valkey-benchmark.dir/mt19937-64.c.o
[ 94%] Building C object src/CMakeFiles/valkey-benchmark.dir/strl.c.o
[ 94%] Building C object src/CMakeFiles/valkey-benchmark.dir/fuzzer_client.c.o
[100%] Building C object src/CMakeFiles/valkey-benchmark.dir/fuzzer_command_generator.c.o
[100%] Linking C executable ../bin/valkey-benchmark
[100%] Built target valkey-benchmark
~/Project/valkey/build-cmake-test fix/link-problem-3070* ❯ ldd ../src/valkey-cli | grep ibverbs
~/Project/valkey/build-cmake-test fix/link-problem-3070* ❯ ldd ../src/valkey-cli               
        linux-vdso.so.1 (0x00007f0723d46000)
        libm.so.6 => /usr/lib/libm.so.6 (0x00007f0723c09000)
        libsystemd.so.0 => /usr/lib/libsystemd.so.0 (0x00007f07236da000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007f07234e9000)
        /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f0723d48000)
        libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f0723bdc000)
~/Project/valkey/build-cmake-test fix/link-problem-3070* ❯ ldd ../src/valkey-benchmark | grep ibverbs
~/Project/valkey/build-cmake-test fix/link-problem-3070* ❯ ldd ../src/valkey-benchmark               
        linux-vdso.so.1 (0x00007f6bf2295000)
        libm.so.6 => /usr/lib/libm.so.6 (0x00007f6bf2158000)
        libsystemd.so.0 => /usr/lib/libsystemd.so.0 (0x00007f6bf1cda000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007f6bf1ae9000)
        /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f6bf2297000)
        libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f6bf1abc000)

ldd shows they didn't return ibverbs.
2.Thanks, I'll pay attention to git workflow later!

bjosv

Look fine, but have not tested it myself.

zuiderkwast

Looks great, thank you!

Let's update the title and PR description.

…alkey-io#3072) This PR decouples the mandatory RDMA library dependency from `valkey-cli` and `valkey-benchmark` by implementing a **runtime dynamic loading (lazy loading)** mechanism. When `BUILD_RDMA=module` is used, the CLI tools are no longer hard-linked to `librdmacm` and `libibverbs`. Instead, these libraries are only loaded via `dlopen` when the `--rdma` flag is explicitly invoked by the user. - **Dynamic Symbol Loader**: Implemented a symbol loader in libvalkey using dlopen() and dlsym(), here: valkey-io/libvalkey#284. That was merged and libvalkey was updated in valkey to include this change. - **Build System Propagation**: - **Makefile**: When building as a module, pass `USE_DLOPEN_RDMA` flag to libvalkey's makefile. - **CMake**: When building as a module, pass `ENABLE_DLOPEN_RDMA` flag to libvalkey's CMake build. Fixes valkey-io#3070 Signed-off-by: Ada-Church-Closure <nunotabashinobu066@gmail.com> Signed-off-by: Harkrishn Patro <bunty.hari@gmail.com>

github-actions Bot assigned quanyeyang Jan 18, 2026

quanyeyang force-pushed the fix/link-problem-3070 branch from 7d13a0f to ddbe27b Compare January 18, 2026 06:16

quanyeyang force-pushed the fix/link-problem-3070 branch 2 times, most recently from 8da1789 to 3d6ebab Compare February 20, 2026 10:55

zuiderkwast reviewed Feb 20, 2026

View reviewed changes

Comment thread deps/libvalkey/src/rdma.c

Comment thread deps/libvalkey/src/rdma.c

quanyeyang force-pushed the fix/link-problem-3070 branch from 3d6ebab to 3e4bddc Compare February 20, 2026 15:10

zuiderkwast reviewed Feb 20, 2026

View reviewed changes

quanyeyang mentioned this pull request Feb 20, 2026

Implement runtime dynamic loading for RDMA libraries valkey-io/libvalkey#284

Merged

pizhenwei reviewed Feb 23, 2026

View reviewed changes

Comment thread deps/libvalkey/src/rdma.c Outdated

Comment thread deps/libvalkey/src/rdma.c

quanyeyang force-pushed the fix/link-problem-3070 branch from 3e4bddc to b77e237 Compare February 23, 2026 04:03

zuiderkwast mentioned this pull request Feb 23, 2026

Update deps/libvalkey to version 0.4.0 #3216

Merged

zuiderkwast added this to Valkey 9.1 Feb 23, 2026

zuiderkwast moved this to In Progress in Valkey 9.1 Feb 23, 2026

madolson added the release-notes This issue should get a line item in the release notes label Feb 23, 2026

quanyeyang force-pushed the fix/link-problem-3070 branch from b77e237 to 88ab6c5 Compare February 24, 2026 06:20

zuiderkwast reviewed Feb 24, 2026

View reviewed changes

Comment thread deps/Makefile Outdated

Comment thread cmake/Modules/ValkeySetup.cmake Outdated

Comment thread src/Makefile Outdated

Comment thread src/valkey-benchmark.c Outdated

bjosv reviewed Feb 24, 2026

View reviewed changes

Comment thread src/valkey-benchmark.c Outdated

Comment thread deps/Makefile Outdated

Comment thread deps/Makefile Outdated

quanyeyang changed the title ~~Fix static linking of RDMA/TLS libs in CLI/Benchmark when building as module~~ Fix static linking of RDMA libs in CLI/Benchmark when building as module Feb 24, 2026

quanyeyang force-pushed the fix/link-problem-3070 branch from 2ad406b to 7ee326c Compare February 24, 2026 14:57

zuiderkwast reviewed Feb 24, 2026

View reviewed changes

Comment thread src/Makefile

quanyeyang force-pushed the fix/link-problem-3070 branch from 7ee326c to aaae83f Compare February 24, 2026 15:18

Fix: Correct CMake/Makefile RDMA module build config for libvalkey 0.4.0

b61dbb5

Signed-off-by: Ada-Church-Closure <nunotabashinobu066@gmail.com>

quanyeyang force-pushed the fix/link-problem-3070 branch from aaae83f to b61dbb5 Compare February 24, 2026 15:21

zuiderkwast reviewed Feb 24, 2026

View reviewed changes

Comment thread cmake/Modules/ValkeySetup.cmake Outdated

Comment thread deps/CMakeLists.txt Outdated

Refactor: simplify Cmake using USE_RDMA EQUAL 2

4f3b3b8

Signed-off-by: Ada-Church-Closure <nunotabashinobu066@gmail.com>

bjosv approved these changes Feb 25, 2026

View reviewed changes

zuiderkwast approved these changes Feb 26, 2026

View reviewed changes

zuiderkwast changed the title ~~Fix static linking of RDMA libs in CLI/Benchmark when building as module~~ Lazy loading of RDMA libs in CLI/Benchmark when building server with RDMA as module Feb 26, 2026

zuiderkwast changed the title ~~Lazy loading of RDMA libs in CLI/Benchmark when building server with RDMA as module~~ Lazy loading of RDMA libs in CLI/Benchmark when building as module Feb 26, 2026

zuiderkwast merged commit 3a1e5d5 into valkey-io:unstable Feb 26, 2026
56 checks passed

github-project-automation Bot moved this from In Progress to Done in Valkey 9.1 Feb 26, 2026

zuiderkwast mentioned this pull request Feb 26, 2026

[BUG] BUILD_RDMA=module and BUILD_TLS=module uses static linking in cli and benchmark #3070

Closed

zuiderkwast mentioned this pull request Mar 20, 2026

allow to disable dlopen for rdma libs #3383

Open

quanyeyang deleted the fix/link-problem-3070 branch April 22, 2026 06:34

Uh oh!

Conversation

quanyeyang commented Jan 18, 2026 • edited by zuiderkwast Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Why this is needed

What has been done already

Verification & Testing (Conducted on Arch Linux)

1. Dependency Analysis

2. Runtime Behavior (Scenario-based)

3. Integration

Future Plans

Uh oh!

codecov Bot commented Jan 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

zuiderkwast commented Jan 19, 2026

Uh oh!

Conan-Kudo commented Jan 19, 2026

Uh oh!

zuiderkwast commented Jan 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Conan-Kudo commented Jan 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zuiderkwast commented Jan 19, 2026

Uh oh!

Conan-Kudo commented Jan 19, 2026

Uh oh!

natoscott commented Jan 25, 2026

Uh oh!

quanyeyang commented Feb 20, 2026

Uh oh!

zuiderkwast left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

zuiderkwast left a comment

Choose a reason for hiding this comment

Uh oh!

pizhenwei left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

natoscott commented Feb 23, 2026

Uh oh!

zuiderkwast commented Feb 23, 2026

Uh oh!

quanyeyang commented Feb 24, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

quanyeyang commented Feb 24, 2026

Uh oh!

Uh oh!

zuiderkwast left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

quanyeyang commented Feb 25, 2026

Uh oh!

bjosv left a comment

Choose a reason for hiding this comment

Uh oh!

zuiderkwast left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

quanyeyang commented Jan 18, 2026 •

edited by zuiderkwast

Loading

codecov Bot commented Jan 18, 2026 •

edited

Loading

zuiderkwast commented Jan 19, 2026 •

edited

Loading

Conan-Kudo commented Jan 19, 2026 •

edited

Loading

zuiderkwast left a comment •

edited

Loading