Add multi-database support to cluster mode by xbasel · Pull Request #1671 · valkey-io/valkey

xbasel · 2025-02-05T15:28:52Z

cluster: add multi-database support in cluster mode

Add multi-database support in cluster mode to align with standalone mode
and facilitate migration. Previously, cluster mode was restricted to a
single database (DB0). This change allows multiple databases while
preserving the existing slot-based key distribution.

Key Features:

Database-Agnostic Hashing. The hashing algorithm is unchanged.
Identical keys always map to the same slot across all databases,
ensuring consistent key distribution and compatibility with
existing single-database setups.
Multi-DB commands support. SELECT, MOVE, and COPY are now supported in
cluster mode.
Fully backward compatible with no API changes.
SWAPDB is not supported in cluster mode. It is unsafe due to inconsistency risks.

Command-Level Changes:

SELECT / MOVE / COPY are now supported in cluster mode.
MOVE / COPY (with db) are rejected (TRYAGAIN error) during slot migration to prevent multi-DB inconsistencies.
SWAPDB will return an error if used when cluster mode is enabled.
GETKEYSINSLOT, COUNTKEYSINSLOT and MIGRATE will operate in the context of the selected database.
This means, for example, that migrating keys in a slot will require iterating and repeating across all databases.

Slot Migration Process:

Multi-DB support in cluster mode affects slot migration. Operators should now iterate over all the configured databases.

Transaction Handling (MULTI/EXEC):

getNodeByQuery key lookup behavior changed:
- No key lookups when queuing commands in MULTI, only cross-slot
  validation.
- Key lookups happen at EXEC time.
- SELECT inside MULTI/EXEC is now checked, ensuring key validation
  uses the selected DB at lookup.

Valkey-cli:

valkey-cli has been updated to support resharding across all databases.

Configuration:

Introduce new configuration cluster-databases.
The new configuration controls the maximal number of databases in cluster mode.

Implements #1319

codecov · 2025-02-05T15:45:24Z

Codecov Report

Attention: Patch coverage is 91.00000% with 9 lines in your changes missing coverage. Please review.

Project coverage is 70.84%. Comparing base (2d200df) to head (f2ed97c).
Report is 7 commits behind head on unstable.

Files with missing lines	Patch %	Lines
src/valkey-cli.c	80.00%	8 Missing ⚠️
src/cluster.c	96.87%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##           unstable    #1671      +/-   ##
============================================
- Coverage     71.02%   70.84%   -0.18%     
============================================
  Files           123      123              
  Lines         66116    66173      +57     
============================================
- Hits          46956    46879      -77     
- Misses        19160    19294     +134

Files with missing lines	Coverage Δ
src/cluster_legacy.c	`86.78% <100.00%> (+0.37%)`	⬆️
src/config.c	`78.39% <ø> (-0.05%)`	⬇️
src/db.c	`89.99% <100.00%> (+0.42%)`	⬆️
src/server.c	`87.94% <100.00%> (+0.03%)`	⬆️
src/server.h	`100.00% <ø> (ø)`
src/valkey-benchmark.c	`62.42% <100.00%> (+0.24%)`	⬆️
src/cluster.c	`90.24% <96.87%> (+0.21%)`	⬆️
src/valkey-cli.c	`54.60% <80.00%> (-1.32%)`	⬇️

... and 13 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

soloestoy · 2025-02-13T09:07:19Z

I'm happy that we did "Unified db rehash method for both standalone and cluster #12848" when developing kvstore , which made the implementation of multi-database simpler.

hpatro

We need to add history to SWAPDB, SELECT, MOVE json files to indicate it's supported since 9.0.

xbasel · 2025-03-03T09:57:25Z

documentation: valkey-io/valkey-doc#242

hwware · 2025-03-05T14:59:08Z

It looks like there are still some test cases failed related to multiply db feature. Please fix them first, Thanks

Re-adds a statement to restore the `singledb` config that was accidentally removed in PR valkey-io#1671. Signed-off-by: xbasel <103044017+xbasel@users.noreply.github.com>

Re-adds a statement to restore the `singledb` config that was accidentally removed in PR #1671. Fixes #2049 Signed-off-by: xbasel <103044017+xbasel@users.noreply.github.com>

One of the new tests that was added uses `CONFIG GET PORT`, which isn't right one for TLS. Also removed some other use of the helper which aren't actually used. Introduced as part of #1671. --------- Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>

## cluster: add multi-database support in cluster mode Add multi-database support in cluster mode to align with standalone mode and facilitate migration. Previously, cluster mode was restricted to a single database (DB0). This change allows multiple databases while preserving the existing slot-based key distribution. ### Key Features: - Database-Agnostic Hashing. The hashing algorithm is unchanged. Identical keys always map to the same slot across all databases, ensuring consistent key distribution and compatibility with existing single-database setups. - Multi-DB commands support. SELECT, MOVE, and COPY are now supported in cluster mode. - Fully backward compatible with no API changes. - SWAPDB is not supported in cluster mode. It is unsafe due to inconsistency risks. ### Command-Level Changes: - SELECT / MOVE / COPY are now supported in cluster mode. - MOVE / COPY (with db) are rejected (TRYAGAIN error) during slot migration to prevent multi-DB inconsistencies. - SWAPDB will return an error if used when cluster mode is enabled. - GETKEYSINSLOT, COUNTKEYSINSLOT and MIGRATE will operate in the context of the selected database. This means, for example, that migrating keys in a slot will require iterating and repeating across all databases. ### Slot Migration Process: - Multi-DB support in cluster mode affects slot migration. Operators should now iterate over all the configured databases. ### Transaction Handling (MULTI/EXEC): - getNodeByQuery key lookup behavior changed: - No key lookups when queuing commands in MULTI, only cross-slot validation. - Key lookups happen at EXEC time. - SELECT inside MULTI/EXEC is now checked, ensuring key validation uses the selected DB at lookup. ### Valkey-cli: - valkey-cli has been updated to support resharding across all databases. ### Configuration: - Introduce new configuration `cluster-databases`. The new configuration controls the maximal number of databases in cluster mode. Implements valkey-io#1319 --------- Signed-off-by: xbasel <103044017+xbasel@users.noreply.github.com> Signed-off-by: zhaozhao.zz <zhaozhao.zz@alibaba-inc.com> Co-authored-by: zhaozhao.zz <zhaozhao.zz@alibaba-inc.com> Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech> Co-authored-by: Madelyn Olson <madelyneolson@gmail.com> Co-authored-by: Ran Shidlansik <ranshid@amazon.com>

Re-adds a statement to restore the `singledb` config that was accidentally removed in PR valkey-io#1671. Fixes valkey-io#2049 Signed-off-by: xbasel <103044017+xbasel@users.noreply.github.com>

One of the new tests that was added uses `CONFIG GET PORT`, which isn't right one for TLS. Also removed some other use of the helper which aren't actually used. Introduced as part of valkey-io#1671. --------- Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>

To support multiple databases in cluster mode (see valkey-io#1671), `getNodeByQuery` temporarily switches databases when tracking `SELECT` statements during slot migration/import. The intended logic is to revert any database change after the operation. However, this approach is flawed: in some transactions the database change is not properly reverted, causing the client to remain on the wrong database. For example, if a transaction includes `SELECT` statements, the current database may be changed even if the transaction is never executed (see added test). Fix the issue by saving the original database once and restoring to it after a switch. Signed-off-by: Simon Baatz <gmbnomis@gmail.com>

To support multiple databases in cluster mode (see #1671), `getNodeByQuery` temporarily switches databases when tracking `SELECT` statements during slot migration/import. The intended logic is to revert any database change after the operation. However, this approach is flawed: in some transactions the database change is not properly reverted, causing the client to remain on the wrong database. For example, if a transaction includes `SELECT` statements, the current database may be changed even if the transaction is never executed (see added test). Fix the issue by saving the original database once and restoring to it after a switch. --------- Signed-off-by: Simon Baatz <gmbnomis@gmail.com> Signed-off-by: Simon Baatz <gmbnomis@users.noreply.github.com> Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>

…#2206) To support multiple databases in cluster mode (see valkey-io#1671), `getNodeByQuery` temporarily switches databases when tracking `SELECT` statements during slot migration/import. The intended logic is to revert any database change after the operation. However, this approach is flawed: in some transactions the database change is not properly reverted, causing the client to remain on the wrong database. For example, if a transaction includes `SELECT` statements, the current database may be changed even if the transaction is never executed (see added test). Fix the issue by saving the original database once and restoring to it after a switch. --------- Signed-off-by: Simon Baatz <gmbnomis@gmail.com> Signed-off-by: Simon Baatz <gmbnomis@users.noreply.github.com> Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>

The comment has been outdated since valkey-io#1671, update it. Signed-off-by: Binbin <binloveplay1314@qq.com>

…wed (#2391) The comment has been outdated since #1671, update it. Signed-off-by: Binbin <binloveplay1314@qq.com>

…wed (valkey-io#2391) The comment has been outdated since valkey-io#1671, update it. Signed-off-by: Binbin <binloveplay1314@qq.com>

The test was accidentally removed in PR valkey-io#1671. Signed-off-by: Binbin <binloveplay1314@qq.com>

The test was accidentally removed in PR #1671. Signed-off-by: Binbin <binloveplay1314@qq.com>

xbasel mentioned this pull request Feb 5, 2025

[NEW] Multiple DB supports in cluster mode #1319

Closed

xbasel marked this pull request as draft February 6, 2025 10:01

xbasel mentioned this pull request Feb 6, 2025

[NEW] Multi-database support in cluster mode - Implementation Plan #1681

Closed

xbasel marked this pull request as ready for review February 10, 2025 21:37

xbasel requested a review from zuiderkwast February 10, 2025 22:13

dvkashapov reviewed Feb 11, 2025

View reviewed changes

Comment thread src/db.c

soloestoy requested review from soloestoy and removed request for zuiderkwast February 12, 2025 06:28

soloestoy reviewed Feb 13, 2025

View reviewed changes

Comment thread src/cluster.c Outdated

ranshid added the release-notes This issue should get a line item in the release notes label Feb 17, 2025

hpatro reviewed Feb 17, 2025

View reviewed changes

Comment thread src/cluster_legacy.c Outdated

Comment thread tests/support/cluster.tcl Outdated

madolson reviewed Feb 17, 2025

View reviewed changes

ranshid added the client-changes-needed Client changes may be required for this feature label Feb 24, 2025

hwware reviewed Feb 26, 2025

View reviewed changes

Comment thread src/db.c Outdated

hwware reviewed Feb 26, 2025

View reviewed changes

Comment thread src/cluster.c

xbasel added the documentation label Feb 26, 2025

xbasel force-pushed the multidb branch 2 times, most recently from 3dc03d6 to af2d46a Compare February 27, 2025 23:03

xbasel mentioned this pull request Mar 3, 2025

Multi-database support in cluster mode valkey-io/valkey-doc#242

Merged

xbasel requested a review from a team March 5, 2025 11:38

xbasel force-pushed the multidb branch from 0e9bb7a to f627245 Compare March 5, 2025 13:41

xbasel marked this pull request as draft March 5, 2025 18:36

xbasel force-pushed the multidb branch 4 times, most recently from 538e23e to 63151ae Compare March 6, 2025 12:14

This was referenced May 5, 2025

Add CLUSTER FLUSHSLOT command #1384

Merged

Test failure: Exception LOADING in valkey_deferring_client #2049

Closed

xbasel mentioned this pull request May 6, 2025

Restore omitted singledb config in tests #2052

Merged

zuiderkwast mentioned this pull request May 6, 2025

Use the correct port for migration #2050

Merged

gmbnomis mentioned this pull request Jun 9, 2025

Multi-database support in cluster mode weakens script key declaration guarantees #2190

Open

gmbnomis mentioned this pull request Jun 11, 2025

Prevent getNodeByQuery from leaking DB changes into client #2206

Merged

zvi-code mentioned this pull request Jul 3, 2025

[BUG] Support multi-db in CME valkey-io/valkey-search#204

Closed

enjoy-binbin added a commit to enjoy-binbin/valkey that referenced this pull request Jul 29, 2025

Update swapdb command comment to mention why cluster mode is not allowed

b62901e

The comment has been outdated since valkey-io#1671, update it. Signed-off-by: Binbin <binloveplay1314@qq.com>

enjoy-binbin mentioned this pull request Jul 29, 2025

Update swapdb command comment to mention why cluster mode is not allowed #2391

Merged

enjoy-binbin added a commit that referenced this pull request Aug 5, 2025

Update swapdb command comment to mention why cluster mode is not allo…

a50690f

…wed (#2391) The comment has been outdated since #1671, update it. Signed-off-by: Binbin <binloveplay1314@qq.com>

affonsov mentioned this pull request Sep 4, 2025

Java: Multi-Database Support for Cluster Mode Valkey 9.0 valkey-io/valkey-glide#4658

Merged

6 tasks

zvi-code mentioned this pull request Sep 10, 2025

Capture 1.1.0 work items needed for Valkey 9.0 valkey-io/valkey-search#364

Closed

enjoy-binbin added a commit to enjoy-binbin/valkey that referenced this pull request Oct 11, 2025

Add Slot migration is ok when the replicas are down test back

fce0859

The test was accidentally removed in PR valkey-io#1671. Signed-off-by: Binbin <binloveplay1314@qq.com>

enjoy-binbin mentioned this pull request Oct 11, 2025

Add Slot migration is ok when the replicas are down test back #2727

Merged

enjoy-binbin added a commit that referenced this pull request Oct 14, 2025

Add Slot migration is ok when the replicas are down test back (#2727)

ccf6fca

The test was accidentally removed in PR #1671. Signed-off-by: Binbin <binloveplay1314@qq.com>

currantw mentioned this pull request Oct 24, 2025

test: add --num-databases option to cluster_manager.py for multi-database cluster testing valkey-io/valkey-glide#4910

Closed

2 tasks

supercaracal mentioned this pull request Dec 22, 2025

New features on server side redis-rb/redis-cluster-client#457

Closed

supercaracal mentioned this pull request Jan 10, 2026

feat: add multiple db support redis-rb/redis-cluster-client#464

Merged

clubanderson mentioned this pull request Mar 16, 2026

🌱 valkey: [NEW] Support database level ACL kubestellar/console-kb#1294

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add multi-database support to cluster mode#1671

Add multi-database support to cluster mode#1671
madolson merged 61 commits into
valkey-io:unstablefrom
xbasel:multidb

xbasel commented Feb 5, 2025 •

edited

Loading

Uh oh!

codecov Bot commented Feb 5, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

soloestoy commented Feb 13, 2025

Uh oh!

hpatro left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xbasel commented Mar 3, 2025

Uh oh!

hwware commented Mar 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

13 participants

Uh oh!

Conversation

xbasel commented Feb 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

cluster: add multi-database support in cluster mode

Key Features:

Command-Level Changes:

Slot Migration Process:

Transaction Handling (MULTI/EXEC):

Valkey-cli:

Configuration:

Uh oh!

codecov Bot commented Feb 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

soloestoy commented Feb 13, 2025

Uh oh!

hpatro left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xbasel commented Mar 3, 2025

Uh oh!

hwware commented Mar 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

13 participants

xbasel commented Feb 5, 2025 •

edited

Loading

codecov Bot commented Feb 5, 2025 •

edited

Loading