Skip to content

CLUSTERSCAN range bounded scanning across contiguous slots#3391

Merged
zuiderkwast merged 6 commits into
valkey-io:unstablefrom
nmvk:slot-range
May 6, 2026
Merged

CLUSTERSCAN range bounded scanning across contiguous slots#3391
zuiderkwast merged 6 commits into
valkey-io:unstablefrom
nmvk:slot-range

Conversation

@nmvk

@nmvk nmvk commented Mar 23, 2026

Copy link
Copy Markdown
Contributor

Instead of scanning one slot at a time CLUSTERSCAN now scans the entire contiguous range of slots owned by the current node.

Implementation details

  • Re-sharding safe as the hash slot is updated based on the local cursor position.
  • Fingerprint remains stable across the entire contiguous slot range instead of being reset per slot.
  • Parsing/validation of parameters for the SCAN commands is refactored and moved to a separate function.
   > CLUSTERSCAN 0
   "0-{06S}-0"                      # start at slot 0

   > CLUSTERSCAN 0-{06S}-0
   "aBcDeF-{06S}-48"                # scanning slot 0...

   > CLUSTERSCAN aBcDeF-{06S}-48
   "aBcDeF-{1Y7}-16"                 # slot 0 done, continues to slot 6 (same node hence FP is unchanged)

   > CLUSTERSCAN aBcDeF-{1Y7}-16
   "aBcDeF-{0or}-32"                # slot 6 done, continues to slot 100  (same node hence FP is unchanged)
   ...
   > CLUSTERSCAN aBcDeF-{...}-64
   "0-{8YG}-0"                      # Current continuous slot boundary reached hence cross-node transition 

Follow-up of #2934

@codecov

codecov Bot commented Mar 23, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.84211% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 76.64%. Comparing base (fea0b40) to head (25eab71).
⚠️ Report is 4 commits behind head on unstable.

Files with missing lines Patch % Lines
src/db.c 95.23% 3 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##           unstable    #3391      +/-   ##
============================================
- Coverage     76.75%   76.64%   -0.12%     
============================================
  Files           162      162              
  Lines         80614    80637      +23     
============================================
- Hits          61879    61801      -78     
- Misses        18735    18836     +101     
Files with missing lines Coverage Δ
src/cluster.c 92.30% <100.00%> (+0.08%) ⬆️
src/expire.c 98.12% <100.00%> (ø)
src/kvstore.c 96.65% <100.00%> (+0.01%) ⬆️
src/server.h 100.00% <ø> (ø)
src/t_hash.c 95.47% <100.00%> (ø)
src/t_set.c 98.05% <100.00%> (+0.22%) ⬆️
src/t_zset.c 96.98% <100.00%> (-0.14%) ⬇️
src/db.c 94.71% <95.23%> (+0.33%) ⬆️

... and 18 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@nmvk nmvk force-pushed the slot-range branch 5 times, most recently from c85e254 to b282231 Compare March 28, 2026 21:05
@nmvk nmvk force-pushed the slot-range branch 4 times, most recently from 1133666 to d25fcba Compare April 6, 2026 14:55
Comment thread src/kvstore.c Outdated
@zuiderkwast zuiderkwast moved this to Needs Review in Valkey 9.1 Apr 7, 2026
@nmvk nmvk force-pushed the slot-range branch 2 times, most recently from a58bb52 to fadbc7f Compare April 8, 2026 08:59
Comment thread src/db.c Outdated
Comment thread src/db.c Outdated
@nmvk nmvk force-pushed the slot-range branch 3 times, most recently from 1697f24 to 405af6e Compare April 30, 2026 09:36
@nmvk

nmvk commented Apr 30, 2026

Copy link
Copy Markdown
Contributor Author

Thank you @zuiderkwast for the suggestions and review. I had not considered grouping the CLUSTERSCAN specific args separately.

I updated the code so callers keep slot and final_slot accurate. The CLUSTERSCAN args are now clusterScanCtx, so SCAN/HSCAN/SSCAN/ZSCAN no longer need to pass dummy CLUSTERSCAN arguments. This solves the -1 and awkward boolean issue for slot advancement.

PTAL.

@zuiderkwast zuiderkwast left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, it's good. We can merge it after a small clean up.

Comment thread src/cluster.c Outdated
Comment thread src/server.h Outdated
nmvk added 2 commits May 5, 2026 19:00
Instead of scanning one slot at a time `CLUSTERSCAN` now scans the entire contiguous range of slots owned by the current node.

Implementation details

* Added `final_slot` parameter to `scanGenericCommand` replacing `cursor_prefix`/`finished_cursor_prefix`
* Re-sharding safe as the hash slot is updated based on the local cursor position.

Signed-off-by: nmvk <r@nmvk.com>
Refactor and rename

Signed-off-by: nmvk <r@nmvk.com>
nmvk added 3 commits May 5, 2026 19:00
Signed-off-by: nmvk <r@nmvk.com>
Move it to data types.

Signed-off-by: nmvk <r@nmvk.com>
Similar to `parseExtendedCommandArgumentsOrReply` to
ensure the common arguments are parsed together.

Signed-off-by: nmvk <r@nmvk.com>
Comment thread src/db.c Outdated
Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
@zuiderkwast zuiderkwast merged commit 6c9d7fc into valkey-io:unstable May 6, 2026
59 of 60 checks passed
@github-project-automation github-project-automation Bot moved this from Needs Review to To be backported in Valkey 9.1 May 6, 2026
@nmvk

nmvk commented May 6, 2026

Copy link
Copy Markdown
Contributor Author

Thank you @zuiderkwast 🎉

lucasyonge pushed a commit that referenced this pull request May 11, 2026
Instead of scanning one slot at a time `CLUSTERSCAN` now scans the
entire contiguous range of slots owned by the current node.

Implementation details

* Re-sharding safe as the hash slot is updated based on the local cursor
  position.
* Fingerprint remains stable across the entire contiguous slot range
  instead of being reset per slot.
* Parsing/validation of parameters for the SCAN commands is refactored
  and moved to a separate function.

```
   > CLUSTERSCAN 0
   "0-{06S}-0"                      # start at slot 0

   > CLUSTERSCAN 0-{06S}-0
   "aBcDeF-{06S}-48"                # scanning slot 0...

   > CLUSTERSCAN aBcDeF-{06S}-48
   "aBcDeF-{1Y7}-16"                 # slot 0 done, continues to slot 6 (same node hence FP is unchanged)

   > CLUSTERSCAN aBcDeF-{1Y7}-16
   "aBcDeF-{0or}-32"                # slot 6 done, continues to slot 100  (same node hence FP is unchanged)
   ...
   > CLUSTERSCAN aBcDeF-{...}-64
   "0-{8YG}-0"                      # Current continuous slot boundary reached hence cross-node transition 
```

Follow-up of #2934

---------

Signed-off-by: nmvk <r@nmvk.com>
Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
lucasyonge pushed a commit that referenced this pull request May 12, 2026
Instead of scanning one slot at a time `CLUSTERSCAN` now scans the
entire contiguous range of slots owned by the current node.

Implementation details

* Re-sharding safe as the hash slot is updated based on the local cursor
  position.
* Fingerprint remains stable across the entire contiguous slot range
  instead of being reset per slot.
* Parsing/validation of parameters for the SCAN commands is refactored
  and moved to a separate function.

```
   > CLUSTERSCAN 0
   "0-{06S}-0"                      # start at slot 0

   > CLUSTERSCAN 0-{06S}-0
   "aBcDeF-{06S}-48"                # scanning slot 0...

   > CLUSTERSCAN aBcDeF-{06S}-48
   "aBcDeF-{1Y7}-16"                 # slot 0 done, continues to slot 6 (same node hence FP is unchanged)

   > CLUSTERSCAN aBcDeF-{1Y7}-16
   "aBcDeF-{0or}-32"                # slot 6 done, continues to slot 100  (same node hence FP is unchanged)
   ...
   > CLUSTERSCAN aBcDeF-{...}-64
   "0-{8YG}-0"                      # Current continuous slot boundary reached hence cross-node transition 
```

Follow-up of #2934

---------

Signed-off-by: nmvk <r@nmvk.com>
Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
@sarthakaggarwal97 sarthakaggarwal97 moved this from To be backported to Done in Valkey 9.1 May 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants