Skip to content

MOD-10714 Integrate Rust RSSortingVector#6887

Merged
JonasKruckenberg merged 6 commits intomasterfrom
jonas-swap-rssorting-vector
Jan 19, 2026
Merged

MOD-10714 Integrate Rust RSSortingVector#6887
JonasKruckenberg merged 6 commits intomasterfrom
jonas-swap-rssorting-vector

Conversation

@JonasKruckenberg
Copy link
Copy Markdown
Collaborator

@JonasKruckenberg JonasKruckenberg commented Sep 25, 2025

Recreate #6855 bc of branch name issue

Mark if applicable

  • This PR introduces API changes
  • This PR introduces serialization changes

Note

Migrates sortable field storage to a Rust implementation with C FFI and updates callers accordingly.

  • Replace C sortable.c/h vector logic with Rust sorting_vector crate exposed through sorting_vector_ffi; add sorting_vector_rs.h
  • Update C callsites (document.c, doc_table.c, debug_commands.c) to use RSSortingVector_* API (New/Free/PutNum/PutStr/PutStrNormalize/Get/Length/GetMemorySize) and remove legacy NewSortingVector/SortingVector_Free
  • Change string handling: sorting vector now accepts raw bytes; normalization is done in Rust (PutStrNormalize) instead of pre-normalizing in C
  • Extend Rust value to work with byte strings (create_string(Vec<u8>), as_str_bytes) and add RSValue_NewCopiedString usage in FFI
  • Add new crates/modules and exports (sorting_vector, sorting_vector_ffi) and wire through redisearch_rs entrypoint; adjust dependencies (drop thiserror from sorting_vector, add redis-module to value for MUSL)
  • Update and expand tests to new APIs and byte-string expectations across rlookup and sorting_vector
  • CI: minor YAML quoting cleanups and temporarily disable select benchmark jobs

Written by Cursor Bugbot for commit 215ca60. This will update automatically on new commits. Configure here.

@JonasKruckenberg JonasKruckenberg marked this pull request as ready for review September 25, 2025 14:24
@jit-ci
Copy link
Copy Markdown

jit-ci bot commented Sep 25, 2025

Hi, I’m Jit, a friendly security platform designed to help developers build secure applications from day zero with an MVS (Minimal viable security) mindset.

In case there are security findings, they will be communicated to you as a comment inside the PR.

Hope you’ll enjoy using Jit.

Questions? Comments? Want to learn more? Get in touch with us.

cursor[bot]

This comment was marked as outdated.

@fcostaoliveira
Copy link
Copy Markdown
Contributor

fcostaoliveira commented Sep 25, 2025

Automated performance analysis summary

This comment was automatically generated given there is performance data available.

In summary:

  • Detected a total of 31 stable tests between versions.
  • Detected a total of 6 highly unstable benchmarks (6 baseline).
  • Detected a total of 6 regressions bellow the regression water line 8.0%.

You can check a comparison in detail via the grafana link

Performance Regressions and Issues - Comparison between master and jonas-swap-rssorting-vector.

Time Period from 30 days ago. (environment used: oss-standalone)

Test Case Baseline master (median obs. +- std.dev) Comparison jonas-swap-rssorting-vector (median obs. +- std.dev) % change (higher-better) Note
ftsb-1M-enwiki_abstract-hashes-fulltext-2word-intersection-query-non-sortable 34 +- 4.0% (2 datapoints) 24 -27.7% REGRESSION
ftsb-10K-enwiki_abstract-hashes-fulltext-sortby 90 +- 5.3% (7 datapoints) 75 -16.0% REGRESSION
search-numeric-sortby-desc-optimize 31 +- 9.7% (2 datapoints) 27 -11.8% waterline=9.7%. REGRESSION
search-ftsb-10K-enwiki_abstract-hashes-fulltext-aggregate-sortby-limit-0-100 935 +- 0.9% (2 datapoints) 826 -11.7% REGRESSION
search-numeric 2482 +- 27.6% UNSTABLE (7 datapoints) 2268 -8.6% UNSTABLE (baseline high variance); server: FT.SEARCH p50 increased 9.9% (baseline CV=21.1%); client: Latency increased 9.5% (baseline CV=20.5%); neither server nor client side confirms regression
search-ftsb-1M-enwiki_abstract-hashes-fulltext-simple-1word-query-non-sortable 168 +- 4.9% (7 datapoints) 154 -8.4% REGRESSION
search-ftsb-10K-enwiki_abstract-hashes-fulltext-search-sortby-limit-0-100 902 +- 1.3% (7 datapoints) 828 -8.2% REGRESSION
search-numeric-sortby 3604 +- 23.0% UNSTABLE (7 datapoints) 3642 1.1% UNSTABLE (baseline high variance); server: p50 latency stable; client: client latency stable; neither server nor client side confirms regression
search-filtering-tag-numeric 287 +- 20.2% UNSTABLE (2 datapoints) 306 6.5% UNSTABLE (baseline high variance); server: FT.AGGREGATE p50 decreased 7.9% (baseline CV=20.3%); client: Latency decreased 18.6% (baseline CV=12.7%); neither server nor client side confirms regression
ftsb-1M-enwiki_abstract-hashes-fulltext-2word-union-query-non-sortable 1108 +- 14.6% UNSTABLE (2 datapoints) 1276 15.1% UNSTABLE (baseline high variance); server: p50 latency stable; client: OverallQuantiles.allCommands.q50 decreased 12.5% (baseline CV=15.9%); neither server nor client side confirms regression
ftsb-1M-enwiki_abstract-hashes-fulltext-simple-1word-query 1036 +- 19.4% UNSTABLE (2 datapoints) 1220 17.8% UNSTABLE (baseline high variance); server: FT.SEARCH p50 decreased 19.7% (baseline CV=17.8%); client: OverallQuantiles.allCommands.q50 decreased 17.4% (baseline CV=20.0%); neither server nor client side confirms regression
search-numeric-sortby-desc 2890 +- 32.3% UNSTABLE (2 datapoints) 3599 24.6% UNSTABLE (baseline high variance); server: FT.SEARCH p50 decreased 37.3% (baseline CV=26.2%); client: Latency decreased 31.6% (baseline CV=28.5%); neither server nor client side confirms regression
Tests with No Significant Changes (31 tests)

Tests with No Significant Changes

Test Case Baseline master (median obs. +- std.dev) Comparison jonas-swap-rssorting-vector (median obs. +- std.dev) % change (higher-better) Note
ftsb-10K-enwiki_abstract-hashes-term-prefix 5898 +- 1.4% (2 datapoints) 5689.00 -3.5% potential REGRESSION
ftsb-10K-enwiki_abstract-hashes-term-suffix 2186 +- 0.4% (2 datapoints) 2242.00 2.6% No Change
ftsb-10K-enwiki_abstract-hashes-term-suffix-withsuffixtrie 16796 +- 0.8% (2 datapoints) 16726.00 -0.4% No Change
ftsb-10K-enwiki_abstract-hashes-term-wildcard 8835 +- 3.2% (7 datapoints) 8417.00 -4.7% potential REGRESSION
ftsb-10K-enwiki_pages-hashes-fulltext-mixed_simple-1word-query_write_1_to_read_20.yml 1014 +- 3.9% (7 datapoints) 986.00 -2.8% No Change
ftsb-10K-enwiki_pages-hashes-load 65068 +- 5.7% (2 datapoints) 65827.00 1.2% No Change
ftsb-10K-multivalue-numeric-json 997 +- 1.5% (7 datapoints) 989.00 -0.8% No Change
ftsb-10K-singlevalue-numeric-json 482 +- 0.6% (7 datapoints) 482.00 0.1% No Change
ftsb-1K-enwiki_abstract-hashes-term-contains 1960 +- 1.8% (7 datapoints) 1980.00 1.0% No Change
ftsb-1M-enwiki_abstract-hashes-fulltext-2word-intersection-query 327 +- 2.4% (2 datapoints) 335.00 2.4% No Change
ftsb-1M-enwiki_abstract-hashes-fulltext-2word-union-query 2662 +- 1.3% (2 datapoints) 2771.00 4.1% potential IMPROVEMENT
ftsb-1M-enwiki_abstract-hashes-load 23356 +- 7.5% (7 datapoints) 22950.00 -1.7% No Change
ftsb-1M-nyc_taxis-ftadd-load 29192 +- 3.2% (7 datapoints) 29727.00 1.8% No Change
ftsb-1M-nyc_taxis-hashes-load 30596 +- 3.5% (7 datapoints) 29840.00 -2.5% No Change
search-aggregate-post-filter-simple.yml 17311 +- 0.1% (2 datapoints) 17509.00 1.1% No Change
search-filtering-tag-numeric-filter-pipeline 11009 +- 1.3% (7 datapoints) 11105.00 0.9% No Change
search-ftsb-10K-enwiki_abstract-hashes-term-withoutsuffix-trie 14376 +- 1.1% (7 datapoints) 14729.00 2.5% No Change
search-ftsb-10K-enwiki_abstract-hashes-term-withsuffix-trie 14121 +- 1.8% (7 datapoints) 14246.00 0.9% No Change
search-ftsb-1700K-docs-union-iterators-q3 8.1 +- 1.2% (7 datapoints) 7.80 -3.3% potential REGRESSION
search-ftsb-1M-enwiki_abstract-hashes-fulltext-simple-1word-query-one-indexed-field 7259 +- 3.0% (7 datapoints) 6921.00 -4.6% potential REGRESSION
search-ftsb-370K-docs-union-iterators-q4 8.3 +- 1.3% (7 datapoints) 8.20 -1.1% No Change
search-ftsb-5200K-docs-union-iterators-q1 0.84 +- 2.0% (7 datapoints) 0.82 -2.4% No Change
search-ftsb-5500K-docs-union-iterators-q2 1.2 +- 2.3% (7 datapoints) 1.20 -3.3% potential REGRESSION
search-geo 218 +- 2.3% (7 datapoints) 227.00 4.2% potential IMPROVEMENT
search-high-cardinality-negation-term-baseline 39 +- 1.5% (2 datapoints) 39.00 0.2% No Change
search-high-cardinality-negation-term-comparison_union_all_other_terms 14 +- 1.8% (2 datapoints) 14.00 2.2% No Change
search-numeric-optimize 8163 +- 2.6% (2 datapoints) 8118.00 -0.5% No Change
search-numeric-sortby-optimize 26 +- 5.9% (7 datapoints) 28.00 7.1% potential IMPROVEMENT
vecsim-arxiv-titles-384-angular-filters-m16-ef-128-fulltext-filter 575 +- 4.3% (7 datapoints) 574.00 -0.2% No Change
vecsim-arxiv-titles-384-angular-filters-m16-ef-128-numeric-filter 155 +- 9.6% (7 datapoints) 156.00 0.6% waterline=9.6%. No Change
vecsim-arxiv-titles-384-angular-filters-m16-ef-128-tag-filter 15690 +- 1.0% (7 datapoints) 16281.00 3.8% potential IMPROVEMENT

cursor[bot]

This comment was marked as outdated.

@codecov
Copy link
Copy Markdown

codecov bot commented Sep 25, 2025

Codecov Report

❌ Patch coverage is 61.83575% with 79 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.85%. Comparing base (fcf5080) to head (215ca60).
⚠️ Report is 4 commits behind head on master.

Files with missing lines Patch % Lines
...arch_rs/c_entrypoint/sorting_vector_ffi/src/lib.rs 0.00% 49 Missing ⚠️
src/redisearch_rs/rlookup/src/load_document/mod.rs 92.38% 5 Missing and 3 partials ⚠️
src/redisearch_rs/value/src/rs_value_ffi.rs 0.00% 7 Missing ⚠️
src/sortable.c 0.00% 6 Missing ⚠️
src/redisearch_rs/sorting_vector/src/lib.rs 72.22% 1 Missing and 4 partials ⚠️
src/document.c 73.33% 4 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6887      +/-   ##
==========================================
+ Coverage   83.83%   83.85%   +0.01%     
==========================================
  Files         365      364       -1     
  Lines       55165    55106      -59     
  Branches    14294    14267      -27     
==========================================
- Hits        46249    46209      -40     
+ Misses       8758     8739      -19     
  Partials      158      158              
Flag Coverage Δ
flow 84.75% <60.00%> (+0.06%) ⬆️
unit 50.71% <55.55%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@JonasKruckenberg JonasKruckenberg force-pushed the jonas-swap-rssorting-vector branch 2 times, most recently from f3f067f to 727466a Compare September 29, 2025 10:23
cursor[bot]

This comment was marked as outdated.

@JonasKruckenberg JonasKruckenberg force-pushed the jonas-swap-rssorting-vector branch 3 times, most recently from 8429b4c to 2cc0fb3 Compare September 29, 2025 11:30
@JonasKruckenberg JonasKruckenberg force-pushed the jonas-swap-rssorting-vector branch 2 times, most recently from 3dcd2ab to 955eb09 Compare November 6, 2025 14:22
@JonasKruckenberg JonasKruckenberg force-pushed the jonas-swap-rssorting-vector branch 5 times, most recently from ee5a6ce to 004d154 Compare November 13, 2025 08:54
github-merge-queue bot pushed a commit that referenced this pull request Dec 18, 2025
* RSSortingVector FFI cleanup

* fix: remove UTF-8 assumption from RSValueFFI

* MOD-10714 Integrate Rust RSSortingVector

* fix workspace hack crate

* fix lint
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Dec 18, 2025
@JonasKruckenberg JonasKruckenberg force-pushed the jonas-swap-rssorting-vector branch 5 times, most recently from 1a062d6 to 58c236a Compare January 14, 2026 11:55
nafraf
nafraf previously approved these changes Jan 14, 2026
@JonasKruckenberg JonasKruckenberg force-pushed the jonas-swap-rssorting-vector branch from 58c236a to 6e54e8a Compare January 15, 2026 14:14
@JonasKruckenberg JonasKruckenberg force-pushed the jonas-swap-rssorting-vector branch from 5d479ff to 215ca60 Compare January 19, 2026 15:38
@JonasKruckenberg JonasKruckenberg added this pull request to the merge queue Jan 19, 2026
Merged via the queue into master with commit e9d1c85 Jan 19, 2026
70 of 71 checks passed
@JonasKruckenberg JonasKruckenberg deleted the jonas-swap-rssorting-vector branch January 19, 2026 19:13
eyalrund pushed a commit that referenced this pull request Jan 22, 2026
* skip `benchmark-search-oss-cluster-04-primaries-threads-6`

This benchmark seems to be affected by the same error as #8005.

* RSSortingVector FFI cleanup

* fix: remove UTF-8 assumption from RSValueFFI

* MOD-10714 Integrate Rust RSSortingVector

* fix workspace hack crate

* fix lint
LukeMathWalker pushed a commit that referenced this pull request Jan 26, 2026
* skip `benchmark-search-oss-cluster-04-primaries-threads-6`

This benchmark seems to be affected by the same error as #8005.

* RSSortingVector FFI cleanup

* fix: remove UTF-8 assumption from RSValueFFI

* MOD-10714 Integrate Rust RSSortingVector

* fix workspace hack crate

* fix lint
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants