Skip to content

MOD-13454 skip benchmark-search-oss-standalone-threads-6#8005

Merged
meiravgri merged 1 commit intomasterfrom
meiravg_skip_failing_benchmarks
Jan 8, 2026
Merged

MOD-13454 skip benchmark-search-oss-standalone-threads-6#8005
meiravgri merged 1 commit intomasterfrom
meiravg_skip_failing_benchmarks

Conversation

@meiravgri
Copy link
Copy Markdown
Collaborator

@meiravgri meiravgri commented Jan 8, 2026 •
edited
Loading

The benchmark-search-oss-standalone-threads-6 benchmark job fails in CI with the error:

Exception: Remote redis is not available. Aborting...

Skipping for now.

Note

Temporarily disables the benchmark-search-oss-standalone-threads-6 job in benchmark-runner.yml by hard-coding if: false, with a note that Redis becomes unavailable after ~3 hours of data loading.

  • Only the oss-standalone-threads-6 search benchmark is affected; all other jobs and settings remain unchanged

Written by Cursor Bugbot for commit 4a1372d. This will update automatically on new commits. Configure here.

@meiravgri meiravgri changed the title MOD-13454 MOD-13454 skip benchmark-search-oss-standalone-threads-6 Jan 8, 2026
@meiravgri meiravgri requested a review from alonre24 January 8, 2026 14:21
@meiravgri meiravgri enabled auto-merge January 8, 2026 14:42
@fcostaoliveira
Copy link
Copy Markdown
Contributor

fcostaoliveira commented Jan 8, 2026

Automated performance analysis summary

This comment was automatically generated given there is performance data available.

In summary:

  • Detected a total of 3 stable tests between versions.
  • Detected a total of 1 regressions bellow the regression water line 8.0%.

You can check a comparison in detail via the grafana link

Performance Regressions and Issues - Comparison between master and meiravg_skip_failing_benchmarks.

Time Period from 30 days ago. (environment used: oss-standalone)

Test Case Baseline master (median obs. +- std.dev) Comparison meiravg_skip_failing_benchmarks (median obs. +- std.dev) % change (higher-better) Note
search-numeric-sortby-optimize 30 +- 6.9% (7 datapoints) 26 -14.6% REGRESSION
Tests with No Significant Changes (3 tests)

Tests with No Significant Changes

Test Case Baseline master (median obs. +- std.dev) Comparison meiravg_skip_failing_benchmarks (median obs. +- std.dev) % change (higher-better) Note
ftsb-1K-enwiki_abstract-hashes-term-contains 1930 +- 0.8% (7 datapoints) 1911 -1.0% No Change
vecsim-arxiv-titles-384-angular-filters-m16-ef-128-fulltext-filter 580 +- 2.6% (7 datapoints) 553 -4.8% potential REGRESSION
vecsim-arxiv-titles-384-angular-filters-m16-ef-128-tag-filter 15832 +- 1.8% (7 datapoints) 15649 -1.2% No Change

@fcostaoliveira
Copy link
Copy Markdown
Contributor

fcostaoliveira commented Jan 8, 2026 •
edited
Loading

Automated performance analysis summary

This comment was automatically generated given there is performance data available.

In summary:

You can check a comparison in detail via the grafana link

Performance Regressions and Issues - Comparison between master and meiravg_skip_failing_benchmarks.

Time Period from 30 days ago. (environment used: oss-standalone)

Test Case Baseline master (median obs. +- std.dev) Comparison meiravg_skip_failing_benchmarks (median obs. +- std.dev) % change (higher-better) Note
ftsb-1M-enwiki_abstract-hashes-fulltext-2word-intersection-query-non-sortable 36 +- 25.9% UNSTABLE (7 datapoints) 22 -38.2% UNSTABLE (baseline high variance); server: FT.SEARCH p50 increased 13.0% (baseline CV=9.7%); client: OverallQuantiles.allCommands.q50 increased 20.9% (baseline CV=11.5%)
search-numeric-sortby 3514 +- 20.2% UNSTABLE (7 datapoints) 2342 -33.4% UNSTABLE (baseline high variance); server: FT.SEARCH p50 increased 51.1% (baseline CV=27.2%); client: Latency increased 42.8% (baseline CV=25.9%)
ftsb-1M-enwiki_abstract-hashes-fulltext-simple-1word-query 1078 +- 15.9% UNSTABLE (7 datapoints) 812 -24.7% UNSTABLE (baseline high variance); server: FT.SEARCH p50 increased 29.6% (baseline CV=18.3%); client: OverallQuantiles.allCommands.q50 increased 32.9% (baseline CV=19.4%)
ftsb-1M-enwiki_abstract-hashes-fulltext-2word-intersection-query 407 +- 7.6% (7 datapoints) 309 -24.1% REGRESSION
ftsb-1M-enwiki_abstract-hashes-fulltext-2word-union-query 2947 +- 6.6% (7 datapoints) 2549 -13.5% REGRESSION
search-numeric-sortby-desc-optimize 32 +- 9.1% (7 datapoints) 28 -12.5% waterline=9.1%. REGRESSION
search-numeric-sortby-desc 2247 +- 30.0% UNSTABLE (7 datapoints) 2476 10.2% UNSTABLE (baseline high variance); server: FT.SEARCH p50 decreased 10.3% (baseline CV=18.8%); client: Latency decreased 9.2% (baseline CV=18.8%); neither server nor client side confirms regression
search-numeric 2287 +- 33.1% UNSTABLE (7 datapoints) 3598 57.4% UNSTABLE (baseline high variance); server: FT.SEARCH p50 decreased 38.1% (baseline CV=26.3%); client: client latency stable; neither server nor client side confirms regression
Tests with No Significant Changes (35 tests)

Tests with No Significant Changes

Test Case Baseline master (median obs. +- std.dev) Comparison meiravg_skip_failing_benchmarks (median obs. +- std.dev) % change (higher-better) Note
ftsb-10K-enwiki_abstract-hashes-fulltext-sortby 93 +- 2.3% (7 datapoints) 94.00 0.2% No Change
ftsb-10K-enwiki_abstract-hashes-term-prefix 6168 +- 2.8% (7 datapoints) 5717.00 -7.3% potential REGRESSION
ftsb-10K-enwiki_abstract-hashes-term-suffix 2222 +- 0.6% (7 datapoints) 2169.00 -2.4% No Change
ftsb-10K-enwiki_abstract-hashes-term-suffix-withsuffixtrie 16440 +- 1.4% (7 datapoints) 16732.00 1.8% No Change
ftsb-10K-enwiki_abstract-hashes-term-wildcard 8592 +- 2.2% (7 datapoints) 8959.00 4.3% potential IMPROVEMENT
ftsb-10K-enwiki_pages-hashes-fulltext-mixed_simple-1word-query_write_1_to_read_20.yml 1007 +- 3.8% (7 datapoints) 997.00 -1.0% No Change
ftsb-10K-enwiki_pages-hashes-load 65554 +- 4.0% (7 datapoints) 65736.00 0.3% No Change
ftsb-10K-multivalue-numeric-json 987 +- 2.6% (7 datapoints) 967.00 -2.0% No Change
ftsb-10K-singlevalue-numeric-json 474 +- 1.0% (7 datapoints) 473.00 -0.2% No Change
ftsb-1K-enwiki_abstract-hashes-term-contains 1927 +- 1.3% (7 datapoints) 1878.00 -2.5% No Change
ftsb-1M-enwiki_abstract-hashes-fulltext-2word-union-query-non-sortable 1161 +- 9.0% (7 datapoints) 1115.00 -4.0% waterline=9.0%. potential REGRESSION
ftsb-1M-enwiki_abstract-hashes-load 24476 +- 3.1% (7 datapoints) 23600.00 -3.6% potential REGRESSION
ftsb-1M-nyc_taxis-ftadd-load 29993 +- 2.6% (7 datapoints) 31145.00 3.8% potential IMPROVEMENT
ftsb-1M-nyc_taxis-hashes-load 32095 +- 2.7% (7 datapoints) 31881.00 -0.7% No Change
search-aggregate-post-filter-simple.yml 17519 +- 1.1% (7 datapoints) 17702.00 1.0% No Change
search-filtering-tag-numeric 264 +- 9.9% (7 datapoints) 282.00 6.9% waterline=9.9%. potential IMPROVEMENT
search-filtering-tag-numeric-filter-pipeline 11310 +- 0.9% (7 datapoints) 11148.00 -1.4% No Change
search-ftsb-10K-enwiki_abstract-hashes-fulltext-aggregate-sortby-limit-0-100 946 +- 2.2% (7 datapoints) 947.00 0.1% No Change
search-ftsb-10K-enwiki_abstract-hashes-fulltext-search-sortby-limit-0-100 949 +- 3.2% (7 datapoints) 924.00 -2.6% No Change
search-ftsb-10K-enwiki_abstract-hashes-term-withoutsuffix-trie 14363 +- 1.2% (7 datapoints) 14304.00 -0.4% No Change
search-ftsb-10K-enwiki_abstract-hashes-term-withsuffix-trie 14476 +- 1.6% (7 datapoints) 14040.00 -3.0% potential REGRESSION
search-ftsb-1700K-docs-union-iterators-q3 8.1 +- 0.9% (7 datapoints) 8.40 3.0% No Change
search-ftsb-1M-enwiki_abstract-hashes-fulltext-simple-1word-query-non-sortable 160 +- 4.8% (7 datapoints) 164.00 2.6% No Change
search-ftsb-1M-enwiki_abstract-hashes-fulltext-simple-1word-query-one-indexed-field 7256 +- 2.8% (7 datapoints) 7042.00 -3.0% No Change
search-ftsb-370K-docs-union-iterators-q4 8.4 +- 1.1% (7 datapoints) 8.40 -0.5% No Change
search-ftsb-5200K-docs-union-iterators-q1 0.87 +- 1.6% (7 datapoints) 0.85 -2.3% No Change
search-ftsb-5500K-docs-union-iterators-q2 1.2 +- 1.4% (7 datapoints) 1.20 0.0%
search-geo 218 +- 2.3% (7 datapoints) 220.00 0.8% No Change
search-high-cardinality-negation-term-baseline 38 +- 0.5% (7 datapoints) 38.00 -0.1% No Change
search-high-cardinality-negation-term-comparison_union_all_other_terms 14 +- 1.9% (7 datapoints) 14.00 1.8% No Change
search-numeric-optimize 8255 +- 1.4% (7 datapoints) 7836.00 -5.1% potential REGRESSION
search-numeric-sortby-optimize 30 +- 5.4% (7 datapoints) 31.00 3.6% potential IMPROVEMENT
vecsim-arxiv-titles-384-angular-filters-m16-ef-128-fulltext-filter 588 +- 1.5% (7 datapoints) 609.00 3.6% potential IMPROVEMENT
vecsim-arxiv-titles-384-angular-filters-m16-ef-128-numeric-filter 154 +- 1.9% (7 datapoints) 156.00 1.1% No Change
vecsim-arxiv-titles-384-angular-filters-m16-ef-128-tag-filter 15832 +- 2.0% (7 datapoints) 16137.00 1.9% No Change

@codecov
Copy link
Copy Markdown

codecov bot commented Jan 8, 2026 •
edited
Loading

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.09%. Comparing base (5b35c7b) to head (4a1372d).
⚠️ Report is 4 commits behind head on master.

Additional details and impacted files 
@@            Coverage Diff             @@
##           master    #8005      +/-   ##
==========================================
+ Coverage   84.08%   84.09%   +0.01%     
==========================================
  Files         361      361              
  Lines       55171    55171              
  Branches    14393    14393              
==========================================
+ Hits        46390    46398       +8     
+ Misses       8619     8611       -8     
  Partials      162      162
Flag Coverage Δ
flow 84.86% <ø> (-0.12%) ⬇️
unit 51.02% <ø> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@meiravgri meiravgri added this pull request to the merge queue Jan 8, 2026
Merged via the queue into master with commit 21ee350 Jan 8, 2026
81 of 83 checks passed
@meiravgri meiravgri deleted the meiravg_skip_failing_benchmarks branch January 8, 2026 18:06
@redisearch-backport-pull-request
Copy link
Copy Markdown
Contributor

redisearch-backport-pull-request bot commented Jan 8, 2026

Backport failed for 2.8, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin 2.8
git worktree add -d .worktree/backport-8005-to-2.8 origin/2.8
cd .worktree/backport-8005-to-2.8
git switch --create backport-8005-to-2.8
git cherry-pick -x 21ee35069c59426827ace415565cc420dc80c088

@redisearch-backport-pull-request
Copy link
Copy Markdown
Contributor

redisearch-backport-pull-request bot commented Jan 8, 2026

Backport failed for 2.10, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin 2.10
git worktree add -d .worktree/backport-8005-to-2.10 origin/2.10
cd .worktree/backport-8005-to-2.10
git switch --create backport-8005-to-2.10
git cherry-pick -x 21ee35069c59426827ace415565cc420dc80c088

@redisearch-backport-pull-request
Copy link
Copy Markdown
Contributor

redisearch-backport-pull-request bot commented Jan 8, 2026

Backport failed for 8.2, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin 8.2
git worktree add -d .worktree/backport-8005-to-8.2 origin/8.2
cd .worktree/backport-8005-to-8.2
git switch --create backport-8005-to-8.2
git cherry-pick -x 21ee35069c59426827ace415565cc420dc80c088

@redisearch-backport-pull-request
Copy link
Copy Markdown
Contributor

redisearch-backport-pull-request bot commented Jan 8, 2026

Backport failed for 8.4, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin 8.4
git worktree add -d .worktree/backport-8005-to-8.4 origin/8.4
cd .worktree/backport-8005-to-8.4
git switch --create backport-8005-to-8.4
git cherry-pick -x 21ee35069c59426827ace415565cc420dc80c088

@meiravgri
Copy link
Copy Markdown
Collaborator Author

meiravgri commented Jan 9, 2026

No need to backport, benchmark set only exist on master

@JonasKruckenberg JonasKruckenberg mentioned this pull request Jan 14, 2026
Closed
2 tasks
JonasKruckenberg added a commit that referenced this pull request Jan 19, 2026
@JonasKruckenberg
skip benchmark-search-oss-cluster-04-primaries-threads-6
e3be5af 
This benchmark seems to be affected by the same error as #8005.
JonasKruckenberg added a commit that referenced this pull request Jan 19, 2026
@JonasKruckenberg
skip benchmark-search-oss-cluster-04-primaries-threads-6
a133da6 
This benchmark seems to be affected by the same error as #8005.
github-merge-queue bot pushed a commit that referenced this pull request Jan 19, 2026
@JonasKruckenberg
MOD-10714 Integrate Rust RSSortingVector (#6887)
e9d1c85 
* skip `benchmark-search-oss-cluster-04-primaries-threads-6`

This benchmark seems to be affected by the same error as #8005.

* RSSortingVector FFI cleanup

* fix: remove UTF-8 assumption from RSValueFFI

* MOD-10714 Integrate Rust RSSortingVector

* fix workspace hack crate

* fix lint
eyalrund pushed a commit that referenced this pull request Jan 22, 2026
@JonasKruckenberg @eyalrund
MOD-10714 Integrate Rust RSSortingVector (#6887)
bb89d07 
* skip `benchmark-search-oss-cluster-04-primaries-threads-6`

This benchmark seems to be affected by the same error as #8005.

* RSSortingVector FFI cleanup

* fix: remove UTF-8 assumption from RSValueFFI

* MOD-10714 Integrate Rust RSSortingVector

* fix workspace hack crate

* fix lint
LukeMathWalker pushed a commit that referenced this pull request Jan 26, 2026
@JonasKruckenberg @LukeMathWalker
MOD-10714 Integrate Rust RSSortingVector (#6887)
8f2f6c1 
* skip `benchmark-search-oss-cluster-04-primaries-threads-6`

This benchmark seems to be affected by the same error as #8005.

* RSSortingVector FFI cleanup

* fix: remove UTF-8 assumption from RSValueFFI

* MOD-10714 Integrate Rust RSSortingVector

* fix workspace hack crate

* fix lint
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@alonre24 alonre24 alonre24 approved these changes

Assignees

No one assigned

Labels

action:run-benchmark backport 2.8 backport 2.10 backport 8.2 backport 8.4 size:XS

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@meiravgri @fcostaoliveira @alonre24