Skip to content

[DiskBBQ] Use the new merge executor for intra-merge parallelism#139942

Merged
iverase merged 21 commits intoelastic:mainfrom
iverase:diskbbq-multi-threaded-mergin
Jan 12, 2026
Merged

[DiskBBQ] Use the new merge executor for intra-merge parallelism#139942
iverase merged 21 commits intoelastic:mainfrom
iverase:diskbbq-multi-threaded-mergin

Conversation

@iverase
Copy link
Copy Markdown
Contributor

@iverase iverase commented Dec 23, 2025 •
edited
Loading

Follow up of #137853. DiskBBQ hierarchical k-means supports multi-threaded execution which should provide a nice speed up.

@iverase iverase requested a review from benwtrent December 23, 2025 12:23
@iverase iverase marked this pull request as draft December 23, 2025 12:23
@iverase iverase marked this pull request as ready for review December 29, 2025 13:22
@elasticsearchmachine elasticsearchmachine added the Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch label Dec 29, 2025
@elasticsearchmachine
Copy link
Copy Markdown
Collaborator

elasticsearchmachine commented Dec 29, 2025

Pinging @elastic/es-search-relevance (Team:Search Relevance)

@iverase iverase marked this pull request as draft December 30, 2025 15:25
@iverase
Copy link
Copy Markdown
Contributor Author

iverase commented Dec 30, 2025

Setting it back to draft because performance test are showing a regression.

@iverase
Copy link
Copy Markdown
Contributor Author

iverase commented Jan 8, 2026

This is ready. quick local benchmarks show the improvements at merge time.

|                       Cumulative merge time of primary shards |                     |      7.65697     |      2.99245     |     -4.66452 |    min |  -60.92% |
|                      Cumulative merge count of primary shards |                     |      5           |      4           |     -1       |        |  -20.00% |
|                Min cumulative merge time across primary shard |                     |      1.8844      |      1.38365     |     -0.50075 |    min |  -26.57% |
|             Median cumulative merge time across primary shard |                     |      3.82848     |      1.49622     |     -2.33226 |    min |  -60.92% |
|                Max cumulative merge time across primary shard |                     |      5.77257     |      1.6088      |     -4.16377 |    min |  -72.13% |
|              Cumulative merge throttle time of primary shards |                     |      3.55443     |      0.497433    |     -3.057   |    min |  -86.01% |
|       Min cumulative merge throttle time across primary shard |                     |      0.3508      |      0.21935     |     -0.13145 |    min |  -37.47% |
|    Median cumulative merge throttle time across primary shard |                     |      1.77722     |      0.248717    |     -1.5285  |    min |  -86.01% |
|       Max cumulative merge throttle time across primary shard |                     |      3.20363     |      0.278083    |     -2.92555 |    min |  -91.32% |

@iverase iverase marked this pull request as ready for review January 8, 2026 11:19
benwtrent
benwtrent reviewed Jan 8, 2026
View reviewed changes
Copy link
Copy Markdown
Member

@benwtrent benwtrent left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't we also do this for the existing format? It should be doable right as the on-disk format for reading stays the same.

@benwtrent benwtrent added the build-benchmark Trigger build benchmark job label Jan 8, 2026
@iverase
Copy link
Copy Markdown
Contributor Author

iverase commented Jan 8, 2026

Buildkite benchmark this with so-vector please

@iverase
Copy link
Copy Markdown
Contributor Author

iverase commented Jan 8, 2026

Damn, that benchmark will not run diskBBQ

@benwtrent
Copy link
Copy Markdown
Member

benwtrent commented Jan 8, 2026

Damn, that benchmark will not run diskBBQ

@iverase I think it should....

@benwtrent
Copy link
Copy Markdown
Member

benwtrent commented Jan 8, 2026

Merge time is indeed twice as fast (using 2x more threads):
image

One thing that was weird (could just be a infra flakiness or something...), match-all kNN is way faster in candidate:

image

Looking at the other metrics, everything else seems pretty even.

@iverase
Copy link
Copy Markdown
Contributor Author

iverase commented Jan 10, 2026

Buildkite benchmark this with so-vector please

@elasticmachine
Copy link
Copy Markdown
Collaborator

elasticmachine commented Jan 10, 2026 •
edited
Loading

💚 Build Succeeded

This build ran two so-vector benchmarks to evaluate performance impact of this PR.

History

@iverase iverase removed the build-benchmark Trigger build benchmark job label Jan 11, 2026
@iverase iverase merged commit a303c24 into elastic:main Jan 12, 2026
35 checks passed
@iverase iverase deleted the diskbbq-multi-threaded-mergin branch January 12, 2026 15:43
szybia added a commit to szybia/elasticsearch that referenced this pull request Jan 12, 2026
@szybia
Merge remote-tracking branch 'upstream/main' into reindex-cancel-mult…
1180837 
…i-project-tests

* upstream/main: (23 commits)
  Fix `testAckListenerReceivesNacksIfPublicationTimesOut` (elastic#140514)
  Reduce priority of clear-cache tasks (elastic#139685)
  Add docs and tests about `StreamOutput` to memory (elastic#140365)
  ES|QL - dense_vector support for COUNT, PRESENT, ABSENT aggregator functions (elastic#139914)
  Add release notes for v9.2.4 release (elastic#140487)
  Add release notes for v9.1.10 release (elastic#140488)
  Add conncectors release notes for 9.1.10, 9.2.4 (elastic#140499)
  Add parameter support in PromQL query durations (elastic#139873)
  Improve testing of STS credentials reloading (elastic#140114)
  Fix zstd native binary publishing script to support newer versions (elastic#140485)
  Add FlattenedFieldBinaryVsSortedSetDocValuesSyntheticSourceIT (elastic#140489)
  Store fallback match only text fields in binary doc values (elastic#140189)
  [DiskBBQ] Use the new merge executor for intra-merge parallelism (elastic#139942)
  ESQL: introduce support for mapping-unavailable fields (elastic#140463)
  Add ESNextOSQVectorsScorerTests (elastic#140436)
  Disable high cardinality tests on release builds (elastic#140503)
  ESQL: TRange timezone support (elastic#139911)
  Directly compressing `StreamOutput` (elastic#140502)
  ES|QL - fix dense vector enrich bug (elastic#139774)
  Use CrossProjectModeDecider in RemoteClusterService (elastic#140481)
  ...
spinscale pushed a commit to spinscale/elasticsearch that referenced this pull request Jan 21, 2026
@iverase @spinscale
[DiskBBQ] Use the new merge executor for intra-merge parallelism (ela…
43ad79a 
…stic#139942)

DiskBBQ hierarchical k-means supports multi-threaded execution which should provide a nice speed up.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@benwtrent benwtrent benwtrent approved these changes

Assignees

No one assigned

Labels

>non-issue :Search Relevance/Vectors Vector search Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch v9.4.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@iverase @elasticsearchmachine @benwtrent @elasticmachine