Make update_worker_fn a blocking task, instead of async#7015
Merged
Conversation
2e6605d to
3c1842e
Compare
ffuugoo
commented
Aug 11, 2025
Comment on lines
+46
to
+71
| tokio::task::block_in_place(|| { | ||
| // Allow only one update at a time, ensure no data races between segments. | ||
| // let _lock = self.update_lock.lock().unwrap(); | ||
|
|
||
| let operation_result = match operation { | ||
| CollectionUpdateOperations::PointOperation(point_operation) => { | ||
| process_point_operation(segments, op_num, point_operation, hw_counter) | ||
| } | ||
| CollectionUpdateOperations::VectorOperation(vector_operation) => { | ||
| process_vector_operation(segments, op_num, vector_operation, hw_counter) | ||
| } | ||
| CollectionUpdateOperations::PayloadOperation(payload_operation) => { | ||
| process_payload_operation(segments, op_num, payload_operation, hw_counter) | ||
| } | ||
| CollectionUpdateOperations::FieldIndexOperation(index_operation) => { | ||
| process_field_index_operation(segments, op_num, &index_operation, hw_counter) | ||
| } | ||
| }; | ||
| let scroll_lock = segments.read().scroll_read_lock.clone(); | ||
| let _scroll_lock = scroll_lock.blocking_write(); | ||
|
|
||
| let operation_result = match operation { | ||
| CollectionUpdateOperations::PointOperation(point_operation) => { | ||
| process_point_operation(segments, op_num, point_operation, hw_counter) | ||
| } | ||
| CollectionUpdateOperations::VectorOperation(vector_operation) => { | ||
| process_vector_operation(segments, op_num, vector_operation, hw_counter) | ||
| } | ||
| CollectionUpdateOperations::PayloadOperation(payload_operation) => { | ||
| process_payload_operation(segments, op_num, payload_operation, hw_counter) | ||
| } | ||
| CollectionUpdateOperations::FieldIndexOperation(index_operation) => { | ||
| process_field_index_operation(segments, op_num, &index_operation, hw_counter) | ||
| } | ||
| }; | ||
|
|
||
| CollectionUpdater::handle_update_result(segments, op_num, &operation_result); | ||
| CollectionUpdater::handle_update_result(segments, op_num, &operation_result); | ||
|
|
||
| operation_result | ||
| operation_result | ||
| }) |
Contributor
Author
There was a problem hiding this comment.
CollectionUpdate::update is blocking, but we call it from async context in a few places. Seems reasonable to wrap the whole update into block_in_place.
block_in_place does have a performance hit when called from async, but
- we already use
block_in_placeinsideupdate, so we already "pay" for this perf hit, we just extendblock_in_placescope here, which may actually improve perf when called from async update_worker_fn(which is the most importantupdatepath) is a blocking task now, and other places where we still callupdatefrom async are less critical (e.g.,LocalShard::load_from_wal)
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
timvisee
reviewed
Aug 11, 2025
Member
timvisee
left a comment
There was a problem hiding this comment.
Can we do a benchmark just to confirm we aren't regressing in any way?
eb37c6c to
05e46fc
Compare
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
05e46fc to
2af6deb
Compare
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Contributor
Author
I've made a few bfb runs, seems about the same? Similar RPS, similar total upload time. dev
first PR run
second PR run
|
timvisee
approved these changes
Aug 27, 2025
Member
timvisee
left a comment
There was a problem hiding this comment.
Thanks!
Yeah the bench differences are within margin of error. Nice.
Contributor
Author
I'll also keep an eye on continuous benchmark in the next few days. |
timvisee
pushed a commit
that referenced
this pull request
Sep 29, 2025
Merged
This was referenced Oct 14, 2025
Merged
This was referenced Nov 29, 2025
3 tasks
This was referenced Mar 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR tweaks
UpdateHandler::update_worker_fnto make it a blocking call instead of async.update_worker_fnis anasync fn, even though it's almost entirely a blocking call (e.g.,CollectionUpdater::updatecall, which makes up 99% of work thatupdate_worker_fndoes, is a blocking call).I don't expect this will have any immediate noticeable effect, but it might improve perf somehow or resolve minor/rare issues, because we've seen a bunch of weird bugs caused by blocking async runtime before, and
updateis a pretty hot path (even if it's run on its own dedicated runtime).All Submissions:
devbranch. Did you create your branch fromdev?New Feature Submissions:
cargo +nightly fmt --allcommand prior to submission?cargo clippy --all --all-featurescommand?Changes to Core Features: