Skip to content

Use atomic BitVec for GraphLayersBuilder::ready_list#7468

Merged
timvisee merged 1 commit intodevfrom
atomic-bitvec
Oct 29, 2025
Merged

Use atomic BitVec for GraphLayersBuilder::ready_list#7468
timvisee merged 1 commit intodevfrom
atomic-bitvec

Conversation

@xzfc
Copy link
Member

@xzfc xzfc commented Oct 29, 2025

This PR changes the type of GraphLayersBuilder::ready_list from RwLock<BitVec> to BitVec<AtomicUsize>, reducing the contention during index building.

Benchmark

20.4s → 13.1s on a subset of deep-image-96 on 8 cores.

Log (dev) 
2025-10-29T02:02:19.278132Z DEBUG segment::index::hnsw_index::hnsw: valid points: 476016, missing points: 0, missing ratio: 0.000, do_heal: true
2025-10-29T02:02:21.257368Z DEBUG segment::index::hnsw_index::hnsw: valid points: 500736, missing points: 0, missing ratio: 0.000, do_heal: true
2025-10-29T02:02:21.257463Z DEBUG segment::index::hnsw_index::hnsw: building HNSW for 976752 vectors with 8 CPUs
2025-10-29T02:02:21.327444Z DEBUG segment::index::hnsw_index::hnsw: Reusing 500736 points from the old index, healing 0 points
2025-10-29T02:02:22.322933Z DEBUG segment::index::hnsw_index::hnsw: Migrated in 784.542751ms
2025-10-29T02:02:42.758644Z DEBUG segment::index::hnsw_index::hnsw: Finish main graph in time 20.435608307s
2025-10-29T02:02:42.758670Z DEBUG segment::index::hnsw_index::hnsw: skip building additional HNSW links
2025-10-29T02:02:43.325604Z DEBUG segment::index::hnsw_index::hnsw: finish additional payload field indexing
Log (this PR) 
2025-10-29T02:00:47.744633Z DEBUG segment::index::hnsw_index::hnsw: valid points: 476016, missing points: 0, missing ratio: 0.000, do_heal: true
2025-10-29T02:00:49.650759Z DEBUG segment::index::hnsw_index::hnsw: valid points: 500736, missing points: 0, missing ratio: 0.000, do_heal: true
2025-10-29T02:00:49.650796Z DEBUG segment::index::hnsw_index::hnsw: building HNSW for 976752 vectors with 8 CPUs
2025-10-29T02:00:49.719778Z DEBUG segment::index::hnsw_index::hnsw: Reusing 500736 points from the old index, healing 0 points
2025-10-29T02:00:50.665746Z DEBUG segment::index::hnsw_index::hnsw: Migrated in 741.815747ms
2025-10-29T02:01:03.810537Z DEBUG segment::index::hnsw_index::hnsw: Finish main graph in time 13.144580351s
2025-10-29T02:01:03.810564Z DEBUG segment::index::hnsw_index::hnsw: skip building additional HNSW links
2025-10-29T02:01:04.397221Z DEBUG segment::index::hnsw_index::hnsw: finish additional payload field indexing

@xzfc xzfc requested review from IvanPleshkov and generall October 29, 2025 02:13
coderabbitai[bot]

This comment was marked as resolved.

Sign in to view

@qdrant qdrant deleted a comment from coderabbitai bot Oct 29, 2025
timvisee
timvisee approved these changes Oct 29, 2025
View reviewed changes
Copy link
Member

@timvisee timvisee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is awesome! I'm curious if we have more candidates like this.

@timvisee
Copy link
Member

timvisee commented Oct 29, 2025 •
edited
Loading

Merging this now since I don't see any negative side effects.

Edit: looks like I didn't press the button 😆

@timvisee timvisee merged commit 5dcfaf1 into dev Oct 29, 2025
15 checks passed
@timvisee timvisee deleted the atomic-bitvec branch October 29, 2025 10:56
timvisee pushed a commit that referenced this pull request Nov 14, 2025
@xzfc @timvisee
Replace RwLock<BitVec> with BitVec<AtomicUsize> (#7468)
1f3cad7
@timvisee timvisee mentioned this pull request Nov 14, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@coderabbitai coderabbitai[bot] coderabbitai[bot] left review comments

@timvisee timvisee timvisee approved these changes

@generall generall generall approved these changes

@IvanPleshkov IvanPleshkov Awaiting requested review from IvanPleshkov

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@xzfc @timvisee @generall