feat: Optimize the collection loading process during startup#8053
feat: Optimize the collection loading process during startup#8053generall merged 10 commits intoqdrant:devfrom
Conversation
91b63b4 to
6db00bc
Compare
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
b33a06b to
cace13d
Compare
cace13d to
b49e74e
Compare
timvisee
left a comment
There was a problem hiding this comment.
Thanks @1995chen! I like this one with configuration and keeping defaults much better. We can change defaults in the future if we have some solid benchmarks across a wide variety of deployments.
I do have some review remarks, most notably that I don't like the use of global state. I'll take care of this and will add the necessary commits to your branch. 👍
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
|
I tested the following setup locally:
The default case (1/1/8) loads in 75 seconds. With a (4/4/8) configuration loading took longer, taking 105 seconds. QDRANT__STORAGE__PERFORMANCE__MAX_CONCURRENT_COLLECTION_LOADS=4 QDRANT__STORAGE__PERFORMANCE__MAX_CONCURRENT_SHARD_LOAD=4 QDRANT__STORAGE__PERFORMANCE__MAX_CONCURRENT_SEGMENT_LOAD=8 ./target/perf/qdrantMy best guess is that the high amount of context switching makes load performance worse on a setup like this. I can imagine it being effective on a networked disk with higher latency though. And so I don't have an objection to merge this. |
Thank you so much, @timvisee and @generall ! |
* feat: optimize the collection loading process during startup * simplify config options * chore: update config.yaml * Remove global state * Disallow zero values * Remove intermediate map * Rename ConcurrentLoadConfig to LoadConcurrencyConfig * Flatten config * Accept string inputs from environment variables * Fix test compilation --------- Co-authored-by: generall <andrey@vasnetsov.com> Co-authored-by: chenliang <chenliang@chenliangdeMacBook-Air.local> Co-authored-by: timvisee <tim@visee.me>
|
We've released Qdrant 1.17.0 which includes this improvement. Thank you for implementing this one. |
Fixes #7975
We have added some configurations to support concurrency settings for collection, shard, and segment loading, while maintaining the original default values, enabling Qdrant to achieve faster loading speeds on AWS EBS network volumes.