Add runtime flag to disable opening multiple mmaps on the same file#7614
Add runtime flag to disable opening multiple mmaps on the same file#7614
Conversation
82b96d3 to
ffb6bca
Compare
coszio
left a comment
There was a problem hiding this comment.
Is it possible to add this as part of settings? I figure it can be more cleanly deployed if this setting is part of the config file.
In this PR I didn't because I pitched this as hidden setting. Though I have no objection in including it in Are you okay with changing this in a separate PR? |
|
To clarify, I definitely do not recommend to deploy with this setting. That's why I kept it hidden in the first place. It shows that the IO behavior on Docker/WSL is broken. There may definitely be more dragons we're not aware of yet, breaking production workloads in unexpected ways. These changes were only intended to not immediately break naive deployments on this environment. |
…7614) * Fix delete_page not dropping sequential mmap * In Gridstore Page, support using a single mmap * In Gridstore Page, only open multiple mmaps if supported at runtime * Support opening single mmap in UniversalMmapChunk * Rename environment var, log warning when QDRANT_NO_MULTI_MMAP is set * Support opening single mmap in MmapDenseVectors * Remove unused result response * Import LazyLock
Fixes #7610, #6676
Add the
QDRANT_NO_MULTI_MMAPenvironment variable to disable multi-mmap at runtime. It disables opening multiple memory maps to the same piece of data on disk in all of Qdrant.This pattern of opening multiple memory maps is broken in some environment, corrupting storage on disk. That's why you can now disable it on demand.
An example of such environment is Docker on Windows with a storage mount to the Windows host. This flag effectively brings us better compatibility.
Usage example:
# run without multiple mmaps for better compatibility QDRANT_NO_MULTI_MMAP=1 ./qdrantOnce memory maps are first used a warning pops up if this feature is disabled:
All Submissions:
devbranch. Did you create your branch fromdev?Changes to Core Features: