Conversation
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
timvisee
left a comment
There was a problem hiding this comment.
Works as I expect.
It currently is not possible for users to change this number in existing collections. It is persisted on collection creation, and our collection update API does not expose it. Maybe that's something we can add in a separate PR. I don't think it's an immediate problem and am fine by merging this as is.
Marked 'request changes' until we bump the WAL dependency.
config/config.yaml
Outdated
| # Number of closed WAL segments to keep. | ||
| wal_retain_closed: 1 |
There was a problem hiding this comment.
I vote to not add it to our config.yaml by default and to keep it a bit more hidden because it is quite specific. If people actively appear to use it, we can always expose it here later.
Wdyt?
There was a problem hiding this comment.
Fine. Closed vs open segments is a bit deeper in terms of abstraction. So it's fine to not expose it right away. We can always do it later.
Cargo.toml
Outdated
| uuid = { version = "1.17", features = ["v4", "serde"] } | ||
| validator = { version = "0.18.1", features = ["derive"] } | ||
| wal = { git = "https://github.com/qdrant/wal.git", rev = "c4b26b9c0ccc0e06ba7391189e4c8eac051ca531" } | ||
| wal = { git = "https://github.com/qdrant/wal.git", branch = "max-closed-to-preserve" } |
There was a problem hiding this comment.
Let's switch this back to a commit hash once merged in qdrant/wal.
|
I think one caveat of this approach is that it makes snapshots larger by default, especially in setup with a high segments count. Currently we have 2 WAL segments (1 open & 1 closed) = 32mb + 32mb = 64mb per data segment. With Not a blocker for sure but something to keep in mind during operations. WDYT? |
|
@agourlay the default is still just 1 closed segment, which is the same as before. Snapshots should therefore be exactly the same size. The mention of If a user chooses to set a higher number of segments to maintain, then we keep more data. And yes, by definition, that means larger snapshots. But there's no way around that. In that case having more history is actually desired. |
* Retain more closed segments * Add back rocksdb * Update WalOptions/WalConfig across the code * Use NonZeroUsize * Expose via APIs * Update gRPC docs * Fix stoarge compat test * recompile openapi.json with rocksdb * default wal retain closed fn * update openapi.json * Use qdrant/wal latest commit and remove from config.yaml
Introduces a new wal config called
wal_retain_closed. It basically makes wal delta transfers (faster) more likely in large clusters over streaming records (slower). The default value is 1 to maintain backwards compatibility.It can be configured using these methods:
Env var
export QDRANT__STORAGE__WAL__WAL_RETAIN_CLOSED=5Qdrant YAML Config
Or while creating the collection you can pass:
Note that it can't be modified after creating collections (same as other wal configs)