HttpHandlers should return correct list of objects#49283
HttpHandlers should return correct list of objects#49283tlrx merged 2 commits intoelastic:masterfrom
Conversation
|
Pinging @elastic/es-distributed (:Distributed/Snapshot/Restore) |
original-brownbear
left a comment
There was a problem hiding this comment.
This looks good I think. Could we add a test for S3 and Azure where we make sure that BlobContainer listing child directories and listing all children never picks up blobs from nested "directories" to make sure this continues to work?
That sounds like the responsibility of |
Jup exactly, that's what we should do. I guess we even have an issue that is asking for that #30424 :) |
|
@original-brownbear Would it be acceptable to do this as a follow up pr? |
original-brownbear
left a comment
There was a problem hiding this comment.
LGTM, yea ofc follow-up is just fine :) Let's just do it at some point IMO.
|
Thanks @original-brownbear ! |
This commit fixes the server side logic of "List Objects" operations of Azure and S3 fixtures. Until today, the fixtures were returning a " flat" view of stored objects and were not correctly handling the delimiter parameter. This causes some objects listing to be wrongly interpreted by the snapshot deletion logic in Elasticsearch which relies on the ability to list child containers of BlobContainer (#42653) to correctly delete stale indices. As a consequence, the blobs were not correctly deleted from the emulated storage service and stayed in heap until they got garbage collected, causing CI failures like #48978. This commit fixes the server side logic of Azure and S3 fixture when listing objects so that it now return correct common blob prefixes as expected by the snapshot deletion process. It also adds an after-test check to ensure that tests leave the repository empty (besides the root index files). Closes #48978
This commit fixes the server side logic of "List Objects" operations of Azure and S3 fixtures. Until today, the fixtures were returning a " flat" view of stored objects and were not correctly handling the delimiter parameter. This causes some objects listing to be wrongly interpreted by the snapshot deletion logic in Elasticsearch which relies on the ability to list child containers of BlobContainer (#42653) to correctly delete stale indices. As a consequence, the blobs were not correctly deleted from the emulated storage service and stayed in heap until they got garbage collected, causing CI failures like #48978. This commit fixes the server side logic of Azure and S3 fixture when listing objects so that it now return correct common blob prefixes as expected by the snapshot deletion process. It also adds an after-test check to ensure that tests leave the repository empty (besides the root index files). Closes #48978
This pull request fixes the server side logic of "List Objects" operations of Azure and S3 fixtures. Until today, the fixtures were returning a "flat" view of stored objects and were not correctly handling the
delimiterparameter. This causes some objects listing to be wrongly interpreted by the snapshot deletion logic in Elasticsearch which relies on the ability to list child containers of BlobContainer (#42653) to correctly delete stale indices.As a consequence, the blobs were not correctly deleted from the emulated storage service and stayed in heap until they got garbage collected, causing CI failures like #48978 where OOM errors happen in the test JVM (but it took me a while to figure this out...).
This pull request fixes the server side logic of Azure and S3 fixture when listing objects so that it now return correct common blob prefixes as expected by the snapshot deletion process. It also adds an after-test check to ensure that tests leave the repository empty (besides the root index files).
Closes #48978