Skip to content

[8.6] Simplify and optimize deduplication of RepositoryData for a non-caching repository instance (#91851)#91866

Merged
elasticsearchmachine merged 3 commits intoelastic:8.6from
original-brownbear:backport/8.6/pr-91851
Jan 3, 2023
Merged

[8.6] Simplify and optimize deduplication of RepositoryData for a non-caching repository instance (#91851)#91866
elasticsearchmachine merged 3 commits intoelastic:8.6from
original-brownbear:backport/8.6/pr-91851

Conversation

@original-brownbear
Copy link
Copy Markdown
Contributor

Backports the following commits to 8.6:

…ng repository instance (#91851)

This makes use of the new deduplicator infrastructure to move to more
efficient deduplication mechanics.
The existing solution hardly ever deduplicated because it would only
deduplicate after the repository entered a consistent state. The
adjusted solution is much simpler, in that it simply deduplicates such
that only a single loading of `RepositoryData` will ever happen at a
time, fixing memory issues from massively concurrent loading of the repo
data as described in #89952.

closes #89952
@original-brownbear original-brownbear added :Distributed/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs >bug auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) backport Team:Distributed Meta label for distributed team. labels Nov 23, 2022
@elasticsearchmachine elasticsearchmachine merged commit 03983ae into elastic:8.6 Jan 3, 2023
@original-brownbear original-brownbear deleted the backport/8.6/pr-91851 branch January 3, 2023 09:47
elasticsearchmachine pushed a commit that referenced this pull request Jan 4, 2023
…-caching repository instance (#91851) (#91866) (#92661)

* Simplify and optimize deduplication of RepositoryData for a non-caching repository instance (#91851)

This makes use of the new deduplicator infrastructure to move to more
efficient deduplication mechanics.
The existing solution hardly ever deduplicated because it would only
deduplicate after the repository entered a consistent state. The
adjusted solution is much simpler, in that it simply deduplicates such
that only a single loading of `RepositoryData` will ever happen at a
time, fixing memory issues from massively concurrent loading of the repo
data as described in #89952.

closes #89952

* fix compile
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) backport >bug :Distributed/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs Team:Distributed Meta label for distributed team. v8.6.1

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants