Save Memory on Large Repository Metadata Blob Writes#74313
Merged
original-brownbear merged 35 commits intoelastic:masterfrom Jun 29, 2021
original-brownbear:save-memory-large-meta-blob-writes
Merged
Save Memory on Large Repository Metadata Blob Writes#74313original-brownbear merged 35 commits intoelastic:masterfrom original-brownbear:save-memory-large-meta-blob-writes
original-brownbear merged 35 commits intoelastic:masterfrom
original-brownbear:save-memory-large-meta-blob-writes
Conversation
| boolean atomic, | ||
| CheckedConsumer<OutputStream, IOException> writer | ||
| ) throws IOException { | ||
| // TODO: this is just a stop-gap solution for until we have an encrypted output stream wrapper |
Contributor
Author
There was a problem hiding this comment.
Given how work on the encrypted repo has stalled yet again at this point, I don't think it's worth it investing a lot of time here to code up an output stream implementation as well. We can do that (and it shouldn't be too hard) when the time comes.
Contributor
There was a problem hiding this comment.
Maybe we should create an issue to keep track of this?
| final String indexBlob = INDEX_FILE_PREFIX + Long.toString(newGen); | ||
| logger.debug("Repository [{}] writing new index generational blob [{}]", metadata.name(), indexBlob); | ||
| try (ReleasableBytesStreamOutput out = new ReleasableBytesStreamOutput(bigArrays)) { | ||
| writeAtomic(blobContainer(), indexBlob, out -> { |
Contributor
Author
There was a problem hiding this comment.
As we do streaming reads of RepositoryData already, adding streaming writes here removes the 2G size limit on it (though the most important part of this is that we save ourselves the trouble of materializing the bytes on heap for large repos where this has put a lot of stress on master's heap often).
original-brownbear
added a commit
that referenced
this pull request
Jun 28, 2021
Extracted the chunked output stream logic from #74313 and added tests for it to make it easier to review.
tlrx
approved these changes
Jun 29, 2021
...pository-s3/src/test/java/org/elasticsearch/repositories/s3/S3BlobContainerRetriesTests.java
Outdated
Show resolved
Hide resolved
| final String indexBlob = INDEX_FILE_PREFIX + Long.toString(newGen); | ||
| logger.debug("Repository [{}] writing new index generational blob [{}]", metadata.name(), indexBlob); | ||
| try (ReleasableBytesStreamOutput out = new ReleasableBytesStreamOutput(bigArrays)) { | ||
| writeAtomic(blobContainer(), indexBlob, out -> { |
server/src/main/java/org/elasticsearch/repositories/blobstore/ChecksumBlobStoreFormat.java
Outdated
Show resolved
Hide resolved
Contributor
Author
|
Thanks Francisco and Tanguy! |
original-brownbear
added a commit
that referenced
this pull request
Jun 29, 2021
This PR adds a new API for doing streaming serialization writes to a repository to enable repository metadata of arbitrary size and at bounded memory during writing. The existing write-APIs require knowledge of the eventual blob size beforehand. This forced us to materialize the serialized blob in memory before writing, costing a lot of memory in case of e.g. very large RepositoryData (and limiting us to 2G max blob size). With this PR the requirement to fully materialize the serialized metadata goes away and the memory overhead becomes completely bounded by the outbound buffer size of the repository implementation. As we move to larger repositories this makes master node stability a lot more predictable since writing out RepositoryData does not take as much memory any longer (same applies to shard level metadata), enables aggregating multiple metadata blobs into a single larger blobs without massive overhead and removes the 2G size limit on RepositoryData. backport of #74313 and #74620
tlrx
added a commit
that referenced
this pull request
Jun 30, 2021
benwtrent
pushed a commit
to benwtrent/elasticsearch
that referenced
this pull request
Jun 30, 2021
See elastic#53119 for more context about why those tests are muted on JDK8. They start failing more often recently now elastic#74313 and elastic#74620 have been merged, as reported in elastic#74739.
benwtrent
added a commit
that referenced
this pull request
Jun 30, 2021
Contributor
|
@original-brownbear , do you know if this PR should change in any way our statement in the docs saying that |
Contributor
Author
|
Just for completeness sake here: @eedugon as discussed on another channel, this is waiting for a docs update only. The |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds a new API for doing streaming serialization writes to a repository to enable repository metadata of arbitrary size and at bounded memory during writing.
The existing write-APIs require knowledge of the eventual blob size beforehand. This forced us to materialize the serialized blob in memory before writing, costing a lot of memory in case of e.g. very large
RepositoryData(and limiting us to2Gmax blob size).With this PR the requirement to fully materialize the serialized metadata goes away and the memory overhead becomes completely bounded by the outbound buffer size of the repository implementation.
As we move to larger repositories this makes master node stability a lot more predictable since writing out
RepositoryDatadoes not take as much memory any longer (same applies to shard level metadata), enables aggregating multiple metadata blobs into a single larger blobs without massive overhead and removes the 2G size limit onRepositoryData.