push: improve performance by 3 to 5 times, use gzip level 1 by default#48106
Open
morotti wants to merge 1 commit intomoby:masterfrom
Open
push: improve performance by 3 to 5 times, use gzip level 1 by default#48106morotti wants to merge 1 commit intomoby:masterfrom
morotti wants to merge 1 commit intomoby:masterfrom
Conversation
107e383 to
555abaf
Compare
Signed-off-by: rmorotti <romain.morotti@man.com>
Author
|
(passed the git signature) |
AkihiroSuda
reviewed
Jul 2, 2024
AkihiroSuda
approved these changes
Jul 2, 2024
vvoland
reviewed
Jul 2, 2024
Contributor
vvoland
left a comment
There was a problem hiding this comment.
+1 on the change, however, I think we should still provide an option to customize this (or at least get the original behavior back).
Member
|
Yes, I think this needs a closer look;
|
thaJeztah
requested changes
Jul 2, 2024
Member
thaJeztah
left a comment
There was a problem hiding this comment.
This needs further discussion (see above)
|
Hi @thaJeztah , thanks for having a look! This change is quite important for us too as docker push takes really long currently. I saw you added a label "1-design-review", what does that mean? Does that imply it will be reviewed further by the Docker team or do you expect @morotti to provide anything else? Would it help if this change was off by default and configurable? |
roolebo
added a commit
to roolebo/moby
that referenced
this pull request
Aug 14, 2024
gzip compression is notoriously slow. The compression on skylake happens at 7MiB/s. The push rate is not sustainable for large multi-gigabyte images. zstd/fastest runs at 81 MiB/s for the same image. That gives tenfold push speedup with default flags. Resolves: moby#1266 See also: moby#48106 Signed-off-by: Roman Bolshakov <rbolshakov@ddn.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hello,
Could you correct the compression level used by docker push?
There has been open issues about this for 11 years and it's not been fixed. The fix is trivial and takes 1 line of code. #1266
If you want to do it very cleanly, you could add a setting to allow specifying a compression level 1-9 or 0-9.
(Level 0 is no compression, it's debatable whether no compression should be allowed because it affects storage on the container registry, docker hub or internal registry).
docker pushis compressing the image on the fly during the push using gzip.gzip compression is slow, the default level 6 is very slow and a poor tradeoff performance/compression.
I see the gzip compression is limited to 12 MB/s on my servers.
I see
docker pushis taking whole minutes at the ends of builds, with all the time taken by the gzip compression during the push. It's single threaded compression by the way.For people working with GPU and ML, images can approach 10 GB of content these days. It takes 10-20 minutes to push an image. All that time is taken by the inefficient gzip compression.
Rather than the default level of 6, you should use the level 1, that should make compression 3 to 5 times faster at the cost of 1-2% larger image size.
You may want to run a build of docker with this fix and try a few images to see how much differences you get.
I've had a look into the go compressor in the standard library (gzip/compress).
From what I have gathered, at the time docker was written and the old tickets were discussed, circa 2015, the go compiler was bugged, setting the compression levels did not work. Levels barely made any difference when levels 9-1 are supposed to be 10-100 MB/s on compression (or that order of magnitude). The go standard library was fixed a few years later.
(There are other improvements to use lz4/zstd or to compress on multiple threads, but that is outside of the scope of this issue, and a much bigger piece of work.)
fixes: 1266
- Description for the changelog
- A picture of a cute animal (not mandatory but encouraged)