Compress container image with zstd#160665
Conversation
There was a problem hiding this comment.
Pull request overview
This PR introduces a breaking change to use zstd compression instead of gzip for container images, resulting in significantly smaller image sizes (~60% reduction) and faster pull times (~30% improvement).
Changes:
- Updated Docker build configuration to use zstd compression with level 9
- Changed OCI media types to reflect the new compression format
On what kind of machine is this measured? Home Assistant can be run on a lot of different machines and a large part of the userbase is still on raspberry pi's for example (I believe the lowest is the raspberry pi 3 64-bit). |
|
@joostlek this was measured on a x64 PC by just comparing the time of |
|
@edenhaus the images inheriting this base with Artifact example: FROM ghcr.io/home-assistant/amd64-homeassistant:edenhaus-test@sha256:e5e72f0f650a7bda368563b8556f2ea5886edb175ef1b404504ea26e471315d3
RUN head -c 10000000 /dev/urandom > /tmp/file1
RUN head -c 10000000 /dev/urandom > /tmp/file2Image generated:
Details |
I did some analysis in the past, see home-assistant/builder#245 (comment). Meanwhile we declared old installations as unsupported and bumped the minimal Docker version (with home-assistant/supervisor#6178), so I think from a support standpoint we are good. |
|
Created to test images, where I compared layer by layer, both with the command above Image zstd compression:Digest: b6ae8fd00c0a80b292a83e96eeffdcd775fbb3a1351f6fcadadaa9c966c3b2f6 Image gzip compressionDigest: 69a3f673ba95e0ad15eefa0051a996aea5e2438e8562af679f4ec705cb4c444b Comparison
Only on one layer, gzip has the smaller output. BuildtimeARM64Before: 3m 9s = 189 seconds https://github.com/home-assistant/core/actions/runs/21666771682/job/62464491628 AMD64Before: 3m 6s = 186 seconds https://github.com/home-assistant/core/actions/runs/21666771682/job/62464491609 |
agners
left a comment
There was a problem hiding this comment.
The only thing I could think of is memory usage. Lower end hardware might not have much memory. But afaik, zstd default levels (which 9 is part of) use a 8MB window size, so it should be fairly low risk.
LGTM!
|
Will do another test with different zstd level to see if we really need level 9 |
|
Complete Comparison Table
We stick to zstd-9 as it will save another 117MB of image size even when the build process takes 18%. As the image building currently takes less than 5 min, the additional 117MB savings have more benefit than a faster build time. |
Breaking change
New Container / Docker images are compressed with
zstdinstead ofgzip.The container manifest will change from
application/vnd.oci.image.layer.v1.tar+gziptoapplication/vnd.oci.image.layer.v1.tar+zstd.NOTE: As of 2021 (+5y), containerd already supports zstd, so this is not expected to break current installations.
Proposed change
Change to build container image with
zstdcompression.The container image size changes from 737.5 MB to 578.1 MB 💾 (~-22%) but the build time will increase by ~19%
Tested with the CI by @edenhaus see #160665 (comment)
Type of change
Additional information
N/A
Checklist
ruff format homeassistant tests)If user exposed functionality or configuration variables are added/changed:
If the code communicates with devices, web services, or third-party tools:
Updated and included derived files by running:
python3 -m script.hassfest.requirements_all.txt.Updated by running
python3 -m script.gen_requirements_all.To help with the load of incoming pull requests: