Skip to content

Compress container image with zstd#160665

Merged
edenhaus merged 3 commits intohome-assistant:devfrom
duhow:docker-zstd-upstream
Feb 4, 2026
Merged

Compress container image with zstd#160665
edenhaus merged 3 commits intohome-assistant:devfrom
duhow:docker-zstd-upstream

Conversation

@duhow
Copy link
Copy Markdown
Contributor

@duhow duhow commented Jan 10, 2026

Breaking change

New Container / Docker images are compressed with zstd instead of gzip.
The container manifest will change from application/vnd.oci.image.layer.v1.tar+gzip to application/vnd.oci.image.layer.v1.tar+zstd.

NOTE: As of 2021 (+5y), containerd already supports zstd, so this is not expected to break current installations.

Proposed change

Change to build container image with zstd compression.

The container image size changes from 737.5 MB to 578.1 MB 💾 (~-22%) but the build time will increase by ~19%
Tested with the CI by @edenhaus see #160665 (comment)

Type of change

  • Dependency upgrade
  • Bugfix (non-breaking change which fixes an issue)
  • New integration (thank you!)
  • New feature (which adds functionality to an existing integration)
  • Deprecation (breaking change to happen in the future)
  • Breaking change (fix/feature causing existing functionality to break)
  • Code quality improvements to existing code or addition of tests

Additional information

N/A

Checklist

  • I understand the code I am submitting and can explain how it works.
  • The code change is tested and works locally.
  • Local tests pass. Your PR cannot be merged unless tests pass
  • There is no commented out code in this PR.
  • I have followed the development checklist
  • I have followed the perfect PR recommendations
  • The code has been formatted using Ruff (ruff format homeassistant tests)
  • Tests have been added to verify that the new code works.
  • Any generated code has been carefully reviewed for correctness and compliance with project standards.

If user exposed functionality or configuration variables are added/changed:

If the code communicates with devices, web services, or third-party tools:

  • The manifest file has all fields filled out correctly.
    Updated and included derived files by running: python3 -m script.hassfest.
  • New or updated dependencies have been added to requirements_all.txt.
    Updated by running python3 -m script.gen_requirements_all.
  • For the updated dependencies - a link to the changelog, or at minimum a diff between library versions is added to the PR description.

To help with the load of incoming pull requests:

Copilot AI review requested due to automatic review settings January 10, 2026 10:59
@duhow duhow requested a review from a team as a code owner January 10, 2026 10:59
@home-assistant home-assistant bot added breaking-change cla-signed small-pr PRs with less than 30 lines. labels Jan 10, 2026
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a breaking change to use zstd compression instead of gzip for container images, resulting in significantly smaller image sizes (~60% reduction) and faster pull times (~30% improvement).

Changes:

  • Updated Docker build configuration to use zstd compression with level 9
  • Changed OCI media types to reflect the new compression format

@joostlek
Copy link
Copy Markdown
Member

pull time + decompression is roughly 30% faster.

On what kind of machine is this measured? Home Assistant can be run on a lot of different machines and a large part of the userbase is still on raspberry pi's for example (I believe the lowest is the raspberry pi 3 64-bit).

@duhow
Copy link
Copy Markdown
Contributor Author

duhow commented Jan 10, 2026

@joostlek this was measured on a x64 PC by just comparing the time of docker pull (current latest image vs new created image). The times in Home Assistant (HAOS) pulling updates might be different, but overall faster than now.

@edenhaus edenhaus changed the title feat!: Build container image with zstd Build container image with zstd Jan 21, 2026
@edenhaus
Copy link
Copy Markdown
Member

@edenhaus edenhaus self-assigned this Jan 31, 2026
@edenhaus
Copy link
Copy Markdown
Member

@sairon Can you please check if all supported versions are supporting it? CC @agners

@duhow Can you please check if we set it in the base images it will be automatically set for all images inheriting

@duhow
Copy link
Copy Markdown
Contributor Author

duhow commented Feb 3, 2026

@edenhaus the images inheriting this base with zstd will continue to build new layers using gzip, unless the option to compress with zstd is added. The previous zstd layers will be used (not re-compressed into another format).

Artifact example:

Dockerfile

FROM ghcr.io/home-assistant/amd64-homeassistant:edenhaus-test@sha256:e5e72f0f650a7bda368563b8556f2ea5886edb175ef1b404504ea26e471315d3

RUN head -c 10000000 /dev/urandom > /tmp/file1
RUN head -c 10000000 /dev/urandom > /tmp/file2

Image generated:

ghcr.io/duhow/amd64-homeassistant:test-inherit@sha256:124a0dea27def46ae3ded9690bd1b71551ea3a9dc59f0241cfa41ade0ef0295f

Details
skopeo inspect --raw  \
docker://ghcr.io/duhow/amd64-homeassistant@sha256:124a0dea27def46ae3ded9690bd1b71551ea3a9dc59f0241cfa41ade0ef0295f | \
 jq -r '.layers | .[] | [.mediaType,.size] | join(" ")'

application/vnd.oci.image.layer.v1.tar+zstd 3553881
application/vnd.oci.image.layer.v1.tar+zstd 95
application/vnd.oci.image.layer.v1.tar+zstd 12584080
application/vnd.oci.image.layer.v1.tar+zstd 248
application/vnd.oci.image.layer.v1.tar+zstd 13990220
application/vnd.oci.image.layer.v1.tar+zstd 2786639
application/vnd.oci.image.layer.v1.tar+zstd 51300115
application/vnd.oci.image.layer.v1.tar+zstd 6227095
application/vnd.oci.image.layer.v1.tar+zstd 16
application/vnd.oci.image.layer.v1.tar+zstd 58240
application/vnd.oci.image.layer.v1.tar+zstd 465461
application/vnd.oci.image.layer.v1.tar+zstd 179
application/vnd.oci.image.layer.v1.tar+zstd 5332832
application/vnd.oci.image.layer.v1.tar+zstd 235742
application/vnd.oci.image.layer.v1.tar+zstd 8417
application/vnd.oci.image.layer.v1.tar+zstd 1479
application/vnd.oci.image.layer.v1.tar+zstd 6791694
application/vnd.oci.image.layer.v1.tar+zstd 17968571
application/vnd.oci.image.layer.v1.tar+zstd 16
application/vnd.oci.image.layer.v1.tar+zstd 918
application/vnd.oci.image.layer.v1.tar+zstd 3164
application/vnd.oci.image.layer.v1.tar+zstd 90624180
application/vnd.oci.image.layer.v1.tar+zstd 14695
application/vnd.oci.image.layer.v1.tar+zstd 345356919
application/vnd.oci.image.layer.v1.tar+zstd 22322570
application/vnd.oci.image.layer.v1.tar+zstd 16189327
application/vnd.oci.image.layer.v1.tar+zstd 82
application/vnd.oci.image.layer.v1.tar+gzip 10003276
application/vnd.oci.image.layer.v1.tar+gzip 10003284

@agners
Copy link
Copy Markdown
Member

agners commented Feb 4, 2026

@sairon Can you please check if all supported versions are supporting it? CC @agners

I did some analysis in the past, see home-assistant/builder#245 (comment). Meanwhile we declared old installations as unsupported and bumped the minimal Docker version (with home-assistant/supervisor#6178), so I think from a support standpoint we are good.

@edenhaus
Copy link
Copy Markdown
Member

edenhaus commented Feb 4, 2026

Created to test images, where I compared layer by layer, both with the command above

Image zstd compression:

Digest: b6ae8fd00c0a80b292a83e96eeffdcd775fbb3a1351f6fcadadaa9c966c3b2f6
Total size: 578,148,244 bytes (578.1 MB / 551.3 MiB)

Image gzip compression

Digest: 69a3f673ba95e0ad15eefa0051a996aea5e2438e8562af679f4ec705cb4c444b
Total size: 737,520,651 bytes (737.5 MB / 703.5 MiB)

Comparison

Layer zstd gzip Winner Difference
1 3,553,881 3,802,452 zstd -248,571 (-6.5%)
2 93 110 zstd -17 (-15.5%)
3 12,583,982 15,335,350 zstd -2,751,368 (-17.9%)
4 247 248 zstd -1 (-0.4%)
5 14,762,340 22,962,847 zstd -8,200,507 (-35.7%)
6 2,904,158 3,779,753 zstd -875,595 (-23.2%)
7 51,300,626 59,717,000 zstd -8,416,374 (-14.1%)
8 5,306,358 8,149,426 zstd -2,843,068 (-34.9%)
9 16 32 zstd -16 (-50.0%)
10 58,235 60,712 zstd -2,477 (-4.1%)
11 465,428 550,413 zstd -84,985 (-15.4%)
12 185 223 zstd -38 (-17.0%)
13 5,332,839 5,377,796 zstd -44,957 (-0.8%)
14 235,736 284,047 zstd -48,311 (-17.0%)
15 8,418 8,974 zstd -556 (-6.2%)
16 1,485 1,495 zstd -10 (-0.7%)
17 6,985,412 7,685,760 zstd -700,348 (-9.1%)
18 18,457,545 23,477,654 zstd -5,020,109 (-21.4%)
19 16 32 zstd -16 (-50.0%)
20 927 914 gzip +13 (+1.4%)
21 3,186 3,192 zstd -6 (-0.2%)
22 62,514,715 74,885,418 zstd -12,370,703 (-16.5%)
23 14,761 15,046 zstd -285 (-1.9%)
24 377,640,479 480,542,839 zstd -102,902,360 (-21.4%)
25 22,592,391 30,190,434 zstd -7,598,043 (-25.2%)
26 17,560,291 24,630,418 zstd -7,070,127 (-28.7%)
27 80 96 zstd -16 (-16.7%)

Only on one layer, gzip has the smaller output.
Size difference: 159,372,407 bytes (159.4 MB / 152.0 MiB)
Compression savings (zstd vs gzip): 21.6% smaller with zstd

Buildtime

ARM64

Before: 3m 9s = 189 seconds https://github.com/home-assistant/core/actions/runs/21666771682/job/62464491628
After: 3m 44s = 224 seconds https://github.com/home-assistant/core/actions/runs/21666776309/job/62464530267
Increase: 35 seconds
Percentage increase: +18.5%

AMD64

Before: 3m 6s = 186 seconds https://github.com/home-assistant/core/actions/runs/21666771682/job/62464491609
After: 3m 40s = 220 seconds https://github.com/home-assistant/core/actions/runs/21666776309/job/62464530306
Increase: 34 seconds
Percentage increase: +18.3%

Copy link
Copy Markdown
Member

@agners agners left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only thing I could think of is memory usage. Lower end hardware might not have much memory. But afaik, zstd default levels (which 9 is part of) use a 8MB window size, so it should be fairly low risk.

LGTM!

@edenhaus
Copy link
Copy Markdown
Member

edenhaus commented Feb 4, 2026

Will do another test with different zstd level to see if we really need level 9

@edenhaus
Copy link
Copy Markdown
Member

edenhaus commented Feb 4, 2026

Complete Comparison Table

Compression AMD64 Size AMD64 Build vs gzip Size vs gzip Time
zstd default 695.6 MB 2m 35s -5.7% ✓ -16.7% ✓
gzip 737.5 MB 3m 6s baseline baseline
zstd level 9 578.1 MB 3m 40s -21.6% ✓ +18.3%

We stick to zstd-9 as it will save another 117MB of image size even when the build process takes 18%. As the image building currently takes less than 5 min, the additional 117MB savings have more benefit than a faster build time.

Copy link
Copy Markdown
Member

@edenhaus edenhaus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @duhow 👍

@edenhaus edenhaus changed the title Build container image with zstd Compress container image with zstd Feb 4, 2026
@edenhaus edenhaus merged commit fcd0b57 into home-assistant:dev Feb 4, 2026
46 checks passed
@github-actions github-actions bot locked and limited conversation to collaborators Feb 5, 2026
@duhow duhow deleted the docker-zstd-upstream branch March 7, 2026 23:24
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

breaking-change cla-signed small-pr PRs with less than 30 lines.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants