Add distroless Docker image variant by roidelapluie · Pull Request #17876 · prometheus/prometheus

roidelapluie · 2026-01-16T14:29:18Z

Introduces distroless image using UID/GID 65532 instead of nobody, and removes VOLUME declaration. Busybox image remains default with unchanged tags for backwards compatibility.

Which issue(s) does the PR fix:

Fixes #8657
Closes #17680

Relates to #16048

Does this PR introduce a user-facing change?

For the top of the CHANGELOG and the release messages, I would add this message:

Prometheus now offers a distroless Docker image variant alongside the default
busybox image. The distroless variant provides enhanced security with a minimal
base image, uses UID/GID 65532 (nonroot) instead of nobody, and removes the
VOLUME declaration. Both variants are available with `-busybox` and `-distroless`
tag suffixes (e.g., `prom/prometheus:latest-busybox`, `prom/prometheus:latest-distroless`).
The busybox image remains the default with no suffix for backwards compatibility
(e.g., `prom/prometheus:latest` points to the busybox variant).

For users migrating existing **named** volumes from the busybox image to the distroless variant, the ownership can be adjusted with:
```
docker run --rm -v prometheus-data:/prometheus alpine chown -R 65532:65532 /prometheus
```
Then, the container can be started with the old volume with:
```
docker run -v prometheus-data:/prometheus prom/prometheus:latest-distroless
```
User migrating from bind mounts might need to ajust permissions too, depending on their setup.

[FEATURE] Dockerfile: Add distroless image variant using UID/GID 65532 and no VOLUME declaration. Busybox image remains default.

Dockerfile.distroless

roidelapluie · 2026-01-16T14:48:49Z

Note: This should work with CI as well.

polarathene · 2026-01-16T23:53:22Z

I think you can avoid a bunch of complexity if adopting Docker Bake is acceptable for the project?

It's effectively a config syntax in HCL that maps into the Docker CLI build command + options. That allows for easily declaring two images and their tags, then you can use the Github Actions support if you like, or docker bake to build (optionally with any configurables to pass into image Dockerfiles).

If that sounds appealing to you, let me know and I can provide some examples.

polarathene

Turns out that this Dockerfile feedback pretty much applies to the existing Dockerfile in the project too, which due to it's USER nobody is likely affected by this concern too (not an issue for distroless that sets USER with UID/GID values).

Have you tested this Dockerfile.distroless build? Won't it have ownership issues? The primary Dockerfile corrects these before switching to a non-root user:

prometheus/Dockerfile

Line 24 in c4b0da9

RUN chown -R nobody:nobody /etc/prometheus /prometheus && chmod g+w /prometheus

You'd need to do the equivalent I believe and then switch back to nonroot, but ensure you do this with the UID/GID for broader compatibility. You may want to include a comment about that distinction to avoid anyone mistakenly changing it as an "improvement".

Since you're differing in ownership, you'd also be wise to more clearly document this difference (there's related markdown docs on this project, but you should possibly also highlight this on the page descriptions at the registries the image is published to).

Your COPY + WORKDIR are probably also with the existing USER context, that minimizes the need to apply some of the corrections, but technically /bin placed files should be root:root 😅 (and presumably chmod g+w /prometheus is still relevant? better reasoning explained here)

EDIT: Regardless of USER the COPY instruction appears to default to root ownership unless using --chown. WORKDIR will use USER if it needs to create the directory. The chmod g+w can be handled with COPY --chmod=775 (requires BuildKit container driver rather than docker build driver, otherwise seems to be ignored).

For the /prometheus location, the UID/GID will be used for new anonymous/named volume mounts, but bind volume mounts will replace it entirely.

This can matter when your container process needs to write to persisted volumes, such as with /prometheus which is probably why it's advising a named data volume (or uses the implicit anonymous data volume in the primary Dockerfile), since there's no docs there mentioning the expectation for matching the containers UID/GID.

Introducing this image variant with a different USER mapping will not be compatible to change between images without going into the container and changing that volumes ownership first, which would need to be done with an intermediary container in this case. You may want to cover this in docs for migrating between image variants as something for users to be mindful of.

I personally find it better to make nonroot use optional/opt-in, since it can come with various caveats (k8s example, OpenShift is another one IIRC). More often than not those interested in nonroot can get many benefits from more explicit configuration (like dropping all capabilities) or running a container rootless (eg: Docker rootless or default for Podman run as a non-root user).

Dockerfile.distroless

roidelapluie · 2026-01-19T08:33:53Z

Thanks for the detailed feedback @polarathene!

Regarding ownership issues:

I've tested the distroless image and it works correctly. The COPY instructions run as root (which sets proper ownership for /bin files), while WORKDIR runs in the context of the current user (65532:65532) -- that's all that is needed for Prometheus to run. No need for the chmod etc. This doesn't cause permission problems in practice

Regarding running as root:

I disagree with making nonroot optional/opt-in. The entire goal of this distroless variant is to increase security, not diminish it. We're not changing the default busybox image - users who need or want the traditional nobody-based image can continue using it. The distroless variant is explicitly for users who want enhanced security by default. And it would be default for Prometheus 4.

Regarding migation

Users explicitly choosing the distroless variant will be making a conscious decision to adopt a different security model. I expect them to pay attention to the details and read the release notes which specify ownership changes.

Regarding the build system:

Makefile.common is integrated across all prometheus/*, prometheus-community repositories, and probably beyond that. For now, I'm intentionally keeping the standard Dockerfile approach so this pattern can be adopted seamlessly by other repos. Changing the build system or modifying the main default image is out of scope - the busybox image is well-known and broadly used.

Makefile.common

Dockerfile.distroless

roidelapluie · 2026-01-19T11:01:12Z

I believe the logic is cleaner now:

100% backwards compatible - if no docker variant is set, it should work OOTB and not push extra tags
We error if Dockerfile is set
In prometheus/prometheus, would push: distroless and busybox.

Makefile.common

Introduces distroless image using UID/GID 65532 instead of nobody, and removes VOLUME declaration. Busybox image remains default with unchanged tags for backwards compatibility. Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>

SuperQ · 2026-01-19T13:48:17Z

Nice

polarathene · 2026-01-19T21:31:08Z

Quite a bit of activity while I was away and unavailable to chime in 😅 Seems I'm a bit too late.

No need for the chmod etc. This doesn't cause permission problems in practice
I expect them to pay attention to the details and read the release notes which specify ownership changes.

Well you had it in the other image for a reason 🤷‍♂️ Might as well try be consistent?

Perhaps I'm one of the few that aren't fond of troubleshooting projects when undocumented differences cause different/unexpected compatibility issues?

Will it be documented alongside the difference in UID/GID? I would hope that that kind of information isn't going to be hidden away in a changelog only, since that'd prevent any new users adopting the images from knowing any better.

Regarding running as root:

I disagree with making nonroot optional/opt-in. The entire goal of this distroless variant is to increase security, not diminish it. We're not changing the default busybox image - users who need or want the traditional nobody-based image can continue using it. The distroless variant is explicitly for users who want enhanced security by default. And it would be default for Prometheus 4.

Regarding migation

Users explicitly choosing the distroless variant will be making a conscious decision to adopt a different security model. I expect them to pay attention to the details and read the release notes which specify ownership changes.

What is the drawback to being consistent by having the distroless variant use the nobody user/group instead of nonroot? If the container was actually making use of different user/group boundaries, I could understand wanting to avoid nobody but I'm not familiar with why it'd be an issue with your use-case?

In the event of a rootful container escape, the nobody user/group would align with that UID/GID on the host. I haven't looked into the decision of the UID/GID for nonroot in distroless, but presumably it was either to try avoid conflicting with a host UID/GID, or to intentionally avoid that scenario with nobody access (rare that it'd be assigned to user/group ownership meaningfully vs the other permission octal).

While for a rootless container (irrelevant to what is run in the container itself), the ID would map to a value outside the host range for users by exceeding 2^16, via offsets in /etc/subuid//etc/subgid.

You still get the perks of non-root user in a container with nobody that you're interested in. Seems a tad odd that you'd not want to align with your existing image UID/GID usage of nobody 🤷‍♂️

Regarding the build system:

Makefile.common is integrated across all prometheus/*, prometheus-community repositories, and probably beyond that. For now, I'm intentionally keeping the standard Dockerfile approach so this pattern can be adopted seamlessly by other repos. Changing the build system or modifying the main default image is out of scope - the busybox image is well-known and broadly used.

I have a WIP example I'll share/PR once I've had time to wrap it up and polish it. You can reject that if you like but it wouldn't interfere with the concerns you've expressed that'd negatively impact anyone.

I did however in the interest of minimizing noise/complexity, unify both busybox and distroless variants into a single Dockerfile with both using nobody as the user/group. It simplifies the support greatly, but it may be too late now given this PR went through quite a bit of activity while I was asleep and has already merged before I could share anymore input 😓

I get the impression my proposal will be rejected by this project, but perhaps there will still be some value that you can extract from it.

codesome · 2026-02-18T17:28:19Z

@roidelapluie this is blocking the v3.10 release as the CI is failing at this step https://github.com/prometheus/prometheus/actions/runs/22120978450/job/63963288131

ERROR: failed to build: failed to solve: gcr.io/distroless/static-debian13:nonroot-riscv64: failed to resolve source metadata for gcr.io/distroless/static-debian13:nonroot-riscv64: gcr.io/distroless/static-debian13:nonroot-riscv64: not found

I will look at mitigating this. Maybe will skip this image that does not exist.

codesome · 2026-02-18T22:14:34Z

#18115 removes riscv64 from distroless

roidelapluie requested review from SuperQ and simonpasquier as code owners January 16, 2026 14:29

github-advanced-security bot found potential problems Jan 16, 2026

View reviewed changes

Dockerfile.distroless Fixed Show fixed Hide fixed

roidelapluie force-pushed the roidelapluie/distroless branch 2 times, most recently from c340495 to 5dcf33a Compare January 16, 2026 14:35

roidelapluie mentioned this pull request Jan 16, 2026

docker: add distroless container build #17680

Closed

roidelapluie mentioned this pull request Jan 16, 2026

Remove usage of the VOLUME instruction #16048

Open

polarathene reviewed Jan 17, 2026

View reviewed changes

Dockerfile.distroless Outdated Show resolved Hide resolved

Dockerfile.distroless Show resolved Hide resolved

Dockerfile.distroless Outdated Show resolved Hide resolved

roidelapluie force-pushed the roidelapluie/distroless branch 2 times, most recently from e56c98a to 8ef71e1 Compare January 19, 2026 08:36