Skip to content

Breaking/Unexpected docker push change in v29/containerd image store #51532

@csuich2

Description

@csuich2

Description

After our build system was updated to docker-ce v29 we noticed that some of our images were being pushed as oci manifests rather than single images when pushed with docker push docker.io/foo:x.y.z. The change to containerd as the default storage seems to be the culprit. We verified this behavior with a default install of v29 and v28 with containerd for storage. We also verified that adding --platform linux/amd64 to the push command restores single image (non-manifest) behavior.

This is breaking our KinD environments on arm Macs because KinD can run amd64 images, but not manifests with a single arm64 image in them.

The switch to containerd as the default storage engine is called out in the v29 release notes, but I believe this default behavior change is also a breaking change.

If this behavior change was intentional, I believe it should be called out in the release notes.

If this behavior change was not intentional, then this appears to be a bug.

To summarize:

  • Running docker push docker.io/foo:x.y.z without containerd image store results in a single image being pushed.
  • Running docker push docker.io/foo:x.y.z with containerd image store results in a manifest with a single architecture being pushed.
  • Running docker push --platform linux/amd64 docker.io/foo:x.y.z with containerd image store results in a single image being pushed.

This appears to be a breaking change that was not explicitly called out in the v29 release notes.

Reproduce

  1. Install docker-ce v28
  2. docker build --platform linux/amd64 -t myregistry.com/docker-test:v28 . with a basic Dockerfile
  3. docker push myregistry.com/docker-test:v28 to your registry
  4. docker manifest inspect myregistry.com/docker-test:v28
    • Note that this is a single image with all layers listed
  5. Completely uninstall docker-ce v28
    • Also rm -rf /var/lib/docker ~/.docker
  6. Install docker-ce v29
  7. docker build --platform linux/amd64 -t myregistry.com/docker-test:v29 . with a basic Dockerfile
  8. docker push myregistry.com/docker-test:v29 to your registry
  9. docker manifest inspect myregistry.com/docker-test:v29
    • Note that this is a manifest with a single platform image in it
  10. docker build --platform linux/amd64 -t myregistry.com/docker-test:v29.1 . with a basic Dockerfile
  11. docker push --platform linux/amd64 myregistry.com/docker-test:v29.1 to your registry
  12. docker manifest inspect myregistry.com/docker-test:v29.1
    • Note that this is a single image with all layers listed

Expected behavior

Running docker push without the --platform flag should result in a single image being pushed, rather than the breaking behavior change.

docker version

Client: Docker Engine - Community
 Version:           28.3.3
 API version:       1.51
 Go version:        go1.24.5
 Git commit:        980b856
 Built:             Fri Jul 25 11:34:09 2025
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          28.3.3
  API version:      1.51 (minimum version 1.24)
  Go version:       go1.24.5
  Git commit:       bea959c
  Built:            Fri Jul 25 11:34:09 2025
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v2.1.5
  GitCommit:        fcd43222d6b07379a4be9786bda52438f0dd16a1
 runc:
  Version:          1.3.3
  GitCommit:        v1.3.3-0-gd842d771
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client: Docker Engine - Community
 Version:    28.3.3
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.29.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.40.3
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 3
 Server Version: 28.3.3
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: fcd43222d6b07379a4be9786bda52438f0dd16a1
 runc version: v1.3.3-0-gd842d771
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.8.0-79-generic
 Operating System: Ubuntu 24.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 6
 Total Memory: 15.62GiB
 Name: csuich-jump
 ID: 7697ad96-f98b-486d-bf90-4bf74c3da0e0
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false

Additional Info

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    containerd-integrationIssues and PRs related to containerd integrationkind/bugBugs are bugs. The cause may or may not be known at triage time so debugging may be needed.status/0-triageversion/29.0

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions