Hello!
We are currently experimenting with using multi-arch images in our infrastructure, as we have a mix of IBM Power and regular AMD64-based machines, and are currently using different tags/repositories to distinguish between the architectures, which is not ideal.
We found that the garbage-collect command with the -m parameter on the registry removes the manifests that enable us to use multi-arch, thus rendering us unable to run garbage collection and keep our registry somewhat clean and space-efficient.
I have compiled the following script which shows the behaviour. Beware that it runs docker system prune --all --volumes:
# Remove old registry, clean slate
docker rm -f registry
docker system prune --all --volumes -f
# Start registry
docker run -d -p 5000:5000 --name registry registry
# Pull AMD64-image
docker pull "docker"
docker tag "docker" "localhost:5000/amd64/docker:latest"
docker push "localhost:5000/amd64/docker:latest"
# ARM-image
docker pull "ppc64le/docker"
docker tag "ppc64le/docker" "localhost:5000/ppc64le/docker:latest"
docker push "localhost:5000/ppc64le/docker:latest"
#Create and push manifest for multi-arch ubuntu
docker manifest create --insecure "localhost:5000/docker:latest" "localhost:5000/amd64/docker:latest" "localhost:5000/ppc64le/docker:latest"
docker manifest push -p "localhost:5000/docker:latest"
# Clear local cache
docker system prune --all -f
# Garbage-collect registry
docker exec registry registry garbage-collect --delete-untagged /etc/docker/registry/config.yml
# Pull multi-arch image
docker pull "localhost:5000/docker:latest"
The last couple of lines of output on my machine is:
Total reclaimed space: 439.4MB
amd64/docker
amd64/docker: marking manifest sha256:5782329618eaf703204c70f61c1697a428275bc8d8e5a58fa324798e78e31937
amd64/docker: marking blob sha256:61d75b305666b5c672c2dc789c526f405402c3d37eb44a77470e9217790b9a41
amd64/docker: marking blob sha256:530afca65e2ea04227630ae746e0c85b2bd1a179379cbf2b6501b49c4cab2ccc
amd64/docker: marking blob sha256:33702c1843d19cf7c37af730bcf50c456ac6456ed789053432b000db75d3bed3
amd64/docker: marking blob sha256:d1c203384d5b9b22606055f5a6708153653df701158a6c04748cb77be8238c9e
amd64/docker: marking blob sha256:146feb07c33136aba6d87c2a8d6882cd4d438d957eaaa8f388f59214f1269bd0
amd64/docker: marking blob sha256:cee6b871713b8386f334ada3f80a6f187b8e9130ce7d69236e34fee9f9d44556
amd64/docker: marking blob sha256:db8946a7c6c2c651e2344e7e9ebb692ff53ae6ab2b108a5f484384e165056beb
amd64/docker: marking blob sha256:649b2db28c4901ee36039690e8086767f9c069ebef2f819f53937774a4b74d6b
amd64/docker: marking blob sha256:e7b044ff4e6b0f04ba45ff9f486263b0741ce55315ac6fb0498aec49a0375ee0
amd64/docker: marking blob sha256:dbd7811a8fce869b1e3e2698db341271fcedfe91a3c90e63b19b243ba4187d91
docker
manifest eligible for deletion: sha256:18d3c182508d401451fe7a0c538dff3686360456b233675b4ddb2371aa9d1608
docker: marking manifest sha256:1abd08c6dc1cea1308a4f3213cdb7d7264548ab21beedbde30aa7eed7f209d2d
docker: marking blob sha256:5782329618eaf703204c70f61c1697a428275bc8d8e5a58fa324798e78e31937
docker: marking blob sha256:18d3c182508d401451fe7a0c538dff3686360456b233675b4ddb2371aa9d1608
manifest eligible for deletion: sha256:5782329618eaf703204c70f61c1697a428275bc8d8e5a58fa324798e78e31937
ppc64le/docker
ppc64le/docker: marking manifest sha256:18d3c182508d401451fe7a0c538dff3686360456b233675b4ddb2371aa9d1608
ppc64le/docker: marking blob sha256:a10903a7995b02bb2bb17f3be3423e93bd2ff6a70618d0d42cfed01639a17ec9
ppc64le/docker: marking blob sha256:d6201b52ea9b908d4d6e950e2a79ded27be48979d6f0f63bd3a57b16b621f188
ppc64le/docker: marking blob sha256:7c5cedde51de1c8ffccd5521fd02fc1efc1cc44ece2d5dccb1e550a65366cd80
ppc64le/docker: marking blob sha256:7ab4b28968b92237adc6438d06c14d4680a3936996176f2cea20d781885af568
ppc64le/docker: marking blob sha256:387f12b774ebb4a88b15e4f754eb9c772618a15230f9c093759496c41f5c6c8f
ppc64le/docker: marking blob sha256:8847e0f1fbcbc029817eb1d930620fd68a65b6bf9dcb73b6d2e96d4e8161ba75
ppc64le/docker: marking blob sha256:b7425929e73f782470c79b835b77e1d923e63be874374d247c1deb271178256f
ppc64le/docker: marking blob sha256:72d865172b780f7e0c045e1d59db4d57be4d6f52d93a27f7d7444edead40260b
time="2022-08-01T13:30:53.876986875Z" level=info msg="deleting manifest: /docker/registry/v2/repositories/docker/_manifests/revisions/sha256/18d3c182508d401451fe7a0c538dff3686360456b233675b4ddb2371aa9d1608" go.version=go1.16.15 instance.id=f87bf88e-d8d2-4dfa-9095-71d90367e1aa service=registry
time="2022-08-01T13:30:53.877150402Z" level=info msg="deleting manifest: /docker/registry/v2/repositories/docker/_manifests/revisions/sha256/5782329618eaf703204c70f61c1697a428275bc8d8e5a58fa324798e78e31937" go.version=go1.16.15 instance.id=f87bf88e-d8d2-4dfa-9095-71d90367e1aa service=registry
21 blobs marked, 0 blobs and 2 manifests eligible for deletion
latest: Pulling from docker
manifest for localhost:5000/docker:latest not found: manifest unknown: manifest unknown
I have confirmed that the issue is present on my workstation, which is running:
docker version:
Client: Docker Engine - Community
Version: 20.10.17
API version: 1.41
Go version: go1.17.11
Git commit: 100c701
Built: Mon Jun 6 23:02:57 2022
OS/Arch: linux/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.17
API version: 1.41 (minimum version 1.12)
Go version: go1.17.11
Git commit: a89b842
Built: Mon Jun 6 23:01:03 2022
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.6
GitCommit: 10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
runc:
Version: 1.1.2
GitCommit: v1.1.2-0-ga916309
docker-init:
Version: 0.19.0
GitCommit: de40ad0
docker info:
Client:
Context: default
Debug Mode: false
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Docker Buildx (Docker Inc., v0.8.2-docker)
scan: Docker Scan (Docker Inc., v0.17.0)
Server:
Containers: 5
Running: 5
Paused: 0
Stopped: 0
Images: 27
Server Version: 20.10.17
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
runc version: v1.1.2-0-ga916309
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: default
cgroupns
Kernel Version: 5.15.0-41-generic
Operating System: Ubuntu 22.04 LTS
OSType: linux
Architecture: x86_64
CPUs: 12
Total Memory: 31.19GiB
Name: FLPC8286
ID: OF2W:RJAA:ZPBP:WO6M:H4GP:GDZC:R5PE:PARX:O3CO:2BPW:CM4U:KJHB
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
docker exec registry registry --version:
registry github.com/docker/distribution v2.8.1+unknown
Hello!
We are currently experimenting with using multi-arch images in our infrastructure, as we have a mix of IBM Power and regular AMD64-based machines, and are currently using different tags/repositories to distinguish between the architectures, which is not ideal.
We found that the garbage-collect command with the -m parameter on the registry removes the manifests that enable us to use multi-arch, thus rendering us unable to run garbage collection and keep our registry somewhat clean and space-efficient.
I have compiled the following script which shows the behaviour. Beware that it runs
docker system prune --all --volumes:The last couple of lines of output on my machine is:
I have confirmed that the issue is present on my workstation, which is running:
docker version:
docker info:
docker exec registry registry --version: