dockerd neglects to populate OCI Descriptor size field #46641
Description
As background, I originally reported this issue in the buildkit repo: moby/buildkit#4328
Since that issue was closed yesterday I felt a bit crazy because on the one hand I can clearly see in my WIP registry implementation that manifests pushed to it by dockerd definitely do not contain the size field in the descriptors as required by the OCI image spec, yet on the other hand the issue doesn't seem reproducible pushing images to a local copy of distribution.
@neersighted mentioned I should open a PR here if I can prove that the size field is missing from manifests on the wire during a PUT /v2/<name>/manifests/<reference> call so that's what I'm doing.
Here are the details of my setup:
- my WIP registry (named portfolio) running on localhost:13030
- a local build of distribution running on localhost:5000 with the following config:
version: 0.1
log:
fields:
service: registry
storage:
cache:
blobdescriptor: inmemory
filesystem:
rootdirectory: ./var/lib/registry
http:
addr: :5001
headers:
X-Content-Type-Options: [nosniff]
health:
storagedriver:
enabled: true
interval: 10s
threshold: 3- mitmdump in reverse proxy mode listening at localhost:8080 pointed at localhost:13030 (portfolio):
mitmdump --mode reverse:http://127.0.0.1:13030@8080 --flow-detail 4 > portfolio.mitmproxy.flow- mitmdump in reverse proxy mode listening at localhost:8081 pointed at localhost:5000 (distribution)
mitmdump --mode reverse:http://127.0.0.1:5000@8081 --flow-detail 4 > distribution.mitmproxy.flowWith this setup I run the following docker build commands:
# aiming at the distribution mitmdump instance
docker buildx build --push ./ -t 127.0.0.1:8081/woof:meowmeow --no-cache
# aiming at the portfolio mitmdump instance
docker buildx build --push ./ -t 127.0.0.1:8080/woof:meowmeow --no-cache(I will upload a tarball with the resulting logs after submitting this issue since I am not using a web browser to write this description)
The relevant snippet of mitmdump output for portfolio (note the missing size filed in the config descriptor):
[08:27:34.419][127.0.0.1:39808] client connect
[08:27:34.421][127.0.0.1:39808] server connect 127.0.0.1:13030
127.0.0.1:39808: PUT http://127.0.0.1:13030/v2/woof/manifests/meowmeow
Host: 127.0.0.1:13030
User-Agent: docker/24.0.6 go/go1.21.1 git-commit/1a7969545d kernel/6.1.54-1-lts os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)
Content-Length: 714
Content-Type: application/vnd.docker.distribution.manifest.v2+json
Accept-Encoding: gzip
Connection: close
{
"config": {
"digest": "sha256:7bddfd52170f7ebde74e065ec731299f182d5fa332bbfbe8cd07331031b75ab9",
"mediaType": "application/vnd.docker.container.image.v1+json"
},
"layers": [
{
"digest": "sha256:96526aa774ef0126ad0fe9e9a95764c5fc37f409ab9e97021e7b4775d82bf6fa",
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 3401967
},
{
"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1",
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 32
}
],
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"schemaVersion": 2
}
<< 201 Created 0b
content-type: text/plain; charset=utf-8
location: /v2/woof/manifests/meowmeow
docker-content-digest: sha256:5d3183be0559955ce465d519e0d6a6803249e20d1d7291febcb9b9ac3e5f1ccf
docker-distribution-api-version: registry/2.0
content-length: 0
date: Fri, 13 Oct 2023 14:27:34 GMT
[08:27:34.538][127.0.0.1:39808] server disconnect 127.0.0.1:13030
[08:27:34.539][127.0.0.1:39808] client disconnect
The relevant snippet of mitmdump output for distribution (note the presence of the config.size field here, in contrast with the above snippet):
[08:49:46.135][127.0.0.1:37552] client connect
[08:49:46.136][127.0.0.1:37552] server connect 127.0.0.1:5000
127.0.0.1:37552: PUT http://127.0.0.1:5000/v2/woof/manifests/meowmeow
Host: 127.0.0.1:5000
User-Agent: docker/24.0.6 go/go1.21.1 git-commit/1a7969545d kernel/6.1.54-1-lts os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)
Content-Length: 733
Content-Type: application/vnd.docker.distribution.manifest.v2+json
Accept-Encoding: gzip
Connection: close
{
"config": {
"digest": "sha256:e94c3024986a16ece2ea2d57034d97c91bf88662316c882f3ffdf8527010b6db",
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": 816
},
"layers": [
{
"digest": "sha256:96526aa774ef0126ad0fe9e9a95764c5fc37f409ab9e97021e7b4775d82bf6fa",
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 3401967
},
{
"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1",
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 32
}
],
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"schemaVersion": 2
}
<< 201 Created 0b
Docker-Content-Digest: sha256:c68777b0ce58b075d8113abb5c9285599d6a203cbd9b9a75babfd486a07e7083
Docker-Distribution-Api-Version: registry/2.0
Location: http://127.0.0.1:5000/v2/woof/manifests/sha256:c68777b0ce58b075d8113abb5c9285599d6a203cbd9b9a75babfd486a07e7083
X-Content-Type-Options: nosniff
Date: Fri, 13 Oct 2023 14:49:46 GMT
Content-Length: 0
Connection: close
[08:49:46.151][127.0.0.1:37552] server disconnect 127.0.0.1:5000
[08:49:46.151][127.0.0.1:37552] client disconnect
I believe that if I build the image without cached layers and push it first to my WIP registry I can get a manifest where the layer descriptors also don't include the size field, but the above logs should be enough to establish the fact that this is happening (and i'm not crazy 😄)
For what it's worth, I would like to share my WIP registry so others can more easily reproduce this but the dev env setup is somewhat complicated by the need for a postgresql wire compatible DB and a live S3 compatible API. I'll eventually implement support for sqlite metadata storage and local file system bulk data storage but that could be a while.
In the short term I intend to work on a cloud deployment of this that I will be happy to share with maintainers who want to see the issue for themselves. (probably won't be ready until next week)
I've also noticed some differences between distribution and my implementation in terms of the /v2/ response (eg distribution includes an empty json in the response body, mine doesn't) and various headers. I suspect that if I align the behavior of my registry more closely to distribution this problem might go away.
But I'm reporting this bug anyway because it does definitely seem like a bug somewhere in dockerd or one of the registry client libraries it might be using to exclude the descriptor size field under any circumstance.