image manifests built with buildkit missing required size field in manifest #4328
Description
I have been working on a new implementation of the distribution spec using https://github.com/containers/oci-spec-rs and when attempting to push my first image, my implementation was rejecting the manifest as invalid. When I looked at my logs I saw
2023-10-12T17:18:37.639919Z WARN request{method=PUT uri=/v2/woof/manifests/0.0.30 version=HTTP/1.1 headers={"host": "localhost:13030", "user-agent": "docker/24.0.5 go/go1.20.6 git-commit/a61e2b4c9c kernel/6.1.54-1-lts os/linux arch/amd64 UpstreamClient(Docker-Client/24.0.5
\(linux\))", "content-length": "2147", "content-type": "application/vnd.docker.distribution.manifest.v2+json", "accept-encoding": "gzip", "connection": "close"}}: portfolio::http::manifests: error deserializing manifest: DistributionSpecError(ManifestInvalid)
(here you can see the user agent is docker/24.0.5, @neersighted mentioned to me earlier today in an OCI Dev call that this version of docker uses buildkit by default)
This error originates with the fact that the type I'm trying to de-serialize the manifest to strictly requires that all descriptors include the size field: https://github.com/containers/oci-spec-rs/blob/833976fc161ee52588d24ac8446c44ed40222006/src/image/descriptor.rs#L34-L40
This requirement is codified in the OCI Image spec here:
This REQUIRED property specifies the size, in bytes, of the raw content.
To be clear, this descriptor without a size is occurring in the image manifest's config field:
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"digest": "sha256:53eff974406dd16e9f7a083eb7da839181f0999d9277bb39891c2612ff78fe84"
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"digest": "sha256:ac7f2e1c758675427623d0da4faa88b336c62466c15a98af61efd3f015282f2f"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"digest": "sha256:730f917270d3956a2c03b6fa96c905b62fbd5d584222e1aaf4441e24569a08a9"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"digest": "sha256:4a4650a2e572ca165a0add1b9a2edc5bbfe5e2d5d85f126eb137cc0fedf43198"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"digest": "sha256:8484ee9d14b6f13c8bfd9a1303032514fe5111b8a208a3691d5b71c58becbead"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"digest": "sha256:11597adb7c4e95f1928e7c7ecc752f20795bfaa7f3b697600770f97c82048a11"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"digest": "sha256:058473d14b1d7d9fbb13b37e8d66a0a85791c65817bd615469d6c81a453d42f1"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"digest": "sha256:793a3dfcbd4a4772c862357be99650cdaecaca972f125ee609b32dd158204286"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"digest": "sha256:42a9a986bf59d4a4d222fe4c65bceff269199bdba365df51d760639f6eacba21"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"digest": "sha256:29a525d13ec607f89fea2f523ea26e8e27869d832c19925df286fefc7053a289"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"digest": "sha256:a506a09b79cdce6a5877042467efd8664bd469cde03ad35dbe010ab7dd3e2379"
}
]
}