docker plugin create on 20.x version makes unusable plugins for 19.x/18.x/17.x #42191
Description
A plugin created and pulled to hub.docker.com using 20.x version will fail to install on 19-17.x version with error: Layers from manifest don't match image configuration
When building on 20.10.5 version of Docker Engine/Cli a docker volume plugins from project docker-volume-plugins I found that I can't install on Docker Engine 19/18/17 version with an error Layers from manifest don't match image configuration.
Steps to reproduce the issue:
Build a plugin using 20.x version of Docker engine/cli
- docker plugin create mochoa/s3fs-volume-plugin-x86_64:latest build
- docker plugin push mochoa/s3fs-volume-plugin-x86_64:latest
Installing on previous release of Docker engine/cli (for example 19.x)
- docker plugin install --alias s3fs mochoa/s3fs-volume-plugin-x86_64:latest --grant-all-permissions --disable
Above command shows this output:
$ docker plugin install --alias s3fs mochoa/s3fs-volume-plugin-x86_64:test --grant-all-permissions --disable
Trying to pull repository docker.io/mochoa/s3fs-volume-plugin-x86_64 ...
test: Pulling from docker.io/mochoa/s3fs-volume-plugin-x86_64
7cb61fa08378: Download complete
layers from manifest don't match image configurationIf same plugin is built using Docker engine/cli 19.x version it works OK:
$ docker plugin install --alias s3fs mochoa/s3fs-volume-plugin-x86_64:latest --grant-all-permissions --disable
Trying to pull repository docker.io/mochoa/s3fs-volume-plugin-x86_64 ...
latest: Pulling from docker.io/mochoa/s3fs-volume-plugin-x86_64
6adc36b99bc7: Download complete
Digest: sha256:7131b6621c320a08ae61aa95ecc8d932997c1e48dbff4dae43708e62b37d906d
Status: Downloaded newer image for mochoa/s3fs-volume-plugin-x86_64:latest
Installed plugin mochoa/s3fs-volume-plugin-x86_64:latestAdditional information you deem important (e.g. issue happens only occasionally):
Output of docker version:
Client: Docker Engine - Community
Version: 20.10.5
API version: 1.41
Go version: go1.13.15
Git commit: 55c4c88
Built: Tue Mar 2 20:18:20 2021
OS/Arch: linux/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.5
API version: 1.41 (minimum version 1.12)
Go version: go1.13.15
Git commit: 363e9a8
Built: Tue Mar 2 20:16:15 2021
OS/Arch: linux/amd64
Experimental: true
containerd:
Version: 1.4.4
GitCommit: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
runc:
Version: 1.0.0-rc93
GitCommit: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
docker-init:
Version: 0.19.0
GitCommit: de40ad0
Output of docker info:
Client:
Context: default
Debug Mode: false
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Build with BuildKit (Docker Inc., v0.4.2)
Server:
Containers: 10
Running: 0
Paused: 0
Stopped: 10
Images: 619
Server Version: 20.10.5
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: active
NodeID: mnzsbflpwe41140g27i79rd7b
Is Manager: true
ClusterID: wo4mmgys6wxs20w2aiw9kqaa1
Managers: 1
Nodes: 1
Default Address Pool: 10.0.0.0/8
SubnetSize: 24
Data Path Port: 4789
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 10
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: 172.29.160.34
Manager Addresses:
172.29.160.34:2377
Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux
Default Runtime: runc
Init Binary: docker-init
containerd version: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 5.4.0-67-generic
Operating System: Ubuntu 20.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.51GiB
Name: pocho
ID: C3WY:YORD:TA4B:M6L5:JAJL:OKAU:5E4H:2QSP:Z3VM:FBYA:AJRO:RBZI
Docker Root Dir: /var/lib/docker
Debug Mode: false
Username: mochoa
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Manifest of plugin created with 20.x:
{
"schemaVersion": 2,
"config": {
"mediaType": "application/vnd.docker.plugin.v1+json",
"digest": "sha256:03a44636bcdb1b661261d654e732b44d0fb15396e0e037ed6099033baedd2347",
"size": 1003
},
"layers": [
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:7cb61fa083786e446f1032b9798b8e3d08f086cfa0dac3b55873f0f1a2b46e18",
"size": 55932817
}
],
"mediaType": "application/vnd.docker.distribution.manifest.v2+json"
}
Manifest of plugin created with 19.x:
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.plugin.v1+json",
"size": 1004,
"digest": "sha256:7032a5148630a0052372c8785ca23d0449a6da3d4910813af741278d2953c80b"
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 55932830,
"digest": "sha256:6adc36b99bc70203bd542a473aa0f76fca363b14e4b08565558be60682162a3c"
}
]
}
Difference on mediaType:
- "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip"
- "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip"
Plugins created with 20.x version works OK on 20.x engine.
Complete discussion is at Slack channel # docker-storage thanks a lot @cpuguy83 for your time.