c8d: Daemon crashes when using ImageLoad #46746
Description
Description
Found by @natalieparellano when testing containerd beta. https://github.com/buildpacks/imgutil has this undocumented fast path for ImageLoad. The idea, I believe, is that if we think the base layers should exist in the daemon we can send a 0 size tar header for those layers and the overlay storage system would end up finding the referenced layers by diffid I believe. The new containerd storage layer hard crashes the daemon with an EOF.
Reproduce
git clone https://github.com/buildpacks/imgutil
cd imgutil
git checkout bug/crash-docker-containerd
go test -count=1 -v ./acceptance/reproducibility_test.goResult:
Dockerd panic: [2023-10-30T19:05:44.292642262Z][dockerd][I] panic: runtime error: invalid memory address or nil pointer dereference
[2023-10-30T19:05:44.292697769Z][dockerd][I] [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x2add536]
[2023-10-30T19:05:44.292705699Z][dockerd][I]
[2023-10-30T19:05:44.292705963Z][dockerd][I] goroutine 297251 [running]:
[2023-10-30T19:05:44.292711163Z][dockerd][I] io.(*PipeWriter).Write(...)
[2023-10-30T19:05:44.292711417Z][dockerd][I] /usr/local/go/src/io/pipe.go:165
[2023-10-30T19:05:44.292713012Z][dockerd][I] github.com/containerd/containerd/remotes/docker.(*pushWriter).Commit(0xc001565570, {0x0?, 0xc34e40?}, 0x0, {0xc017f28a00, 0x47}, {0xc1481d821119c3ee?, 0x7a38d2c3f5?, 0x3a605a0?})
[2023-10-30T19:05:44.292714437Z][dockerd][I] /go/src/github.com/docker/docker/vendor/github.com/containerd/containerd/remotes/docker/pusher.go:441 +0x76
[2023-10-30T19:05:44.292716408Z][dockerd][I] github.com/containerd/containerd/content.Copy({0xc5cac0, 0xc018140c30}, {0xc63168, 0xc001565570}, {0xc3d3e0, 0xc0180fe090}, 0x0, {0xc017f28a00, 0x47}, {0x0, ...})
[2023-10-30T19:05:44.292716948Z][dockerd][I] /go/src/github.com/docker/docker/vendor/github.com/containerd/containerd/content/helpers.go:186 +0x3df
[2023-10-30T19:05:44.292718684Z][dockerd][I] github.com/containerd/containerd/remotes.push({0xc5cac0, 0xc018140c30}, {0x7f16504bff18, 0xc018087a70}, {0xc3e9c0?, 0xc0180ce8a0?}, {{0xc017f30990, 0x2c}, {0xc017f28a00, 0x47}, ...})
[2023-10-30T19:05:44.292718930Z][dockerd][I] /go/src/github.com/docker/docker/vendor/github.com/containerd/containerd/remotes/handlers.go:199 +0x5bc
[2023-10-30T19:05:44.292720358Z][dockerd][I] github.com/containerd/containerd/remotes.PushHandler.func1({0xc5ca18, 0xc017f47a90}, {{0xc017f30990, 0x2c}, {0xc017f28a00, 0x47}, 0x0, {0x0, 0x0, 0x0}, ...})
[2023-10-30T19:05:44.292720937Z][dockerd][I] /go/src/github.com/docker/docker/vendor/github.com/containerd/containerd/remotes/handlers.go:166 +0x2a5
[2023-10-30T19:05:44.292722473Z][dockerd][I] github.com/containerd/containerd/images.HandlerFunc.Handle(0xc017f28a00?, {0xc5ca18?, 0xc017f47a90?}, {{0xc017f30990, 0x2c}, {0xc017f28a00, 0x47}, 0x0, {0x0, 0x0, ...}, ...})
[2023-10-30T19:05:44.292722818Z][dockerd][I] /go/src/github.com/docker/docker/vendor/github.com/containerd/containerd/images/handlers.go:59 +0x7f
[2023-10-30T19:05:44.292724433Z][dockerd][I] github.com/containerd/containerd/images.Handlers.func1({0xc5ca18, 0xc017f47a90}, {{0xc017f30990, 0x2c}, {0xc017f28a00, 0x47}, 0x0, {0x0, 0x0, 0x0}, ...})
[2023-10-30T19:05:44.292724687Z][dockerd][I] /go/src/github.com/docker/docker/vendor/github.com/containerd/containerd/images/handlers.go:69 +0x15e
[2023-10-30T19:05:44.292762509Z][dockerd][I] github.com/containerd/containerd/images.HandlerFunc.Handle(0xc3c901?, {0xc5ca18?, 0xc017f47a90?}, {{0xc017f30990, 0x2c}, {0xc017f28a00, 0x47}, 0x0, {0x0, 0x0, ...}, ...})
[2023-10-30T19:05:44.292763387Z][dockerd][I] /go/src/github.com/docker/docker/vendor/github.com/containerd/containerd/images/handlers.go:59 +0x7f
[2023-10-30T19:05:44.292765983Z][dockerd][I] github.com/containerd/containerd/images.Handlers.func1({0xc5ca18, 0xc017f47a90}, {{0xc017f30990, 0x2c}, {0xc017f28a00, 0x47}, 0x0, {0x0, 0x0, 0x0}, ...})
[2023-10-30T19:05:44.292766218Z][dockerd][I] /go/src/github.com/docker/docker/vendor/github.com/containerd/containerd/images/handlers.go:69 +0x15e
[2023-10-30T19:05:44.292767842Z][dockerd][I] github.com/containerd/containerd/images.HandlerFunc.Handle(0xc001121ec8?, {0xc5ca18?, 0xc017f47a90?}, {{0xc017f30990, 0x2c}, {0xc017f28a00, 0x47}, 0x0, {0x0, 0x0, ...}, ...})
[2023-10-30T19:05:44.292768075Z][dockerd][I] /go/src/github.com/docker/docker/vendor/github.com/containerd/containerd/images/handlers.go:59 +0x7f
[2023-10-30T19:05:44.292769615Z][dockerd][I] github.com/containerd/containerd/images.Dispatch.func1()
[2023-10-30T19:05:44.292770438Z][dockerd][I] /go/src/github.com/docker/docker/vendor/github.com/containerd/containerd/images/handlers.go:168 +0xd6
[2023-10-30T19:05:44.292772048Z][dockerd][I] golang.org/x/sync/errgroup.(*Group).Go.func1()
[2023-10-30T19:05:44.292772303Z][dockerd][I] /go/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x64
[2023-10-30T19:05:44.292774110Z][dockerd][I] created by golang.org/x/sync/errgroup.(*Group).Go
[2023-10-30T19:05:44.292774360Z][dockerd][I] /go/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:72 +0xa5
[2023-10-30T19:05:44.344824989Z][dockerd][I] EOF
Expected behavior
I would expect the ImageLoad to fail with an error, allowing the caller to handle the error. In this case imgutil has a slow path for downloading base layers if we fail to load the image into the daemon.
Obviously we would love to keep the faster behavior - but a crash should be resolved if nothing else. #44369 could allow us to correctly probe the daemon storage to see if we should do the work or not.
docker version
Client:
Cloud integration: v1.0.35+desktop.5
Version: 24.0.6
API version: 1.43
Go version: go1.20.7
Git commit: ed223bc
Built: Mon Sep 4 12:28:49 2023
OS/Arch: darwin/amd64
Context: defaultdocker info
Client:
Version: 24.0.6
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.11.2-desktop.5
Path: /Users/jesse.brown/.docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.22.0-desktop.2
Path: /Users/jesse.brown/.docker/cli-plugins/docker-compose
dev: Docker Dev Environments (Docker Inc.)
Version: v0.1.0
Path: /Users/jesse.brown/.docker/cli-plugins/docker-dev
extension: Manages Docker extensions (Docker Inc.)
Version: v0.2.20
Path: /Users/jesse.brown/.docker/cli-plugins/docker-extension
init: Creates Docker-related starter files for your project (Docker Inc.)
Version: v0.1.0-beta.8
Path: /Users/jesse.brown/.docker/cli-plugins/docker-init
sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
Version: 0.6.0
Path: /Users/jesse.brown/.docker/cli-plugins/docker-sbom
scan: Docker Scan (Docker Inc.)
Version: v0.26.0
Path: /Users/jesse.brown/.docker/cli-plugins/docker-scan
scout: Docker Scout (Docker Inc.)
Version: v1.0.7
Path: /Users/jesse.brown/.docker/cli-plugins/docker-scout
Server:Additional Info
No response
Metadata
Metadata
Assignees
Labels
Type
Projects
Status