Skip to content

panic when pulling an image with docker compose #50165

New issue
New issue
Closed
#50135
Closed
panic when pulling an image with docker compose#50165
#50135
Labels
area/imagesImage Distributioncontainerd-integrationIssues and PRs related to containerd integrationkind/bugBugs are bugs. The cause may or may not be known at triage time so debugging may be needed.status/confirmedversion/28.2
@vrischmann

Description

@vrischmann
vrischmann
opened on Jun 10, 2025

Description

Today I've been getting errors while trying to pull images with docker compose on my mac:

unable to get image 'clickhouse/clickhouse-server:24.8-alpine': request returned 502 Bad Gateway for API route and version http://%2Fvar%2Frun%2Fdocker.sock/v1.49/images/clickhouse/clickhouse-server:24.8-alpine/json, check if the server supports the requested API version

The same image pulls fine with docker pull.

Looking at the dockerd log I see it panics:

2025/06/10 15:12:00 http: panic serving @: runtime error: invalid memory address or nil pointer dereference
goroutine 286683 [running]:
net/http.(*conn).serve.func1()
	/usr/local/go/src/net/http/server.go:1947 +0xb0
panic({0xaaaac0e6f400?, 0xaaaac27719e0?})
	/usr/local/go/src/runtime/panic.go:792 +0x124
go.opentelemetry.io/otel/sdk/trace.(*recordingSpan).End.deferwrap1()
	/root/build-deb/engine/vendor/go.opentelemetry.io/otel/sdk/trace/span.go:422 +0x2c
go.opentelemetry.io/otel/sdk/trace.(*recordingSpan).End(0x4001b2af00, {0x0, 0x0, 0x40016e5180?})
	/root/build-deb/engine/vendor/go.opentelemetry.io/otel/sdk/trace/span.go:461 +0xa04
panic({0xaaaac0e6f400?, 0xaaaac27719e0?})
	/usr/local/go/src/runtime/panic.go:792 +0x124
github.com/docker/docker/api/server/router/image.(*imageRouter).getImagesByName(0x4000aa88c0, {0xaaaac1366428, 0x40016c6060}, {0xaaaac135f3e0, 0x4006912960}, 0x4003fdc140, 0x4003257dd0)
	/root/build-deb/engine/api/server/router/image/image_routes.go:414 +0x454
github.com/docker/docker/api/server/middleware.(*ExperimentalMiddleware).WrapHandler.ExperimentalMiddleware.WrapHandler.func1({0xaaaac1366428, 0x40016c6060}, {0xaaaac135f3e0, 0x4006912960}, 0x4003fdc140, 0x4003257dd0)
	/root/build-deb/engine/api/server/middleware/experimental.go:26 +0x94
github.com/docker/docker/api/server/middleware.(*VersionMiddleware).WrapHandler.VersionMiddleware.WrapHandler.func1({0xaaaac1366428, 0x40016c6030}, {0xaaaac135f3e0, 0x4006912960}, 0x4003fdc140, 0x4003257dd0)
	/root/build-deb/engine/api/server/middleware/version.go:84 +0x240
github.com/docker/docker/pkg/authorization.(*Middleware).WrapHandler.func1({0xaaaac1366428, 0x40016c6030}, {0xaaaac135f3e0, 0x4006912960}, 0x4003fdc140, 0x4003257dd0)
	/root/build-deb/engine/pkg/authorization/middleware.go:59 +0x274
github.com/docker/docker/api/server.(*Server).makeHTTPHandler.func1({0xaaaac135f3e0, 0x4006912960}, 0x4003fdc000)
	/root/build-deb/engine/api/server/server.go:61 +0x250
net/http.HandlerFunc.ServeHTTP(0xaaaac1366428?, {0xaaaac135f3e0?, 0x4006912960?}, 0xaaaac0b46ff8?)
	/usr/local/go/src/net/http/server.go:2294 +0x38
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP(0x4000d81040, {0xaaaac1358510, 0x4002a5b960}, 0x400055fa40, {0xaaaac1338a20, 0x4000d8a948})
	/root/build-deb/engine/vendor/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp/handler.go:171 +0xd98
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1({0xaaaac1358510?, 0x4002a5b960?}, 0x4000b869b8?)
	/root/build-deb/engine/vendor/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp/handler.go:65 +0x40
net/http.HandlerFunc.ServeHTTP(0xaaaac1366428?, {0xaaaac1358510?, 0x4002a5b960?}, 0xaaaac0b400b8?)
	/usr/local/go/src/net/http/server.go:2294 +0x38
net/http.HandlerFunc.ServeHTTP(0x400055f400?, {0xaaaac1358510?, 0x4002a5b960?}, 0x0?)
	/usr/local/go/src/net/http/server.go:2294 +0x38
github.com/gorilla/mux.(*Router).ServeHTTP(0x4000ab4cc0, {0xaaaac1358510, 0x4002a5b960}, 0x400055edc0)
	/root/build-deb/engine/vendor/github.com/gorilla/mux/mux.go:212 +0x194
net/http.serverHandler.ServeHTTP({0x4003257cb0?}, {0xaaaac1358510?, 0x4002a5b960?}, 0x1?)
	/usr/local/go/src/net/http/server.go:3301 +0xbc
net/http.(*conn).serve(0x40029edb00, {0xaaaac1366428, 0x4000afc240})
	/usr/local/go/src/net/http/server.go:2102 +0x52c
created by net/http.(*Server).Serve in goroutine 302
	/usr/local/go/src/net/http/server.go:3454 +0x3d8
2025/06/10 15:12:00 http: panic serving @: runtime error: invalid memory address or nil pointer dereference
goroutine 286709 [running]:
net/http.(*conn).serve.func1()
	/usr/local/go/src/net/http/server.go:1947 +0xb0
panic({0xaaaac0e6f400?, 0xaaaac27719e0?})
	/usr/local/go/src/runtime/panic.go:792 +0x124
go.opentelemetry.io/otel/sdk/trace.(*recordingSpan).End.deferwrap1()
	/root/build-deb/engine/vendor/go.opentelemetry.io/otel/sdk/trace/span.go:422 +0x2c
go.opentelemetry.io/otel/sdk/trace.(*recordingSpan).End(0x40019fc5a0, {0x0, 0x0, 0x4000583c00?})
	/root/build-deb/engine/vendor/go.opentelemetry.io/otel/sdk/trace/span.go:461 +0xa04
panic({0xaaaac0e6f400?, 0xaaaac27719e0?})
	/usr/local/go/src/runtime/panic.go:792 +0x124
github.com/docker/docker/api/server/router/image.(*imageRouter).getImagesByName(0x4000aa88c0, {0xaaaac1366428, 0x40096b7230}, {0xaaaac135f3e0, 0x400c08f5c0}, 0x4002399900, 0x40096b7080)
	/root/build-deb/engine/api/server/router/image/image_routes.go:414 +0x454
github.com/docker/docker/api/server/middleware.(*ExperimentalMiddleware).WrapHandler.ExperimentalMiddleware.WrapHandler.func1({0xaaaac1366428, 0x40096b7230}, {0xaaaac135f3e0, 0x400c08f5c0}, 0x4002399900, 0x40096b7080)
	/root/build-deb/engine/api/server/middleware/experimental.go:26 +0x94
github.com/docker/docker/api/server/middleware.(*VersionMiddleware).WrapHandler.VersionMiddleware.WrapHandler.func1({0xaaaac1366428, 0x40096b7200}, {0xaaaac135f3e0, 0x400c08f5c0}, 0x4002399900, 0x40096b7080)
	/root/build-deb/engine/api/server/middleware/version.go:84 +0x240
github.com/docker/docker/pkg/authorization.(*Middleware).WrapHandler.func1({0xaaaac1366428, 0x40096b7200}, {0xaaaac135f3e0, 0x400c08f5c0}, 0x4002399900, 0x40096b7080)
	/root/build-deb/engine/pkg/authorization/middleware.go:59 +0x274
github.com/docker/docker/api/server.(*Server).makeHTTPHandler.func1({0xaaaac135f3e0, 0x400c08f5c0}, 0x40023997c0)
	/root/build-deb/engine/api/server/server.go:61 +0x250
net/http.HandlerFunc.ServeHTTP(0xaaaac1366428?, {0xaaaac135f3e0?, 0x400c08f5c0?}, 0xaaaac0b46ff8?)
	/usr/local/go/src/net/http/server.go:2294 +0x38
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP(0x4000d81040, {0xaaaac1358510, 0x4005c80d20}, 0x4002399680, {0xaaaac1338a20, 0x4000d8a948})
	/root/build-deb/engine/vendor/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp/handler.go:171 +0xd98
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1({0xaaaac1358510?, 0x4005c80d20?}, 0x4000c4c9b8?)
	/root/build-deb/engine/vendor/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp/handler.go:65 +0x40
net/http.HandlerFunc.ServeHTTP(0xaaaac1366428?, {0xaaaac1358510?, 0x4005c80d20?}, 0xaaaac0b400b8?)
	/usr/local/go/src/net/http/server.go:2294 +0x38
net/http.HandlerFunc.ServeHTTP(0x40023992c0?, {0xaaaac1358510?, 0x4005c80d20?}, 0x0?)
	/usr/local/go/src/net/http/server.go:2294 +0x38
github.com/gorilla/mux.(*Router).ServeHTTP(0x4000ab4cc0, {0xaaaac1358510, 0x4005c80d20}, 0x4000e2ba40)
	/root/build-deb/engine/vendor/github.com/gorilla/mux/mux.go:212 +0x194
net/http.serverHandler.ServeHTTP({0x40020a5440?}, {0xaaaac1358510?, 0x4005c80d20?}, 0x1?)
	/usr/local/go/src/net/http/server.go:3301 +0xbc
net/http.(*conn).serve(0x40069ae750, {0xaaaac1366428, 0x4000afc240})
	/usr/local/go/src/net/http/server.go:2102 +0x52c
created by net/http.(*Server).Serve in goroutine 302
	/usr/local/go/src/net/http/server.go:3454 +0x3d8

This doesn't happen all the time. For some reason after I clean/purge the data it can work fine again, however soon enough another image will fail.

This didn't happen yesterday, I believe it may related to the latst Docker for Mac update.

Reproduce

If I know the image is breaking dockerd, this is a dead simple reproducer:

Put this in a compose.yaml file:

services:
  foo:
    image: clickhouse/clickhouse-server:24.8-alpine

Run this:

docker compose pull foo

Get this error:

unable to get image 'clickhouse/clickhouse-server:24.8-alpine': request returned 502 Bad Gateway for API route and version http://%2Fvar%2Frun%2Fdocker.sock/v1.49/images/clickhouse/clickhouse-server:24.8-alpine/json, check if the server supports the requested API version

but since this doesn't happen all the time, it's tricky to reproduce.

Expected behavior

I expect the docker daemon to not crash and my pull to work.

docker version

Client:
 Version:           28.2.2
 API version:       1.50
 Go version:        go1.24.3
 Git commit:        e6534b4
 Built:             Fri May 30 12:07:35 2025
 OS/Arch:           darwin/arm64
 Context:           default

Server: Docker Desktop 4.42.0 (195023)
 Engine:
  Version:          28.2.2
  API version:      1.50 (minimum version 1.24)
  Go version:       go1.24.3
  Git commit:       45873be
  Built:            Fri May 30 12:07:27 2025
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.7.27
  GitCommit:        05044ec0a9a75232cad458027ca83437aae3f4da
 runc:
  Version:          1.2.5
  GitCommit:        v1.2.5-0-g59923ef
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    28.2.2
 Context:    default
 Debug Mode: false
 Plugins:
  ai: Docker AI Agent - Ask Gordon (Docker Inc.)
    Version:  v1.4.0
    Path:     /Users/vincent/.docker/cli-plugins/docker-ai
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.24.0-desktop.2
    Path:     /Users/vincent/.docker/cli-plugins/docker-buildx
  cloud: Docker Cloud (Docker Inc.)
    Version:  v0.3.9
    Path:     /Users/vincent/.docker/cli-plugins/docker-cloud
  compose: Docker Compose (Docker Inc.)
    Version:  v2.36.2-desktop.1
    Path:     /Users/vincent/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.41
    Path:     /Users/vincent/.docker/cli-plugins/docker-debug
  desktop: Docker Desktop commands (Docker Inc.)
    Version:  v0.1.9
    Path:     /Users/vincent/.docker/cli-plugins/docker-desktop
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.29
    Path:     /Users/vincent/.docker/cli-plugins/docker-extension
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.4.0
    Path:     /Users/vincent/.docker/cli-plugins/docker-init
  mcp: Docker MCP Plugin (Docker Inc.)
    Version:  dev
    Path:     /Users/vincent/.docker/cli-plugins/docker-mcp
  model: Docker Model Runner (Docker Inc.)
    Version:  v0.1.24
    Path:     /Users/vincent/.docker/cli-plugins/docker-model
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/vincent/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.18.1
    Path:     /Users/vincent/.docker/cli-plugins/docker-scout

Server:
 Containers: 20
  Running: 19
  Paused: 0
  Stopped: 1
 Images: 12
 Server Version: 28.2.2
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Discovered Devices:
  cdi: docker.com/gpu=webgpu
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05044ec0a9a75232cad458027ca83437aae3f4da
 runc version: v1.2.5-0-g59923ef
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.10.14-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 10
 Total Memory: 15.6GiB
 Name: docker-desktop
 ID: 58fcade1-f2da-44b0-82eb-9a2465f60825
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///Users/vincent/Library/Containers/com.docker.docker/Data/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false

Additional Info

I believe this bug appeared since I've updated to 4.42.0

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/imagesImage Distributioncontainerd-integrationIssues and PRs related to containerd integrationkind/bugBugs are bugs. The cause may or may not be known at triage time so debugging may be needed.status/confirmedversion/28.2

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions