Skip to content

invalid memory address or nil pointer dereference on image.NewFromJSON #45454

Closed
Closed
invalid memory address or nil pointer dereference on image.NewFromJSON#45454
Labels
kind/bugBugs are bugs. The cause may or may not be known at triage time so debugging may be needed.status/0-triageversion/23.0
@kub3let

Description

@kub3let
kub3let
opened on May 3, 2023

Description

Since some docker update (about 2-5 weeks) I'm seeing 1-2 crashes a week, this is affecting multiple systems so it's not a hardware issue.

The only thing changed is install of zabbix 2 agent, wondering whether this has to do with it.

stack trace:

May 03 14:06:34 docker1 dockerd[858]: panic: runtime error: invalid memory address or nil pointer dereference
May 03 14:06:34 docker1 dockerd[858]: runtime/debug.Stack()
May 03 14:06:34 docker1 dockerd[858]:         /usr/local/go/src/runtime/debug/stack.go:24 +0x65
May 03 14:06:34 docker1 dockerd[858]: github.com/docker/docker/vendor/golang.org/x/sync/singleflight.newPanicError({0x560c0aa89800?, 0x560c0c0b6590})
May 03 14:06:34 docker1 dockerd[858]:         /go/src/github.com/docker/docker/vendor/golang.org/x/sync/singleflight/singleflight.go:35 +0x2c
May 03 14:06:34 docker1 dockerd[858]: github.com/docker/docker/vendor/golang.org/x/sync/singleflight.(*Group).doCall.func2.1()
May 03 14:06:34 docker1 dockerd[858]:         /go/src/github.com/docker/docker/vendor/golang.org/x/sync/singleflight/singleflight.go:184 +0x3b
May 03 14:06:34 docker1 dockerd[858]: panic({0x560c0aa89800, 0x560c0c0b6590})
May 03 14:06:34 docker1 dockerd[858]:         /usr/local/go/src/runtime/panic.go:884 +0x212
May 03 14:06:34 docker1 dockerd[858]: encoding/json.(*decodeState).object(0xc00190cbd0, {0x560c0ac6e960?, 0xc001531900?, 0x0?})
May 03 14:06:34 docker1 dockerd[858]:         /usr/local/go/src/encoding/json/decode.go:745 +0x1b5e
May 03 14:06:34 docker1 dockerd[858]: encoding/json.(*decodeState).value(0xc00190cbd0, {0x560c0ac6e960?, 0xc001531900?, 0x90?})
May 03 14:06:34 docker1 dockerd[858]:         /usr/local/go/src/encoding/json/decode.go:374 +0x45
May 03 14:06:34 docker1 dockerd[858]: encoding/json.(*decodeState).unmarshal(0xc00190cbd0, {0x560c0ac6e960?, 0xc001531900?})
May 03 14:06:34 docker1 dockerd[858]:         /usr/local/go/src/encoding/json/decode.go:181 +0x1de
May 03 14:06:34 docker1 dockerd[858]: encoding/json.Unmarshal({0xc0013b4000, 0x2913, 0x2914}, {0x560c0ac6e960, 0xc001531900})
May 03 14:06:34 docker1 dockerd[858]:         /usr/local/go/src/encoding/json/decode.go:108 +0x125
May 03 14:06:34 docker1 dockerd[858]: github.com/docker/docker/image.NewFromJSON({0xc0013b4000, 0x2913, 0x2914})
May 03 14:06:34 docker1 dockerd[858]:         /go/src/github.com/docker/docker/image/image.go:279 +0x56
May 03 14:06:34 docker1 dockerd[858]: github.com/docker/docker/image.(*store).Get(0xc0000a9090, {0xc00043d590, 0x47})
May 03 14:06:34 docker1 dockerd[858]:         /go/src/github.com/docker/docker/image/store.go:211 +0x4e
May 03 14:06:34 docker1 dockerd[858]: github.com/docker/docker/image.(*store).imagesMap(0xc0000a9090, 0x1)
May 03 14:06:34 docker1 dockerd[858]:         /go/src/github.com/docker/docker/image/store.go:329 +0x21b
May 03 14:06:34 docker1 dockerd[858]: github.com/docker/docker/image.(*store).Map(0x0?)
May 03 14:06:34 docker1 dockerd[858]:         /go/src/github.com/docker/docker/image/store.go:316 +0x1e
May 03 14:06:34 docker1 dockerd[858]: github.com/docker/docker/daemon/images.(*ImageService).Images(0xc0005100c0, {0x560c0ae27638?, 0xc001d66640?}, {0x0?, {0xc000d24df
May 03 14:06:34 docker1 dockerd[858]:         /go/src/github.com/docker/docker/daemon/images/images.go:80 +0x3c5
May 03 14:06:34 docker1 dockerd[858]: github.com/docker/docker/daemon/images.(*ImageService).ImageDiskUsage.func1()
May 03 14:06:34 docker1 dockerd[858]:         /go/src/github.com/docker/docker/daemon/images/service.go:246 +0xb0
May 03 14:06:34 docker1 dockerd[858]: github.com/docker/docker/vendor/golang.org/x/sync/singleflight.(*Group).doCall.func2(0xc0009c9f56, 0xc001491180, 0xc001a9e768?)
May 03 14:06:34 docker1 dockerd[858]:         /go/src/github.com/docker/docker/vendor/golang.org/x/sync/singleflight/singleflight.go:189 +0x6f
May 03 14:06:34 docker1 dockerd[858]: github.com/docker/docker/vendor/golang.org/x/sync/singleflight.(*Group).doCall(0xc001491130?, 0x560c0ae0c520?, {0x560c0a367bb8?, 
May 03 14:06:34 docker1 dockerd[858]:         /go/src/github.com/docker/docker/vendor/golang.org/x/sync/singleflight/singleflight.go:191 +0xa5
May 03 14:06:34 docker1 dockerd[858]: created by github.com/docker/docker/vendor/golang.org/x/sync/singleflight.(*Group).DoChan
May 03 14:06:34 docker1 dockerd[858]:         /go/src/github.com/docker/docker/vendor/golang.org/x/sync/singleflight/singleflight.go:129 +0x315
May 03 14:06:34 docker1 dockerd[858]: goroutine 1110528 [running]:
May 03 14:06:34 docker1 dockerd[858]: github.com/docker/docker/vendor/golang.org/x/sync/singleflight.(*Group).doCall.func1.2()
May 03 14:06:34 docker1 dockerd[858]:         /go/src/github.com/docker/docker/vendor/golang.org/x/sync/singleflight/singleflight.go:158 +0x2a
May 03 14:06:34 docker1 dockerd[858]: created by github.com/docker/docker/vendor/golang.org/x/sync/singleflight.(*Group).doCall.func1
May 03 14:06:34 docker1 dockerd[858]:         /go/src/github.com/docker/docker/vendor/golang.org/x/sync/singleflight/singleflight.go:158 +0x2b2
May 03 14:06:34 docker1 systemd[1]: docker.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
May 03 14:06:34 docker1 systemd[1]: docker.service: Failed with result 'exit-code'.
May 03 14:06:36 docker1 systemd[1]: docker.service: Service RestartSec=2s expired, scheduling restart.
May 03 14:06:36 docker1 systemd[1]: docker.service: Scheduled restart job, restart counter is at 1.
May 03 14:06:36 docker1 systemd[1]: Stopped Docker Application Container Engine.

update history:

root@docker1:/var/log/apt# zcat history.log.* | grep docker -B 2
Start-Date: 2023-01-27  09:21:59
Commandline: apt full-upgrade -y
Upgrade: containerd.io:amd64 (1.6.12-1, 1.6.15-1), docker-ce-rootless-extras:amd64 (5:20.10.21~3-0~debian-buster, 5:20.10.23~3-0~debian-buster), docker-scan-plugin:amd64 (0.21.0~debian-buster, 0.23.0~debian-buster), docker-ce:amd64 (5:20.10.21~3-0~debian-buster, 5:20.10.23~3-0~debian-buster), docker-ce-cli:amd64 (5:20.10.21~3-0~debian-buster, 5:20.10.23~3-0~debian-buster)
--
Start-Date: 2023-03-19  12:13:08
Commandline: apt full-upgrade -y
Install: docker-buildx-plugin:amd64 (0.10.2-1~debian.10~buster, automatic), docker-compose-plugin:amd64 (2.16.0-1~debian.10~buster, automatic)
Upgrade: containerd.io:amd64 (1.6.15-1, 1.6.18-1), docker-ce-rootless-extras:amd64 (5:20.10.23~3-0~debian-buster, 5:23.0.1-1~debian.10~buster), docker-ce:amd64 (5:20.10.23~3-0~debian-buster, 5:23.0.1-1~debian.10~buster), docker-ce-cli:amd64 (5:20.10.23~3-0~debian-buster, 5:23.0.1-1~debian.10~buster)
--
Start-Date: 2023-04-03  12:44:17
Commandline: apt full-upgrade -y
Upgrade: zabbix-agent2:amd64 (1:6.4.0-1+debian10, 1:6.4.1-1+debian10), containerd.io:amd64 (1.6.18-1, 1.6.20-1), docker-ce-rootless-extras:amd64 (5:23.0.1-1~debian.10~buster, 5:23.0.2-1~debian.10~buster), docker-buildx-plugin:amd64 (0.10.2-1~debian.10~buster, 0.10.4-1~debian.10~buster), docker-compose-plugin:amd64 (2.16.0-1~debian.10~buster, 2.17.2-1~debian.10~buster), docker-ce:amd64 (5:23.0.1-1~debian.10~buster, 5:23.0.2-1~debian.10~buster), docker-ce-cli:amd64 (5:23.0.1-1~debian.10~buster, 5:23.0.2-1~debian.10~buster)
--
Start-Date: 2023-04-14  19:22:33
Commandline: apt full-upgrade -y
Upgrade: docker-ce-rootless-extras:amd64 (5:23.0.2-1~debian.10~buster, 5:23.0.3-1~debian.10~buster), docker-ce:amd64 (5:23.0.2-1~debian.10~buster, 5:23.0.3-1~debian.10~buster), docker-ce-cli:amd64 (5:23.0.2-1~debian.10~buster, 5:23.0.3-1~debian.10~buster)
--
Start-Date: 2023-04-19  10:01:52
Commandline: apt full-upgrade -y
Upgrade: docker-ce-rootless-extras:amd64 (5:23.0.3-1~debian.10~buster, 5:23.0.4-1~debian.10~buster), docker-ce:amd64 (5:23.0.3-1~debian.10~buster, 5:23.0.4-1~debian.10~buster), docker-ce-cli:amd64 (5:23.0.3-1~debian.10~buster, 5:23.0.4-1~debian.10~buster)
--
Start-Date: 2023-04-28  20:34:00
Commandline: apt full-upgrade -y
Upgrade: zabbix-agent2:amd64 (1:6.4.1-1+debian10, 1:6.4.2-1+debian10), zabbix-agent2-plugin-postgresql:amd64 (6.4.0-1+debian10, 0:6.4.2-1+debian10), docker-ce-rootless-extras:amd64 (5:23.0.4-1~debian.10~buster, 5:23.0.5-1~debian.10~buster), zabbix-agent2-plugin-mongodb:amd64 (6.4.0-1+debian10, 0:6.4.2-1+debian10), docker-compose-plugin:amd64 (2.17.2-1~debian.10~buster, 2.17.3-1~debian.10~buster), docker-ce:amd64 (5:23.0.4-1~debian.10~buster, 5:23.0.5-1~debian.10~buster), docker-ce-cli:amd64 (5:23.0.4-1~debian.10~buster, 5:23.0.5-1~debian.10~buster)

Reproduce

Actually can't tell, docker is just crashing randomly, no other system events that show why.

Expected behavior

Docker should not crash

docker version

Client: Docker Engine - Community
 Version:           23.0.5
 API version:       1.42
 Go version:        go1.19.8
 Git commit:        bc4487a
 Built:             Wed Apr 26 16:17:29 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          23.0.5
  API version:      1.42 (minimum version 1.12)
  Go version:       go1.19.8
  Git commit:       94d3ad6
  Built:            Wed Apr 26 16:17:29 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.20
  GitCommit:        2806fc1057397dbaeefbea0e4e17bddfbd388f38
 runc:
  Version:          1.1.5
  GitCommit:        v1.1.5-0-gf19387a
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.4
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.17.3
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
  scan: Docker Scan (Docker Inc.)
    Version:  v0.23.0
    Path:     /usr/libexec/docker/cli-plugins/docker-scan

Server:
 Containers: 22
  Running: 21
  Paused: 0
  Stopped: 1
 Images: 104
 Server Version: 23.0.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2806fc1057397dbaeefbea0e4e17bddfbd388f38
 runc version: v1.1.5-0-gf19387a
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
 Kernel Version: 4.19.0-23-amd64
 Operating System: Debian GNU/Linux 10 (buster)
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 5.817GiB
 Name: docker1
 ID: PUTV:DSOR:53TR:UVXJ:FKSL:LSNN:R4S3:P6G5:I4CX:RUNN:ZTOT:ATLT
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional Info

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugBugs are bugs. The cause may or may not be known at triage time so debugging may be needed.status/0-triageversion/23.0

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions