dockerd dns memory leak/usage issue #48644
Description
Description
I recently had to deal with a container doing some badness in that it was trying to connect to redis server 1000s of times a second. While this was happening dockerd started consuming a large amount of memory. With a bit more debugging I was able to narrow down to memory being related to dockerd's internal dns service.
Reproduce
root@poller2.lab003:~/tmp# cat docker-compose.yml
name: dns-memory-leak
services:
busybox:
image: busybox:stable
command:
- "/bin/sh"
- "-c"
- "while [ 1 ];do sleep 10;done"
root@poller2.lab003:~/tmp# docker compose up -d
[+] Running 1/1
✔ Container dns-memory-leak-busybox-1 Started 0.6s
root@poller2.lab003:~/tmp# docker exec -it dns-memory-leak-busybox-1 /bin/sh
/ # grep nameserver /etc/resolv.conf
nameserver 127.0.0.11
/ # while [ 1 ];do nslookup www.google.com;done >/dev/null
In a 2nd terminal to monitor dockerd memory usage every 10 seconds
root@poller2.lab003:/home/jwestfall# while [ 1 ];do (ps auwwx | grep dockerd | grep -v grep);sleep 10;done
root 610462 0.8 1.0 2281572 89168 ? Ssl 20:06 0:00 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
root 610462 0.6 1.0 2281572 89168 ? Ssl 20:06 0:00 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
root 610462 0.5 1.0 2281572 89168 ? Ssl 20:06 0:00 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
root 610462 0.4 1.1 2281572 92032 ? Ssl 20:06 0:00 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
root 610462 0.3 1.1 2281572 92032 ? Ssl 20:06 0:00 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
root 610462 5.7 4.2 2546628 343408 ? Ssl 20:06 0:04 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock <--- test starts here
root 610462 10.7 8.0 2812744 651644 ? Ssl 20:06 0:09 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
root 610462 14.9 11.6 3144128 943696 ? Ssl 20:06 0:15 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
root 610462 18.4 15.3 3410128 1249472 ? Ssl 20:06 0:20 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
root 610462 21.3 19.1 3741656 1557688 ? Ssl 20:06 0:25 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
root 610462 23.6 22.3 4006824 1815224 ? Ssl 20:06 0:30 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
root 610462 25.8 25.8 4272804 2098292 ? Ssl 20:06 0:36 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
root 610462 27.7 29.8 4604408 2427164 ? Ssl 20:06 0:41 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
root 610462 29.2 33.7 4936332 2746728 ? Ssl 20:06 0:46 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
root 610462 30.9 37.5 5268036 3053592 ? Ssl 20:06 0:52 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
root 610462 32.1 40.3 5467536 3280180 ? Ssl 20:06 0:58 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
root 610462 33.3 42.1 5666396 3425328 ? Ssl 20:06 1:03 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
root 610462 34.5 34.1 5666396 2774640 ? Ssl 20:06 1:09 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
root 610462 35.3 34.9 5666396 2842744 ? Ssl 20:06 1:14 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
root 610462 36.2 39.3 5666396 3201248 ? Ssl 20:06 1:20 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
root 610462 37.1 42.5 5666396 3463928 ? Ssl 20:06 1:25 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
root 610462 37.6 45.1 5865000 3668368 ? Ssl 20:06 1:30 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
root 610462 38.1 47.8 6064436 3892812 ? Ssl 20:06 1:35 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
root 610462 38.5 36.5 6064436 2974384 ? Ssl 20:06 1:40 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
root 610462 39.1 37.9 6064436 3085252 ? Ssl 20:06 1:46 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
root 610462 39.4 39.8 6064436 3242856 ? Ssl 20:06 1:50 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
root 610462 39.8 42.8 6064436 3483624 ? Ssl 20:06 1:55 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
6th column is ram usage. It starts out at ~90MB and within 100 seconds its up over 3G. Then kinda bounces around 2.7G to 3.8G.
Expected behavior
dockerd shouldn't be consuming that much ram dealing with dns requests.
docker version
Client: Docker Engine - Community
Version: 27.3.1
API version: 1.47
Go version: go1.22.7
Git commit: ce12230
Built: Fri Sep 20 11:41:11 2024
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 27.3.1
API version: 1.47 (minimum version 1.24)
Go version: go1.22.7
Git commit: 41ca978
Built: Fri Sep 20 11:41:11 2024
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.7.22
GitCommit: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c
runc:
Version: 1.1.14
GitCommit: v1.1.14-0-g2c9f560
docker-init:
Version: 0.19.0
GitCommit: de40ad0
docker info
Client: Docker Engine - Community
Version: 27.3.1
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.17.1
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.29.7
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 7
Running: 1
Paused: 0
Stopped: 6
Images: 4
Server Version: 27.3.1
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: local
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c
runc version: v1.1.14-0-g2c9f560
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.1.0-26-amd64
Operating System: Debian GNU/Linux 12 (bookworm)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.755GiB
Name: poller2.lab003.jwestfall.net
ID: 8aca0fa1-ad53-49b3-b442-18ee214c0183
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Default Address Pools:
Base: 198.18.2.0/23, Size: 27
Additional Info
No response