docker stats net I/O do not work after 24.0.x upgrade, works after container restart #45653
Description
Description
We run docker with live-restore enabled and iptables disabled.
With pre-24 versions we were used to make upgrades without issue, but when we upgraded to 24.0.1, our monitoring systems failed to retrieve docker containers net I/O stats.
After investigation, it appears docker stats do not show them either.
We also noticed errors in logs:
May 27 10:33:39 trille dockerd[622073]: time="2023-05-27T10:33:39.243194146Z" level=error msg="failed to populate fields for osl sandbox 67abfd820a74ce968552ae633d70594dd45e72915dbc79365ed1e97e7a387362"
May 27 10:33:39 trille dockerd[622073]: time="2023-05-27T10:33:39.243596763Z" level=error msg="failed to populate fields for osl sandbox 8189efe2c5341321692d45fd69a2dc6fc12140b398919e3dc78ec97055d2ae7d"
May 27 10:33:39 trille dockerd[622073]: time="2023-05-27T10:33:39.243972135Z" level=error msg="failed to populate fields for osl sandbox ae7c36e8222b5adecd6b8abe22894f50d5822b6cff2909b98cb4b83cb50c0599"
May 27 10:33:39 trille dockerd[622073]: time="2023-05-27T10:33:39.244346826Z" level=error msg="failed to populate fields for osl sandbox d733fd814745bfa09df3c8b3109b413788d5d96b4a7622adbd4f63714ede86a3"
May 27 10:33:39 trille dockerd[622073]: time="2023-05-27T10:33:39.244727886Z" level=error msg="failed to populate fields for osl sandbox 208c1f938e071ad10384b7e85b3b17438d589fc3d14d67022a625c0c9810f574"
May 27 10:33:39 trille dockerd[622073]: time="2023-05-27T10:33:39.245092720Z" level=error msg="failed to populate fields for osl sandbox 3f855669b02f7642b472080249c2a580c5fccba08b39362a4d8d19526359d42e"
May 27 10:33:39 trille dockerd[622073]: time="2023-05-27T10:33:39.245489716Z" level=error msg="failed to populate fields for osl sandbox 360f7f8af8121c78bbcc718e3dfc74a94ca8cd6cd9beae785813d911242daf33"
May 27 10:33:39 trille dockerd[622073]: time="2023-05-27T10:33:39.245856577Z" level=error msg="failed to populate fields for osl sandbox 39271985b3b9e8b4ceadce0ad42741ff2858873a3cc199611b421099dc75b550"
May 27 10:33:39 trille dockerd[622073]: time="2023-05-27T10:33:39.246958805Z" level=error msg="failed to populate fields for osl sandbox cfae9b4cd4ed57c869513cfd502ab484b1d523560b18c52e7861aca9905e586e"
May 27 10:33:39 trille dockerd[622073]: time="2023-05-27T10:33:39.247697784Z" level=error msg="failed to populate fields for osl sandbox 1700f904e5cf331891916a871d2e97b2341c904ac317c660efa43687255da9b5"
May 27 10:33:39 trille dockerd[622073]: time="2023-05-27T10:33:39.248084187Z" level=error msg="failed to populate fields for osl sandbox 1f71901cf67d829a19db7a2c404952247418c6846c15b4a0d725612b61d950b3"
May 27 10:33:39 trille dockerd[622073]: time="2023-05-27T10:33:39.248472626Z" level=error msg="failed to populate fields for osl sandbox 9649704effa323dc45215a7808ec09d18a1104548ef173fcb90b91b2fb30f4fd"
May 27 10:33:39 trille dockerd[622073]: time="2023-05-27T10:33:39.248855398Z" level=error msg="failed to populate fields for osl sandbox a48e709fd327c98f7b031dcee623e4b5354ccc9160ca013fce504945b7d1674d"
May 27 10:33:39 trille dockerd[622073]: time="2023-05-27T10:33:39.249654180Z" level=error msg="failed to populate fields for osl sandbox 440cd65de19a83813a6fc35862120ab934d9fe4261d533926195c7bcfd439d8d"
May 27 10:33:39 trille dockerd[622073]: time="2023-05-27T10:33:39.250046212Z" level=error msg="failed to populate fields for osl sandbox 4c2fa2735c5fba0d558d06a4daa8e6f150a8a0366d7cd4f4283091e37f52e4de"
May 27 10:33:39 trille dockerd[622073]: time="2023-05-27T10:33:39.250781711Z" level=error msg="failed to populate fields for osl sandbox ee60047aa69c6efa813b02d4f79c5bc1721eadcb1d18eabb7125c3b89b642d1d"
May 27 10:33:39 trille dockerd[622073]: time="2023-05-27T10:33:39.251210452Z" level=error msg="failed to populate fields for osl sandbox 3b7edff711835f691adf3d3e0dd4446a545132b378054682671149d23cde98e8"
May 27 10:33:39 trille dockerd[622073]: time="2023-05-27T10:33:39.251604818Z" level=error msg="failed to populate fields for osl sandbox 4ecf6f9887014db6fdfc6ede7c639cf4f2843b88bd7c9383035f0c74c0337885"
That's for all containers running on this node, just after the upgrade (Ubuntu 20.04).
Reproduce
Runs systemctl restart docker with live-restore = true
Expected behavior
Net I/O should work even after the upgrade.
docker version
Client: Docker Engine - Community
Version: 24.0.2
API version: 1.43
Go version: go1.20.4
Git commit: cb74dfc
Built: Thu May 25 21:52:22 2023
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 24.0.2
API version: 1.43 (minimum version 1.12)
Go version: go1.20.4
Git commit: 659604f
Built: Thu May 25 21:52:22 2023
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.21
GitCommit: 3dce8eb055cbb6872793272b4f20ed16117344f8
runc:
Version: 1.1.7
GitCommit: v1.1.7-0-g860f061
docker-init:
Version: 0.19.0
GitCommit: de40ad0docker info
Client: Docker Engine - Community
Version: 24.0.2
Context: default
Debug Mode: false
Server:
Containers: 25
Running: 24
Paused: 0
Stopped: 1
Images: 124
Server Version: 24.0.2
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
runc version: v1.1.7-0-g860f061
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 5.15.0-72-generic
Operating System: Ubuntu 20.04.6 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 62.59GiB
Name: trille
ID: LDN5:JIPJ:YCQZ:ZTE2:6JHV:BGAJ:T3QP:2YJ6:HSCY:IZZZ:IN3C:P6XD
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: true
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabledAdditional Info
No response