Skip to content

Docker stats doesn't work on 128 CPUs core system (error scanning '/proc/stat' file: bufio.Scanner: token too long") #49709

@ShaggyGS

Description

@ShaggyGS

Description

We are unable to get stats from docker containers on ARM server with 128 CPU cores... The stats just shows zero values and /var/log/messages is spammed with this error message:

CONTAINER ID   NAME      CPU %     MEM USAGE / LIMIT   MEM %     NET I/O   BLOCK I/O   PIDS
75a2d37961f0   mysql1    0.00%     0B / 0B             0.00%     0B / 0B   0B / 0B     0
4da2f013161e   apache1   0.00%     0B / 0B             0.00%     0B / 0B   0B / 0B     0
Mar 28 11:40:36 localhost dockerd[2820]: time="2025-03-28T11:40:36.493046461+01:00" level=error msg="collecting stats for container /apache1: error scanning '/proc/stat' file: bufio.Scanner: token too long"
Mar 28 11:40:37 localhost dockerd[2820]: time="2025-03-28T11:40:37.495361805+01:00" level=error msg="collecting stats for container /mysql1: error scanning '/proc/stat' file: bufio.Scanner: token too long"
Mar 28 11:40:37 localhost dockerd[2820]: time="2025-03-28T11:40:37.497097407+01:00" level=error msg="collecting stats for container /apache1: error scanning '/proc/stat' file: bufio.Scanner: token too long"
Mar 28 11:40:38 localhost dockerd[2820]: time="2025-03-28T11:40:38.499816392+01:00" level=error msg="collecting stats for container /mysql1: error scanning '/proc/stat' file: bufio.Scanner: token too long"
Mar 28 11:40:38 localhost dockerd[2820]: time="2025-03-28T11:40:38.501417114+01:00" level=error msg="collecting stats for container /apache1: error scanning '/proc/stat' file: bufio.Scanner: token too long"

Reproduce

Run any container on 128 CPU core system and check docker stats.

Expected behavior

No response

docker version

Client: Docker Engine - Community
 Version:           28.0.4
 API version:       1.48
 Go version:        go1.23.7
 Git commit:        b8034c0
 Built:             Tue Mar 25 15:10:30 2025
 OS/Arch:           linux/arm64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          28.0.4
  API version:      1.48 (minimum version 1.24)
  Go version:       go1.23.7
  Git commit:       6430e49
  Built:            Tue Mar 25 15:08:58 2025
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.7.26
  GitCommit:        753481ec61c7c8955a23d6ff7bc8e4daed455734
 runc:
  Version:          1.2.5
  GitCommit:        v1.2.5-0-g59923ef
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client: Docker Engine - Community
 Version:    28.0.4
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.22.0
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.34.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 3
  Running: 2
  Paused: 0
  Stopped: 1
 Images: 10
 Server Version: 28.0.4
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 753481ec61c7c8955a23d6ff7bc8e4daed455734
 runc version: v1.2.5-0-g59923ef
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.14.0-503.33.1.el9_5.aarch64+64k
 Operating System: Rocky Linux 9.5 (Blue Onyx)
 OSType: linux
 Architecture: aarch64
 CPUs: 128
 Total Memory: 126.4GiB
 Name: localhost.localdomain
 ID: 75fc5d8d-e34f-458a-b3da-7ad00033e73c
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false

Additional Info

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/runtimeRuntimehelp wantedkind/bugBugs are bugs. The cause may or may not be known at triage time so debugging may be needed.version/28.0

    Type

    Projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions