Skip to content

Bug:[scale] generate_dump fails to fetch routes in scale environment #24537

@YairRaviv

Description

@YairRaviv

Is it platform specific

generic

Importance or Severity

Critical

Description of the bug

in a scale environment (t0-isolated-d256u256s2), there are 2 issues in generate_dump:

  1. ip route show table all fails with “Error: Buffer too small for object.”
  2. Copying /proc/self/net/ (specifically /ipv6_route) during generate_dump hangs for a long time (~75 min).
    Returned dump is partial (routing tables are missing due to error 1)

Steps to Reproduce

  1. Bring up a testbed with large IP routes. e.g. t0-isolated-d256u256s2 topology.
  2. Run sudo generate_dump -s "30 seconds"

Actual Behavior and Expected Behavior

Actual behavior - generate_dump doesn't generate a full dump, and takes a long time.
Expected behavior - generate_dump should return a full dump in a few minutes.

Relevant log output

time sudo generate_dump -s "30 seconds"
Lock succesfully accquired and installed signal handlers
Error: Buffer too small for object.
....
real    78m35.514s
user    1m43.084s
sys     73m38.202s

Output of show version, show techsupport

admin@:~$ uname -r
6.1.0-29-2-amd64
admin@:~$ ip -V
ip utility, iproute2-6.1.0, libbpf 1.1.0
admin@:~$ show version

SONiC Software Version: SONiC.202412_RC.181-b1a0bb022_Internal
SONiC OS Version: 12
Distribution: Debian 12.9
Kernel: 6.1.0-29-2-amd64
Build commit: d92e2bc58
Build date: Thu Nov 13 15:00:44 UTC 2025
Built by: sw-r2d2-bot@fit-build-vrt-120-124

Platform: x86_64-nvidia_sn5640-r0
HwSKU: Mellanox-SN5640-C512S2
ASIC: mellanox
ASIC Count: 1
Serial Number: MT2512602H0T
Model Number: MSN5640
Hardware Revision: A4
Uptime: 15:39:38 up  6:35,  1 user,  load average: 0.96, 1.04, 1.12
Date: Sun 16 Nov 2025 15:39:38

Docker images:
REPOSITORY                                         TAG                                IMAGE ID       SIZE
docker-syncd-mlnx                                  202412_RC.181-b1a0bb022_Internal   e18b2195befa   809MB
docker-syncd-mlnx                                  latest                             e18b2195befa   809MB
urm.nvidia.com/sw-nbu-sws-sonic-docker/sonic-wjh   2.0.7-202412-024                   41c5b87dd7d4   431MB
docker-orchagent                                   202412_RC.181-b1a0bb022_Internal   816950da6b9a   357MB
docker-orchagent                                   latest                             816950da6b9a   357MB
docker-fpm-frr                                     202412_RC.181-b1a0bb022_Internal   9210b0e7e585   379MB
docker-fpm-frr                                     latest                             9210b0e7e585   379MB
docker-nat                                         202412_RC.181-b1a0bb022_Internal   ca8cb0802a0b   348MB
docker-nat                                         latest                             ca8cb0802a0b   348MB
docker-sflow                                       202412_RC.181-b1a0bb022_Internal   af3f0fc9baf5   346MB
docker-sflow                                       latest                             af3f0fc9baf5   346MB
docker-macsec                                      latest                             ccd1a6dd7d32   348MB
docker-teamd                                       202412_RC.181-b1a0bb022_Internal   de3894862c4d   345MB
docker-teamd                                       latest                             de3894862c4d   345MB
docker-telemetry-watchdog                          202412_RC.181-b1a0bb022_Internal   90ad6f1ed27a   404MB
docker-telemetry-watchdog                          latest                             90ad6f1ed27a   404MB
docker-dhcp-server                                 latest                             1076de348c1e   335MB
docker-dhcp-relay                                  latest                             241f935d155e   321MB
docker-snmp                                        202412_RC.181-b1a0bb022_Internal   39edca472751   356MB
docker-snmp                                        latest                             39edca472751   356MB
docker-platform-monitor                            202412_RC.181-b1a0bb022_Internal   f75ef1b615a7   433MB
docker-platform-monitor                            latest                             f75ef1b615a7   433MB
docker-sonic-mgmt-framework                        202412_RC.181-b1a0bb022_Internal   b216775a859e   399MB
docker-sonic-mgmt-framework                        latest                             b216775a859e   399MB
docker-telemetry-sidecar                           202412_RC.181-b1a0bb022_Internal   bc0693e35a6b   312MB
docker-telemetry-sidecar                           latest                             bc0693e35a6b   312MB
docker-router-advertiser                           202412_RC.181-b1a0bb022_Internal   ef9132d21762   312MB
docker-router-advertiser                           latest                             ef9132d21762   312MB
docker-sonic-restapi                               202412_RC.181-b1a0bb022_Internal   9737db1f30e4   331MB
docker-sonic-restapi                               latest                             9737db1f30e4   331MB
docker-eventd                                      202412_RC.181-b1a0bb022_Internal   9e999b8578d4   312MB
docker-eventd                                      latest                             9e999b8578d4   312MB
docker-lldp                                        202412_RC.181-b1a0bb022_Internal   320a41222b3a   358MB
docker-lldp                                        latest                             320a41222b3a   358MB
docker-database                                    202412_RC.181-b1a0bb022_Internal   8c1b8455e16d   321MB
docker-database                                    latest                             8c1b8455e16d   321MB
docker-sonic-bmp                                   202412_RC.181-b1a0bb022_Internal   32ad9fd374dc   313MB
docker-sonic-bmp                                   latest                             32ad9fd374dc   313MB
docker-mux                                         202412_RC.181-b1a0bb022_Internal   07178c928b1f   364MB
docker-mux                                         latest                             07178c928b1f   364MB
docker-sonic-gnmi                                  202412_RC.181-b1a0bb022_Internal   893632148313   402MB
docker-sonic-gnmi                                  latest

Attach files (if any)

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions