[build] Add build timing report and dependency analysis tools by rustiqly · Pull Request #25643 · sonic-net/sonic-buildimage

rustiqly · 2026-02-24T15:44:12Z

What I did

Add three build instrumentation scripts and a make build-report target for analyzing SONiC build performance.

Scripts

scripts/build-timing-report.sh
Parses per-package timing from build logs (HEADER/FOOTER timestamps in target/*.log). Generates:

Top 30 slowest packages sorted by duration
Phase breakdown (bookworm/trixie/docker/wheels) with parallelism efficiency
Parallelism timeline (concurrent builds over time)
Summary stats (total CPU vs wall time, max concurrency)
CSV export for further analysis

scripts/build-dep-graph.py
Parses all rules/*.mk files to extract the dependency graph (_DEPENDS, _AFTER, _RDEPENDS, etc.). Generates:

Critical path analysis (longest dependency chain)
Fan-out/fan-in bottleneck identification
Root/leaf package counts (parallelism ceiling)
DOT graph for Graphviz visualization
JSON adjacency list

scripts/build-resource-monitor.sh
Samples system resources during builds (CPU, memory, disk I/O, Docker containers) at configurable intervals. Outputs CSV for correlation with build timeline.

Make target

make build-report runs timing + dependency analysis after a build completes.

Why I did it

Build performance optimization requires measurement. Currently there is no aggregated view of per-package timing or dependency bottlenecks. These tools provide the baseline data needed to identify and prioritize optimization opportunities.

Example findings from a VS build (24-core, 30GB RAM, JOBS=4):

210 packages built in 53m wall (173m CPU time)
Max concurrency: 5 — significant room for improvement
Critical path: 14 packages deep (libnl → libswsscommon → utilities)
Top bottleneck: LIBSWSSCOMMON with 48 downstream dependents
Parallelism drops to 0-1 during libswsscommon → sairedis → swss serialization

How I verified it

Ran all three scripts against a successful nightly VS build:

$ bash scripts/build-timing-report.sh ./target
Total packages with timing data: 210
Top slowest: linux-headers (16m), p4lang-p4c (14m), dhcp4relay (12m)
Parallelism ratio: 322% (CPU/wall)

$ python3 scripts/build-dep-graph.py .
Total packages: 284, Dependency edges: 520
Critical path length: 14 packages
LIBSWSSCOMMON fan-out: 48

$ bash scripts/build-resource-monitor.sh 10 /tmp/test.csv
# Successfully samples CPU/mem/disk every 10s

mssonicbld · 2026-02-24T15:44:20Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2026-02-24T15:44:32Z

Azure Pipelines successfully started running 1 pipeline(s).

Copilot

Pull request overview

This pull request adds build performance analysis and instrumentation tools to sonic-buildimage. The PR introduces three new scripts and a Make target to help developers identify build bottlenecks and optimization opportunities.

Changes:

Added scripts/build-timing-report.sh to parse build logs and generate per-package timing analysis with parallelism statistics
Added scripts/build-dep-graph.py to parse dependency relationships from Makefile rules and generate critical path analysis
Added scripts/build-resource-monitor.sh to sample system resources during builds
Added make build-report target in slave.mk to run timing and dependency analysis

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 22 comments.

File	Description
slave.mk	Added `build-report` Makefile target and updated .PHONY declarations
scripts/build-timing-report.sh	Shell script that parses build log timestamps to generate timing reports
scripts/build-resource-monitor.sh	Shell script that samples CPU, memory, and disk I/O metrics during builds
scripts/build-dep-graph.py	Python script that extracts and analyzes the build dependency graph from .mk files

scripts/build-resource-monitor.sh

scripts/build-dep-graph.py

scripts/build-resource-monitor.sh

scripts/build-timing-report.sh

scripts/build-resource-monitor.sh

scripts/build-timing-report.sh

scripts/build-dep-graph.py

slave.mk

scripts/build-dep-graph.py

mssonicbld · 2026-02-25T15:07:11Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2026-02-25T15:07:24Z

Azure Pipelines successfully started running 1 pipeline(s).

yxieca

LGTM. AI agent on behalf of Ying.

yxieca

LGTM. AI agent on behalf of Ying.

mssonicbld · 2026-02-28T23:47:27Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2026-02-28T23:47:39Z

Azure Pipelines successfully started running 1 pipeline(s).

rustiqly · 2026-02-28T23:49:09Z

Copilot Review — All 22 Threads Addressed

Pushed commit 4733d4e with fixes for all valid issues and resolved all threads.

Code Fixes (17):

free -g rounding → free -m with /1024 division for sub-GB precision
Missing = regex → added plain = to Makefile dep patterns
CPU % cumulative → delta between two /proc/stat reads 1s apart
"Critical path estimate" comment → updated to "Build parallelism analysis"
60s vs 10s mismatch → fixed section comment to say "60-second samples"
Fan stats missing after-only pkgs → compute_fan_stats now includes after relationships
Missing ?= regex → added ? to all regex character classes
Disk I/O div-by-zero → guard on INTERVAL > 0
elapsed_line unused → removed
Redundant LIBSWSSCOMMON_DBG → removed (caught by _DBG suffix)
active_make_jobs in header → removed from comment
_RDEPENDS not used → now populates rdeps dict
if v filter on rdeps → removed unnecessary filter
/proc/stat idle field → new delta code handles all 7 fields explicitly
REPORT_FORMAT unused → removed parameter
Critical path cycles → added in_progress set for cycle detection
Execute permission check → added chmod +x loop for companion scripts

Not Real Issues (5) — Resolved with explanation:

date -d safety — build logs are trusted, not user input
CSV commas — SONiC packages follow Debian naming, no commas
Path traversal — build system controls target/ paths
DOT red/white — standard graphviz convention, renders fine
if v redundant — harmless defensive filter (removed anyway in sync submodules (linux-kernel, sairedis, swss, and swss-common to latest master) #13)

yxieca · 2026-03-01T00:11:48Z

AI agent on behalf of Ying: CI is still running; will re-check and approve once green.

yxieca · 2026-03-01T06:21:02Z

AI agent on behalf of Ying: CI not clean yet (Azure.sonic-buildimage still in progress; optional kvmtest-t1-lag-vpp failed). Please update once green.

yxieca · 2026-03-01T09:20:33Z

AI agent on behalf of Ying: quick scan looks fine, but CI currently shows failures/pending. Please check the failing jobs and rerun; I’ll re-review once green.

yxieca · 2026-03-01T12:20:30Z

CI shows failures (Azure/impacted-area tests). Please re-run or confirm if infra flake. Once green, I can approve.\n\nAI agent on behalf of Ying.

yxieca · 2026-03-01T15:20:28Z

AI agent on behalf of Ying: quick check shows outstanding issues.

CI failing: Azure.sonic-buildimage (Test kvmtest-t1-lag-vpp by Elastictest [OPTIONAL]).
Please address/re-run; I’ll re-check.

yxieca

AI agent on behalf of Ying. Quick review: [build]. No issues found.

yxieca

AI agent on behalf of Ying. Quick review: [build] Add build timing report and dependency analysis tools. No issues found.

yxieca

AI agent on behalf of Ying.

yxieca · 2026-03-03T21:54:14Z

/azp run

azure-pipelines · 2026-03-03T21:54:31Z

Azure Pipelines successfully started running 1 pipeline(s).

Add three scripts for build performance instrumentation: - scripts/build-timing-report.sh: Parse per-package timing from build logs (HEADER/FOOTER timestamps), generate sorted duration table, phase breakdown, parallelism timeline, and CSV export. - scripts/build-dep-graph.py: Parse rules/*.mk dependency graph, compute critical path, fan-out/fan-in bottleneck analysis, and generate DOT/JSON output for visualization. - scripts/build-resource-monitor.sh: Sample CPU, memory, disk I/O, and Docker container count during builds for resource utilization analysis. Add "make build-report" target to slave.mk that runs the timing report and dependency analysis after a build completes. Example output from a VS build on 24-core/30GB machine: - 210 packages built in 53m wall time (173m CPU) - Max concurrency: 5 (with SONIC_CONFIG_BUILD_JOBS=4) - Critical path: 14 packages deep (libnl -> libswsscommon -> utilities) - Top bottleneck: LIBSWSSCOMMON with 48 downstream dependents Signed-off-by: Rustiqly <rustiqly@users.noreply.github.com>

- Use free -m with division instead of free -g to avoid rounding (#1) - Add = and ?= to Makefile dependency regex patterns (sonic-net#2, sonic-net#7) - CPU calculation now uses /proc/stat delta (two reads) (sonic-net#3, sonic-net#14) - Fix misleading 'critical path estimate' comment (sonic-net#4) - Fix parallelism timeline comment (60s not 10s) (sonic-net#5) - Include after-relationship packages in fan stats (sonic-net#6) - Guard disk I/O division by zero when INTERVAL<=1 (sonic-net#8) - Remove unused elapsed_line variable (sonic-net#9) - Remove redundant LIBSWSSCOMMON_DBG check (sonic-net#10) - Remove active_make_jobs from CSV header comment (sonic-net#11) - Wire up _RDEPENDS parsing to build reverse deps (sonic-net#12) - Remove unnecessary 'if v' filter on rdeps JSON (sonic-net#13) - Remove unused REPORT_FORMAT parameter (sonic-net#15) - Add cycle detection to critical path algorithm (sonic-net#16) - Add execute permission check for companion scripts (sonic-net#17) Signed-off-by: Rustiqly <rustiqly@users.noreply.github.com>

lihuay · 2026-03-04T17:49:57Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2026-03-04T17:50:11Z

Azure Pipelines successfully started running 1 pipeline(s).

…net#25643) * [build] Add build timing report and dependency analysis tools Add three scripts for build performance instrumentation: - scripts/build-timing-report.sh: Parse per-package timing from build logs (HEADER/FOOTER timestamps), generate sorted duration table, phase breakdown, parallelism timeline, and CSV export. - scripts/build-dep-graph.py: Parse rules/*.mk dependency graph, compute critical path, fan-out/fan-in bottleneck analysis, and generate DOT/JSON output for visualization. - scripts/build-resource-monitor.sh: Sample CPU, memory, disk I/O, and Docker container count during builds for resource utilization analysis. Add "make build-report" target to slave.mk that runs the timing report and dependency analysis after a build completes. Example output from a VS build on 24-core/30GB machine: - 210 packages built in 53m wall time (173m CPU) - Max concurrency: 5 (with SONIC_CONFIG_BUILD_JOBS=4) - Critical path: 14 packages deep (libnl -> libswsscommon -> utilities) - Top bottleneck: LIBSWSSCOMMON with 48 downstream dependents Signed-off-by: Rustiqly <rustiqly@users.noreply.github.com> * Address Copilot review: fix 17 bugs in build analysis scripts - Use free -m with division instead of free -g to avoid rounding (sonic-net#1) - Add = and ?= to Makefile dependency regex patterns (sonic-net#2, sonic-net#7) - CPU calculation now uses /proc/stat delta (two reads) (sonic-net#3, sonic-net#14) - Fix misleading 'critical path estimate' comment (sonic-net#4) - Fix parallelism timeline comment (60s not 10s) (sonic-net#5) - Include after-relationship packages in fan stats (sonic-net#6) - Guard disk I/O division by zero when INTERVAL<=1 (sonic-net#8) - Remove unused elapsed_line variable (sonic-net#9) - Remove redundant LIBSWSSCOMMON_DBG check (sonic-net#10) - Remove active_make_jobs from CSV header comment (sonic-net#11) - Wire up _RDEPENDS parsing to build reverse deps (sonic-net#12) - Remove unnecessary 'if v' filter on rdeps JSON (sonic-net#13) - Remove unused REPORT_FORMAT parameter (sonic-net#15) - Add cycle detection to critical path algorithm (sonic-net#16) - Add execute permission check for companion scripts (sonic-net#17) Signed-off-by: Rustiqly <rustiqly@users.noreply.github.com> --------- Signed-off-by: Rustiqly <rustiqly@users.noreply.github.com> Co-authored-by: Rustiqly <rustiqly@users.noreply.github.com> Signed-off-by: Feng Pan <fenpan@microsoft.com>

rustiqly requested review from lguohan, qiluo-msft and xumia as code owners February 24, 2026 15:44

lihuay requested a review from Copilot February 24, 2026 15:45

Copilot started reviewing on behalf of lihuay February 24, 2026 15:46 View session

lihuay requested a review from yxieca February 24, 2026 15:49

rustiqly mentioned this pull request Feb 24, 2026

[build] Auto-detect SONIC_CONFIG_BUILD_JOBS from CPU and RAM #25644

Merged

Copilot AI reviewed Feb 24, 2026

View reviewed changes

rustiqly mentioned this pull request Feb 24, 2026

[build] Add INCLUDE_PTF config to skip PTF test containers #25646

Merged

rustiqly force-pushed the feat/build-timing-report branch 2 times, most recently from 6ad8bf2 to f4c184b Compare February 25, 2026 15:01

rustiqly force-pushed the feat/build-timing-report branch from f4c184b to fd858c6 Compare February 26, 2026 15:02

rustiqly mentioned this pull request Feb 26, 2026

[ci] Azure Pipelines '/azp run' fails for external contributors — 'does not have sufficient privileges' #25709

Open

yxieca approved these changes Feb 28, 2026

View reviewed changes

yxieca previously approved these changes Feb 28, 2026

View reviewed changes

rustiqly dismissed yxieca’s stale review via 4733d4e February 28, 2026 23:47

yxieca approved these changes Mar 1, 2026

View reviewed changes

rustiqly force-pushed the feat/build-timing-report branch from 4733d4e to 6d1c672 Compare March 2, 2026 15:01

yxieca approved these changes Mar 3, 2026

View reviewed changes

rustiqly force-pushed the feat/build-timing-report branch from 6d1c672 to 8111d36 Compare March 3, 2026 15:02

rustiqly added 2 commits March 4, 2026 07:04

rustiqly force-pushed the feat/build-timing-report branch from 8111d36 to da18f0d Compare March 4, 2026 15:04

lihuay merged commit 31bdc3f into sonic-net:master Mar 5, 2026
25 of 26 checks passed

Conversation

rustiqly commented Feb 24, 2026

What I did

Scripts

Make target

Why I did it

Example findings from a VS build (24-core, 30GB RAM, JOBS=4):

How I verified it

Uh oh!

mssonicbld commented Feb 24, 2026

Uh oh!

azure-pipelines bot commented Feb 24, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mssonicbld commented Feb 25, 2026

Uh oh!

azure-pipelines bot commented Feb 25, 2026

Uh oh!

yxieca left a comment

Choose a reason for hiding this comment

Uh oh!

yxieca left a comment

Choose a reason for hiding this comment

Uh oh!

mssonicbld commented Feb 28, 2026

Uh oh!

azure-pipelines bot commented Feb 28, 2026

Uh oh!

rustiqly commented Feb 28, 2026

Copilot Review — All 22 Threads Addressed

Code Fixes (17):

Not Real Issues (5) — Resolved with explanation:

Uh oh!

yxieca commented Mar 1, 2026

Uh oh!

yxieca commented Mar 1, 2026

Uh oh!

yxieca commented Mar 1, 2026

Uh oh!

yxieca commented Mar 1, 2026

Uh oh!

yxieca commented Mar 1, 2026

Uh oh!

yxieca left a comment

Choose a reason for hiding this comment

Uh oh!

yxieca left a comment

Choose a reason for hiding this comment

Uh oh!

yxieca left a comment

Choose a reason for hiding this comment

Uh oh!

yxieca commented Mar 3, 2026

Uh oh!

azure-pipelines bot commented Mar 3, 2026

Uh oh!

lihuay commented Mar 4, 2026

Uh oh!

azure-pipelines bot commented Mar 4, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone