feat: change the benchmark report to json format by zirain · Pull Request #6818 · envoyproxy/gateway

zirain · 2025-08-19T08:24:30Z

Change the benchmark report to json format for better post-process(e.g. show it on doc site).

codecov · 2025-08-19T08:44:24Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 72.30%. Comparing base (3052ea4) to head (55152f4).
⚠️ Report is 122 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6818      +/-   ##
==========================================
- Coverage   72.33%   72.30%   -0.04%     
==========================================
  Files         232      232              
  Lines       34114    34143      +29     
==========================================
+ Hits        24678    24688      +10     
- Misses       7661     7676      +15     
- Partials     1775     1779       +4

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

arkodg · 2025-08-20T00:47:42Z

thanks, can you share a before / after

github-actions · 2025-09-20T12:03:06Z

This pull request has been automatically marked as stale because it has not had activity in the last 30 days. Please feel free to give a status update now, ping for review, when it's ready. Thank you for your contributions!

arkodg · 2025-09-24T21:20:03Z

tools/make/kube.mk

@@ -259,8 +259,8 @@ run-benchmark: install-benchmark-server prepare-ip-family ## Run benchmark tests
 .PHONY: install-benchmark-server
 install-benchmark-server: ## Install nighthawk server for benchmark test
 	@$(LOG_TARGET)
-	kubectl create namespace benchmark-test
-	kubectl -n benchmark-test create configmap test-server-config --from-file=test/benchmark/config/nighthawk-test-server-config.yaml -o yaml
+	kubectl create namespace benchmark-test --dry-run=client -o yaml | kubectl apply -f -


why is this needed ?

make this the target could be run multiple times.

test/benchmark/tests/scale_httproutes.go

github-actions · 2025-10-25T00:07:23Z

This pull request has been automatically marked as stale because it has not had activity in the last 30 days. Please feel free to give a status update now, ping for review, when it's ready. Thank you for your contributions!

Signed-off-by: zirain <zirain2009@gmail.com>

zirain · 2025-11-25T07:57:25Z

before:

benchmark_report.md

zirain · 2025-11-25T07:57:53Z

after:

benchmark_result.json
benchmark_result.md

Signed-off-by: zirain <zirain2009@gmail.com>

nacx · 2025-12-16T16:23:08Z

bump to get a final review and see if we can have this merged? 🙇‍♂️

zhaohuabing

LGTM Thanks!

* benchmark json output Signed-off-by: zirain <zirain2009@gmail.com> * fix Signed-off-by: zirain <zirain2009@gmail.com> * fix Signed-off-by: zirain <zirain2009@gmail.com> * fix lint Signed-off-by: zirain <zirain2009@gmail.com> * fix Signed-off-by: zirain <zirain2009@gmail.com> * revert Signed-off-by: zirain <zirain2009@gmail.com> * fix seconds Signed-off-by: zirain <zirain2009@gmail.com> --------- Signed-off-by: zirain <zirain2009@gmail.com> Signed-off-by: Matt Miller <millermatt@outlook.com>

* benchmark json output Signed-off-by: zirain <zirain2009@gmail.com> * fix Signed-off-by: zirain <zirain2009@gmail.com> * fix Signed-off-by: zirain <zirain2009@gmail.com> * fix lint Signed-off-by: zirain <zirain2009@gmail.com> * fix Signed-off-by: zirain <zirain2009@gmail.com> * revert Signed-off-by: zirain <zirain2009@gmail.com> * fix seconds Signed-off-by: zirain <zirain2009@gmail.com> --------- Signed-off-by: zirain <zirain2009@gmail.com> Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com>

* fix: set observedGeneration in envoy patch policy (#7715) * fix: set observedGeneration in envoy patch policy Signed-off-by: kkk777-7 <kota.kimura0725@gmail.com> * add release note Signed-off-by: kkk777-7 <kota.kimura0725@gmail.com> --------- Signed-off-by: kkk777-7 <kota.kimura0725@gmail.com> Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com> * fix: add validation for request buffer limit (#7687) Signed-off-by: kkk777-7 <kota.kimura0725@gmail.com> Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com> * fix: nil pointer error when applying BackendTrafficPolicy to HTTPRoute with no backendRefs (#7765) * fix: checking route section name in backend traffic policy Signed-off-by: kkk777-7 <kota.kimura0725@gmail.com> Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com> * fix: setting externalTrafficPolicy for NodePort service type (#7823) Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com> * fix: add indexing and processing for CRL references in ClientTrafficPolicies (#7829) Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com> * feat: change the benchmark report to json format (#6818) * benchmark json output Signed-off-by: zirain <zirain2009@gmail.com> * fix Signed-off-by: zirain <zirain2009@gmail.com> * fix Signed-off-by: zirain <zirain2009@gmail.com> * fix lint Signed-off-by: zirain <zirain2009@gmail.com> * fix Signed-off-by: zirain <zirain2009@gmail.com> * revert Signed-off-by: zirain <zirain2009@gmail.com> * fix seconds Signed-off-by: zirain <zirain2009@gmail.com> --------- Signed-off-by: zirain <zirain2009@gmail.com> Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com> * bechmark: scale up RPS to test data plane CPU performance (#7810) * Scale up RPS to test data plane CPU performance Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com> * set duration to 120s Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com> * discard invalid samples Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com> * change scrape interval to 10s Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com> * remove invalid cpu sampling data Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com> * reduce duration to 60 Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com> * fix benchmark end time Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com> * fix data plane benchmark start time Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com> * increase test time to get more samples Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com> * adjust rps for each scale Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com> * address comments Signed-off-by: Huabing(Robin) Zhao <zhaohuabing@gmail.com> --------- Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com> Signed-off-by: Huabing(Robin) Zhao <zhaohuabing@gmail.com> Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com> * fix: make port-forward worked for OTel collector on port 19001 (#7860) Signed-off-by: zirain <zirain2009@gmail.com> Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com> * chore: fix goroutine leak (#7880) fix goroutine leak Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com> Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com> * fix gen-check Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com> --------- Signed-off-by: kkk777-7 <kota.kimura0725@gmail.com> Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com> Signed-off-by: zirain <zirain2009@gmail.com> Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com> Signed-off-by: Huabing(Robin) Zhao <zhaohuabing@gmail.com> Co-authored-by: Kota Kimura <86363983+kkk777-7@users.noreply.github.com> Co-authored-by: zirain <zirain2009@gmail.com> Co-authored-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com>

zirain requested a review from a team as a code owner August 19, 2025 08:24

zirain force-pushed the bench/output-format branch from 271e9d6 to 9832c57 Compare August 20, 2025 06:45

zirain closed this Aug 20, 2025

zirain reopened this Aug 21, 2025

github-actions bot added the stale label Sep 20, 2025

arkodg removed the stale label Sep 20, 2025

arkodg reviewed Sep 24, 2025

View reviewed changes

test/benchmark/tests/scale_httproutes.go Show resolved Hide resolved

github-actions bot added the stale label Oct 25, 2025

zirain force-pushed the bench/output-format branch 3 times, most recently from c1cf3ac to a89dee8 Compare November 24, 2025 02:34

github-actions bot removed the stale label Nov 24, 2025

zirain force-pushed the bench/output-format branch 5 times, most recently from 366127d to 477ad88 Compare November 24, 2025 12:33

arkodg added this to the v1.7.0-rc.1 Release milestone Nov 25, 2025

zirain added 6 commits November 25, 2025 12:44

benchmark json output

cc5911c

Signed-off-by: zirain <zirain2009@gmail.com>

fix

5d55568

Signed-off-by: zirain <zirain2009@gmail.com>

fix

9440c77

Signed-off-by: zirain <zirain2009@gmail.com>

fix lint

0a4a7da

Signed-off-by: zirain <zirain2009@gmail.com>

fix

5a24eb2

Signed-off-by: zirain <zirain2009@gmail.com>

revert

4304320

Signed-off-by: zirain <zirain2009@gmail.com>

zirain force-pushed the bench/output-format branch from c498990 to 4304320 Compare November 25, 2025 04:44

zirain requested a review from arkodg November 28, 2025 13:10

arkodg previously approved these changes Dec 1, 2025

View reviewed changes

arkodg requested review from a team and arkodg December 1, 2025 19:42

fix seconds

55152f4

Signed-off-by: zirain <zirain2009@gmail.com>

zirain dismissed arkodg’s stale review via 55152f4 December 2, 2025 02:40

arkodg approved these changes Dec 19, 2025

View reviewed changes

arkodg requested review from a team December 19, 2025 05:24

zhaohuabing approved these changes Dec 19, 2025

View reviewed changes

zirain merged commit d4c7220 into envoyproxy:main Dec 19, 2025
32 checks passed

zirain deleted the bench/output-format branch December 19, 2025 06:42

rudrakhp added the cherrypick/release-v1.6.2 label Jan 8, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: change the benchmark report to json format#6818

feat: change the benchmark report to json format#6818
zirain merged 7 commits intoenvoyproxy:mainfrom
zirain:bench/output-format

zirain commented Aug 19, 2025

Uh oh!

codecov bot commented Aug 19, 2025 •

edited

Loading

Uh oh!

arkodg commented Aug 20, 2025

Uh oh!

github-actions bot commented Sep 20, 2025

Uh oh!

arkodg Sep 24, 2025

Uh oh!

zirain Nov 24, 2025

Uh oh!

Uh oh!

github-actions bot commented Oct 25, 2025

Uh oh!

zirain commented Nov 25, 2025

Uh oh!

zirain commented Nov 25, 2025

Uh oh!

nacx commented Dec 16, 2025

Uh oh!

zhaohuabing left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

zirain commented Aug 19, 2025

Uh oh!

codecov bot commented Aug 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

arkodg commented Aug 20, 2025

Uh oh!

github-actions bot commented Sep 20, 2025

Uh oh!

arkodg Sep 24, 2025

Choose a reason for hiding this comment

Uh oh!

zirain Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot commented Oct 25, 2025

Uh oh!

zirain commented Nov 25, 2025

Uh oh!

zirain commented Nov 25, 2025

Uh oh!

nacx commented Dec 16, 2025

Uh oh!

zhaohuabing left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

codecov bot commented Aug 19, 2025 •

edited

Loading