Skip to content

feat: add benchmark comparison and PR comments#594

Draft
meyer9 wants to merge 1 commit intomainfrom
meyer9/gha-benchmark-comparison
Draft

feat: add benchmark comparison and PR comments#594
meyer9 wants to merge 1 commit intomainfrom
meyer9/gha-benchmark-comparison

Conversation

@meyer9
Copy link
Copy Markdown
Contributor

@meyer9 meyer9 commented Jan 28, 2026

Summary

  • Add compare-benchmarks.js script to parse and compare benchmark results
  • Download baseline from main branch when running on PRs
  • Post well-formatted markdown comment with comparison table
  • Handle missing baseline gracefully (show results without comparison)
  • Format metrics with proper units (gas/s, ms, s, etc.)

Changes

  • .github/scripts/compare-benchmarks.js - Benchmark comparison script
  • .github/workflows/benchmark.yml - Updated workflow with PR comment steps

Test plan

  • Verify comparison script works with benchmark output
  • Verify PR comment is posted correctly
  • Verify baseline download works from main branch

Depends on #591

@cb-heimdall
Copy link
Copy Markdown
Collaborator

cb-heimdall commented Jan 28, 2026

🟡 Heimdall Review Status

Requirement Status More Info
Reviews 🟡 0/1
Denominator calculation
Show calculation
1 if user is bot 0
1 if user is external 0
2 if repo is sensitive 0
From .codeflow.yml 1
Additional review requirements
Show calculation
Max 0
0
From CODEOWNERS 0
Global minimum 0
Max 1
1
1 if commit is unverified 1
Sum 2

@meyer9 meyer9 changed the base branch from main to meyer9/gha-benchmark January 28, 2026 20:06
@meyer9 meyer9 force-pushed the meyer9/gha-benchmark-comparison branch from a99208d to dd4045f Compare January 28, 2026 20:09
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jan 28, 2026

📊 Benchmark Results: Base builder transfer throughput test

📋 Test Configuration
Setting Value
BenchmarkRun test-1770665567762858
BlockTimeMilliseconds 1000
GasLimit 1000000000
NodeArgs ``
NodeType builder
TransactionPayload transfer-only
ValidatorNodeType base-reth-node

Comparison with main

Sequencer Metrics

Metric Current Baseline Change
Gas/Second 530.66 Mgas/s 568.12 Mgas/s ⚠️ -6.6%
Transactions/Block 31,456 33,867.5 ⚠️ -7.1%
Get Payload Latency 236.35 ms 243.74 ms ✅ -3.0%
Send Txs Latency 2.97 s 2.94 s ⚠️ +1.1%

Validator Metrics

Metric Current Baseline Change
Gas/Second 390.81 Mgas/s 421.33 Mgas/s ⚠️ -7.2%
New Payload Latency 1.71 s 1.72 s ➡️ -0.3%
Execution Duration 1.39 s 1.41 s ➡️ -0.9%
Flashblock Processing 2.00 s 310.03 ms 🔴 +544.5%
📈 Summary Metrics
Metric Sequencer Validator
Gas/Second 530.66 Mgas/s 390.81 Mgas/s
Fork Choice Updated 5.62 ms -
Get Payload 236.35 ms -
New Payload - 1.71 s
📖 Legend
  • 🚀 Significant improvement (>10%)
  • ✅ Improvement
  • ➡️ No significant change (<1%)
  • ⚠️ Regression
  • 🔴 Significant regression (>10%)

@meyer9 meyer9 force-pushed the meyer9/gha-benchmark branch 2 times, most recently from 05c292d to 2d23652 Compare February 4, 2026 17:15
Base automatically changed from meyer9/gha-benchmark to main February 4, 2026 18:41
- Add compare-benchmarks.js script to parse and compare benchmark results
- Download baseline from main branch when running on PRs
- Post well-formatted markdown comment with comparison table
- Handle missing baseline gracefully (show results without comparison)
- Format metrics with proper units (gas/s, ms, s, etc.)
@meyer9 meyer9 force-pushed the meyer9/gha-benchmark-comparison branch from dd4045f to 75bad4a Compare February 9, 2026 19:30
refcell pushed a commit that referenced this pull request Feb 18, 2026
refcell pushed a commit that referenced this pull request Feb 18, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 5, 2026

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the Stale label Mar 5, 2026
mw2000 pushed a commit that referenced this pull request Mar 7, 2026
mw2000 pushed a commit that referenced this pull request Mar 9, 2026
@refcell refcell removed the Stale label Mar 9, 2026
mw2000 pushed a commit that referenced this pull request Mar 16, 2026
haardikk21 pushed a commit that referenced this pull request Mar 17, 2026
mw2000 pushed a commit that referenced this pull request Mar 19, 2026
@github-actions github-actions bot added the M-prevent-stale Meta: Not Stale label Mar 20, 2026
mw2000 pushed a commit that referenced this pull request Mar 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

M-prevent-stale Meta: Not Stale

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants