Skip to content

fix(testing/specs): Fix benchmark pre-alloc grouping#1708

Merged
marioevz merged 4 commits into
ethereum:forks/osakafrom
marioevz:fix-benchmarking-pre-alloc-filling
Oct 29, 2025
Merged

fix(testing/specs): Fix benchmark pre-alloc grouping#1708
marioevz merged 4 commits into
ethereum:forks/osakafrom
marioevz:fix-benchmarking-pre-alloc-filling

Conversation

@marioevz

@marioevz marioevz commented Oct 28, 2025

Copy link
Copy Markdown
Member

🗒️ Description

This PR addresses issues when generating pre-allocation groups with the BenchmarkTest spec type.

There were two main issues that had to be addressed for this:

BenchmarkTest.get_genesis_environment not using BlockchainTest.get_genesis_environment

BenchmarkTest.get_genesis_environment was directly generating the genesis from env without adding the extra required argument that BlockchainTest.get_genesis_environment uses.

One potential proper (follow-up) fix could be that BenchmarkTest inherits BlockchainTest.

BenchmarkTest.generate modifying pre

BenchmarkTest modified an invariant that the pre-alloc grouping relied on: pre is not touched after the test function completes.

This resulted in, correctly, test filling failing on the test running step because the base Alloc does not implement deploy_contract because there should not be any more deployments.

But the problem was that BenchmarkTest required the fork instance in order to determine how many blocks to use (because of the transaction gas limit cap), and this value was only passed when the generate or the execution functions are called.

Solution was to include fork in the BaseTest class and make it mandatory for the filler, execute, or any other code that instantiated a test spec, to pass the fork during the initialization:
https://github.com/marioevz/execution-specs/blob/fc2bd475b8e7caf506b39ec6453e9d049e8f418e/packages/testing/src/execution_testing/cli/pytest_commands/plugins/filler/filler.py#L1353
Example in an unit test:
https://github.com/marioevz/execution-specs/blob/fc2bd475b8e7caf506b39ec6453e9d049e8f418e/packages/testing/src/execution_testing/cli/pytest_commands/plugins/filler/tests/test_prealloc_group.py#L56

This resulted in fork being removed from generate, execute and other instance methods within BaseTest, which is a nice bonus.

🔗 Related Issues or PRs

N/A.

✅ Checklist

  • All: Ran fast tox checks to avoid unnecessary CI fails, see also Code Standards and Enabling Pre-commit Checks:
    uvx tox -e static
  • All: PR title adheres to the repo standard - it will be used as the squash commit message and should start type(scope):.
  • All: Considered adding an entry to CHANGELOG.md.
  • All: Considered updating the online docs in the ./docs/ directory.
  • All: Set appropriate labels for the changes (only maintainers can apply labels).
  • Tests: Ran mkdocs serve locally and verified the auto-generated docs for new tests in the Test Case Reference are correctly formatted.
  • Tests: For PRs implementing a missed test case, update the post-mortem document to add an entry the list.
  • Ported Tests: All converted JSON/YML tests from ethereum/tests or tests/static have been assigned @ported_from marker.

Cute Animal Picture

Put a link to a cute animal picture inside the parenthesis-->

@marioevz marioevz changed the title fix(testing/specs): Fix benchmark spec genesis fix(testing/specs): Fix benchmark pre-alloc grouping Oct 28, 2025
@marioevz marioevz requested a review from fselmo October 28, 2025 20:21
@marioevz marioevz marked this pull request as ready for review October 28, 2025 21:07

@fselmo fselmo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm! 🚀

One potential proper (follow-up) fix could be that BenchmarkTest inherits BlockchainTest.

I'd be curious to explore this and see if it makes sense but I think the current flow makes more sense with separation of concerns. It would sort of violate Liskov principle since BenchmarkTest seems more like a blockchain test factory - blocks is required in BlockchainTest but can be generated and isn't required for BenchmarkTest, etc.

I think I prefer to leave them separate tbh 🤔.

@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.07%. Comparing base (de8973e) to head (fc2bd47).
⚠️ Report is 1 commits behind head on forks/osaka.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@             Coverage Diff              @@
##           forks/osaka    #1708   +/-   ##
============================================
  Coverage        86.07%   86.07%           
============================================
  Files              743      743           
  Lines            44078    44078           
  Branches          3894     3894           
============================================
  Hits             37938    37938           
  Misses            5659     5659           
  Partials           481      481           
Flag Coverage Δ
unittests 86.07% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@marioevz marioevz merged commit 15b5cfd into ethereum:forks/osaka Oct 29, 2025
11 checks passed
@marioevz marioevz deleted the fix-benchmarking-pre-alloc-filling branch October 29, 2025 02:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants