fix(testing/specs): Fix benchmark pre-alloc grouping#1708
Conversation
fselmo
left a comment
There was a problem hiding this comment.
lgtm! 🚀
One potential proper (follow-up) fix could be that BenchmarkTest inherits BlockchainTest.
I'd be curious to explore this and see if it makes sense but I think the current flow makes more sense with separation of concerns. It would sort of violate Liskov principle since BenchmarkTest seems more like a blockchain test factory - blocks is required in BlockchainTest but can be generated and isn't required for BenchmarkTest, etc.
I think I prefer to leave them separate tbh 🤔.
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## forks/osaka #1708 +/- ##
============================================
Coverage 86.07% 86.07%
============================================
Files 743 743
Lines 44078 44078
Branches 3894 3894
============================================
Hits 37938 37938
Misses 5659 5659
Partials 481 481
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
🗒️ Description
This PR addresses issues when generating pre-allocation groups with the
BenchmarkTestspec type.There were two main issues that had to be addressed for this:
BenchmarkTest.get_genesis_environmentnot usingBlockchainTest.get_genesis_environmentBenchmarkTest.get_genesis_environmentwas directly generating the genesis fromenvwithout adding the extra required argument thatBlockchainTest.get_genesis_environmentuses.One potential proper (follow-up) fix could be that
BenchmarkTestinheritsBlockchainTest.BenchmarkTest.generatemodifyingpreBenchmarkTestmodified an invariant that the pre-alloc grouping relied on:preis not touched after the test function completes.This resulted in, correctly, test filling failing on the test running step because the base
Allocdoes not implementdeploy_contractbecause there should not be any more deployments.But the problem was that
BenchmarkTestrequired theforkinstance in order to determine how many blocks to use (because of the transaction gas limit cap), and this value was only passed when thegenerateor theexecutionfunctions are called.Solution was to include
forkin theBaseTestclass and make it mandatory for the filler, execute, or any other code that instantiated a test spec, to pass the fork during the initialization:https://github.com/marioevz/execution-specs/blob/fc2bd475b8e7caf506b39ec6453e9d049e8f418e/packages/testing/src/execution_testing/cli/pytest_commands/plugins/filler/filler.py#L1353
Example in an unit test:
https://github.com/marioevz/execution-specs/blob/fc2bd475b8e7caf506b39ec6453e9d049e8f418e/packages/testing/src/execution_testing/cli/pytest_commands/plugins/filler/tests/test_prealloc_group.py#L56
This resulted in
forkbeing removed fromgenerate,executeand other instance methods withinBaseTest, which is a nice bonus.🔗 Related Issues or PRs
N/A.
✅ Checklist
toxchecks to avoid unnecessary CI fails, see also Code Standards and Enabling Pre-commit Checks:uvx tox -e statictype(scope):.mkdocs servelocally and verified the auto-generated docs for new tests in the Test Case Reference are correctly formatted.@ported_frommarker.Cute Animal Picture