feat(forks,plugins): Fork transition test marker by marioevz · Pull Request #1081 · ethereum/execution-spec-tests

marioevz · 2025-01-16T00:44:54Z

🗒️ Description

Adds two keyword parameters to the valid_at_transition_to marker:

subsequent_forks: bool: If set to True, the test will be generated for the transition fork to the specified fork name, but also for all subsequent transition forks. Default is False.
until: str | None: Specify an end to the fork transitions to be filed when using subsequent_forks=True. Implies subsequent_forks=True. Default is None.

E.g.

@pytest.mark.valid_at_transition_to("Cancun", subsequent_forks=True)

produces tests on ShanghaiToCancunAtTime15k and CancunToPragueAtTime15k, and any transition fork after that.

Also added unit tests in src/pytest_plugins/forks/tests/test_markers.py.

Implementing tests for EIP-7691 in a separate PR, as required by lack of coverage pointed out in this thread: https://discord.com/channels/595666850260713488/1329093054698881087

Validity Markers Refactor

All validity markers are now defined as classes that subclass a base validity marker class, ValidityMarker.

This allows simpler validity marker definition, and easier handling of keyword arguments if it is the case that a validity marker supports them. See ValidAtTransitionTo for an example of how such a validity marker is defined.

🔗 Related Issues

None

✅ Checklist

All: Set appropriate labels for the changes.
All: Considered squashing commits to improve commit history.
All: Added an entry to CHANGELOG.md.
All: Considered updating the online docs in the ./docs/ directory.
Tests: All converted JSON/YML tests from ethereum/tests have been added to converted-ethereum-tests.txt.
Tests: A PR with removal of converted JSON/YML tests from ethereum/tests have been opened.
Tests: Included the type and version of evm t8n tool used to locally execute test cases: e.g., ref with commit hash or geth 1.13.1-stable-3f40e65.
Tests: Ran mkdocs serve locally and verified the auto-generated docs for new tests in the Test Case Reference are correctly formatted.

danceratopz

How about:

@pytest.mark.valid_at_transition_to("Cancun", subsequent_transitions=True, until="Osaka")

marioevz · 2025-01-20T16:29:00Z

How about:

@pytest.mark.valid_at_transition_to("Cancun", subsequent_transitions=True, until="Osaka")

Addresed on the latest commit, which also includes a nice refactor of all validity markers into separate classes:

execution-spec-tests/src/pytest_plugins/forks/forks.py

Lines 711 to 747 in 73ab2c9

    
           class ValidAtTransitionTo(ValidityMarker, mutually_exclusive=True): 
        
               """Marker to specify that a test is valid at the transition to a specific fork.""" 
        
               def process_with_args( 
        
                   self, *fork_args, subsequent_forks: bool = False, until: str | None = None 
        
               ) -> Set[Fork]: 
        
                   """Process the fork arguments.""" 
        
                   forks: Set[Fork] = self.process_fork_arguments(*fork_args) 
        
                   until_forks: Set[Fork] | None = ( 
        
                       None if until is None else self.process_fork_arguments(until) 
        
                   ) 
        
                   if len(forks) == 0: 
        
                       pytest.fail( 
        
                           f"'{self.test_name}': Missing fork argument with 'valid_at_transition_to' " 
        
                           "marker." 
        
                       ) 
        
                   if len(forks) > 1: 
        
                       pytest.fail( 
        
                           f"'{self.test_name}': Too many forks specified to 'valid_at_transition_to' " 
        
                           "marker." 
        
                       ) 
        
                   resulting_set: Set[Fork] = set() 
        
                   for fork in forks: 
        
                       resulting_set |= transition_fork_to(fork) 
        
                       if subsequent_forks: 
        
                           for transition_forks in ( 
        
                               transition_fork_to(f) for f in self.all_forks if f > fork 
        
                           ): 
        
                               for transition_fork in transition_forks: 
        
                                   if transition_fork and ( 
        
                                       until_forks is None 
        
                                       or any(transition_fork <= until_fork for until_fork in until_forks) 
        
                                   ): 
        
                                       resulting_set.add(transition_fork) 
        
                   return resulting_set

src/ethereum_test_forks/tests/test_forks.py

src/pytest_plugins/forks/forks.py

src/ethereum_test_forks/tests/test_forks.py

winsvega

have a couple of questions

* refactor(forks): make `name()` an abstractmethod * refactor(forks): add helper methods to simplify fork comparisons

marioevz · 2025-01-22T16:06:52Z

@danceratopz merged your suggestion from the other PR, this one should be ready 😄

danceratopz

The refactoring of the validity markers is really nice! It adds much more structure to this plugin (I think this is one of the most complex parts of our pytest code). And really nice additional tests for these, btw! Few minor comments below.

docs/writing_tests/test_markers.md

src/ethereum_test_forks/base_fork.py

src/pytest_plugins/forks/forks.py

Co-authored-by: danceratopz <danceratopz@gmail.com>

* feat(forks): Adds `supports_blobs` method * fix(forks): Transition fork comparisons * refactor,fix(plugins/forks): `fork_transition_test` marker * docs: add documentation, changelog * refactor(plugins/forks): Validity markers as classes * fix: fork set handling * fix(plugins): config variable usage in other plugins * fix(plugins/forks): Self-documenting code for validity marker classes * whitelist * refactor(forks): suggestions for fork transition marker changes (ethereum#1104) * refactor(forks): make `name()` an abstractmethod * refactor(forks): add helper methods to simplify fork comparisons * Update src/pytest_plugins/forks/forks.py Co-authored-by: danceratopz <danceratopz@gmail.com> * Update src/pytest_plugins/forks/forks.py Co-authored-by: danceratopz <danceratopz@gmail.com> * Update src/pytest_plugins/forks/forks.py Co-authored-by: danceratopz <danceratopz@gmail.com> * fixup --------- Co-authored-by: danceratopz <danceratopz@gmail.com>

marioevz requested a review from danceratopz January 16, 2025 00:45

marioevz mentioned this pull request Jan 16, 2025

feat(tests): EIP-7691: Fork transition tests #1082

Merged

3 tasks

marioevz force-pushed the fork-transition-test-marker branch from 420e221 to 270870b Compare January 16, 2025 12:51

danceratopz reviewed Jan 16, 2025

View reviewed changes

marioevz force-pushed the fork-transition-test-marker branch from 0cf221b to cef281f Compare January 20, 2025 15:48

marioevz mentioned this pull request Jan 20, 2025

Prague Devnet-5 Release 1.3.0 Tracker #1094

Closed

16 tasks

marioevz added 9 commits January 20, 2025 21:53

feat(forks): Adds supports_blobs method

98c77fb

fix(forks): Transition fork comparisons

19954d2

refactor,fix(plugins/forks): fork_transition_test marker

8c3c955

docs: add documentation, changelog

ae634b9

refactor(plugins/forks): Validity markers as classes

b3daf3c

fix: fork set handling

3a9e005

fix(plugins): config variable usage in other plugins

41fa523

fix(plugins/forks): Self-documenting code for validity marker classes

eceb88f

whitelist

53754e9

marioevz force-pushed the fork-transition-test-marker branch from 10905cb to 53754e9 Compare January 20, 2025 21:53

winsvega reviewed Jan 21, 2025

View reviewed changes

src/ethereum_test_forks/tests/test_forks.py Show resolved Hide resolved

winsvega reviewed Jan 21, 2025

View reviewed changes

src/pytest_plugins/forks/forks.py Show resolved Hide resolved

winsvega reviewed Jan 21, 2025

View reviewed changes

src/ethereum_test_forks/tests/test_forks.py Show resolved Hide resolved

winsvega reviewed Jan 21, 2025

View reviewed changes

This was referenced Jan 22, 2025

refactor(forks): suggestions for fork transition marker changes #1104

Merged

feat(fill): make fork cli args case-insensitive #1106

Closed

refactor(forks): suggestions for fork transition marker changes (#1104)

f7b652b

* refactor(forks): make `name()` an abstractmethod * refactor(forks): add helper methods to simplify fork comparisons

Merge branch 'main' into fork-transition-test-marker

1826bd7

danceratopz approved these changes Jan 22, 2025

View reviewed changes

marioevz and others added 3 commits January 22, 2025 11:07

Update src/pytest_plugins/forks/forks.py

d8ce30f

Co-authored-by: danceratopz <danceratopz@gmail.com>

Update src/pytest_plugins/forks/forks.py

3c1c109

Co-authored-by: danceratopz <danceratopz@gmail.com>

Update src/pytest_plugins/forks/forks.py

5a9fb9a

Co-authored-by: danceratopz <danceratopz@gmail.com>

fixup

a873632

marioevz merged commit 607d1b6 into main Jan 22, 2025

marioevz deleted the fork-transition-test-marker branch January 22, 2025 17:35

danceratopz mentioned this pull request Jan 29, 2025

fix(execute): fix selected forks incompatibility #1157

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(forks,plugins): Fork transition test marker#1081

feat(forks,plugins): Fork transition test marker#1081
marioevz merged 15 commits intomainfrom
fork-transition-test-marker

marioevz commented Jan 16, 2025 •

edited

Loading

Uh oh!

danceratopz left a comment

Uh oh!

marioevz commented Jan 20, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

winsvega left a comment •

edited

Loading

Uh oh!

marioevz commented Jan 22, 2025

Uh oh!

danceratopz left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

marioevz commented Jan 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🗒️ Description

Validity Markers Refactor

🔗 Related Issues

✅ Checklist

Uh oh!

danceratopz left a comment

Choose a reason for hiding this comment

Uh oh!

marioevz commented Jan 20, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

winsvega left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

marioevz commented Jan 22, 2025

Uh oh!

danceratopz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

marioevz commented Jan 16, 2025 •

edited

Loading

winsvega left a comment •

edited

Loading