feat(tests): add filler_to_python converter and port static tests#2563
Conversation
dd1c3a4 to
dee3d4c
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## forks/amsterdam #2563 +/- ##
===================================================
- Coverage 86.35% 86.24% -0.12%
===================================================
Files 599 599
Lines 36904 36984 +80
Branches 3771 3795 +24
===================================================
+ Hits 31868 31895 +27
- Misses 4485 4525 +40
- Partials 551 564 +13
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
|
This is how tests/ported_static/stBugs/test_staticcall_createfails.py looks now: """
Test_staticcall_createfails.
Ported from:
state_tests/stBugs/staticcall_createfailsFiller.json
"""
import pytest
from execution_testing import (
EOA,
Account,
Address,
Alloc,
Environment,
StateTestFiller,
Transaction,
)
from execution_testing.forks import Fork
from execution_testing.specs.static_state.expect_section import (
resolve_expect_post,
)
from execution_testing.vm import Op
REFERENCE_SPEC_GIT_PATH = "N/A"
REFERENCE_SPEC_VERSION = "N/A"
TX_DATA = [
"000000000000000000000000c94f5374fce5edbc8e2a8697c15331677e6ebf0b",
"000000000000000000000000d94f5374fce5edbc8e2a8697c15331677e6ebf0b",
]
TX_GAS = [120000]
TX_VALUE = [0]
def _tx_data(d: int) -> bytes:
"""Convert TX_DATA[d] hex string to bytes."""
return bytes.fromhex(TX_DATA[d])
@pytest.mark.ported_from(
["state_tests/stBugs/staticcall_createfailsFiller.json"],
)
@pytest.mark.valid_from("Cancun")
@pytest.mark.parametrize(
"d, g, v",
[
pytest.param(
0,
0,
0,
id="d0",
),
pytest.param(
1,
0,
0,
id="d1",
),
],
)
@pytest.mark.pre_alloc_mutable
def test_staticcall_createfails(
state_test: StateTestFiller,
pre: Alloc,
fork: Fork,
d: int,
g: int,
v: int,
) -> None:
"""Test_staticcall_createfails."""
coinbase = Address("0x1000000000000000000000000000000000000000")
contract_0 = Address("0xb94f5374fce5edbc8e2a8697c15331677e6ebf0b")
contract_1 = Address("0xc94f5374fce5edbc8e2a8697c15331677e6ebf0b")
contract_2 = Address("0xd94f5374fce5edbc8e2a8697c15331677e6ebf0b")
sender = EOA(
key=0x45A915E4D060149EB4365960E6A7A45F334393093061116B197E3240065FF2D8
)
env = Environment(
fee_recipient=coinbase,
number=1,
timestamp=1000,
prev_randao=0x20000,
difficulty=0x20000,
base_fee_per_gas=10,
gas_limit=23826461031063688,
)
pre[sender] = Account(balance=0x38BEEC8FEECA2598)
# Source: lll
# { [[1]] (STATICCALL 70000 (CALLDATALOAD 0) 0 0 0 0) }
contract_0 = pre.deploy_contract( # noqa: F841
code=Op.SSTORE(
key=0x1,
value=Op.STATICCALL(
gas=0x11170,
address=Op.CALLDATALOAD(offset=0x0),
args_offset=0x0,
args_size=0x0,
ret_offset=0x0,
ret_size=0x0,
),
)
+ Op.STOP,
storage={1: 1},
nonce=63,
address=Address("0xb94f5374fce5edbc8e2a8697c15331677e6ebf0b"), # noqa: E501
)
# Source: lll
# { (MSTORE 1 1) [[2]] (CREATE 1 1 1) }
contract_1 = pre.deploy_contract( # noqa: F841
code=Op.MSTORE(offset=0x1, value=0x1)
+ Op.SSTORE(key=0x2, value=Op.CREATE(value=0x1, offset=0x1, size=0x1))
+ Op.STOP,
nonce=63,
address=Address("0xc94f5374fce5edbc8e2a8697c15331677e6ebf0b"), # noqa: E501
)
# Source: raw
# 0x60006000f0
contract_2 = pre.deploy_contract( # noqa: F841
code=Op.PUSH1[0x0] * 2 + Op.CREATE,
nonce=63,
address=Address("0xd94f5374fce5edbc8e2a8697c15331677e6ebf0b"), # noqa: E501
)
expect_entries_: list[dict] = [
{
"indexes": {"data": -1, "gas": -1, "value": -1},
"network": [">=Cancun"],
"result": {
contract_0: Account(storage={1: 0}),
Address(
"0x1d0384eb7c2b1a9d9862c8e180f9e4d1696a2a8e"
): Account.NONEXISTENT,
},
},
]
post, _exc = resolve_expect_post(expect_entries_, d, g, v, fork)
tx = Transaction(
sender=sender,
to=contract_0,
data=_tx_data(d),
gas_limit=TX_GAS[g],
gas_price=10,
error=_exc,
)
state_test(env=env, pre=pre, post=post, tx=tx) |
|
the CI fails on the test |
spencer-tb
left a comment
There was a problem hiding this comment.
The slow pytest marker was not being applied. This seems to fix it on my end!
spencer-tb
left a comment
There was a problem hiding this comment.
Skip difficulty when prev_randao is set (post-merge forks) and omit default gas_price=10 as its what we set in the framework (less noise). Matches original script.
filler_to_python converter and port static tests
Is that for a specific test? In my code, it does add it for example to tests/ported_static/stAttackTest/test_contract_creation_spam.py |
Ahh these were added by us manually from last script! They are safe to add as slow imo! |
Commit Review:
|
|
Confirm that not slow tests matched the fixtures. Running the slow ones now. |
|
Also with slow, the fixtures match 100% |
Ok, I thought that only the ones marked as slow in the test YAML/JSON itself should be marked as such, but it seems these other categories, were previously hardcoded as always slow? |
|
I was wrong believing blindly the AI agent about the reason emptyBlobhashList was failing. The simple fix from Spencer fixed it. |
|
It seems that everything is ok now, with the advantage of having a script that is easier to improve without introducing problems in other tests in unexpected ways. |
marioevz
left a comment
There was a problem hiding this comment.
Amazing work! The resulting tests keep improving and are more readable now. Thanks!
🗒️ Description
New
filler_to_pythonscript that converts static YAML/JSON filler filesdirectly into Python test files, replacing the previous
fixture_to_python.pyapproach that required compiled fixtures as anintermediate step.
Key improvement: Single source of truth. The new script works from
filler files only — no compiled fixtures needed. It reuses the existing
Pydantic models from
execution_testing.specs.static_statefor parsing,tag resolution, and code compilation, then generates Python tests via a
Jinja2 template.
Results:
postVerifications) between static fill and Python fill(
emptyBlobhashList—Transactionrejects emptyblob_versioned_hashes)tox -e staticNew files
scripts/filler_to_python/__init__.pyscripts/filler_to_python/__main__.pyscripts/filler_to_python/analyzer.pyscripts/filler_to_python/ir.pyscripts/filler_to_python/render.pyrender_test()scripts/filler_to_python/templates/state_test.py.j2Modified files
packages/testing/.../expect_section.pyresolve_expect_post()andresolve_expect_post_fork()runtime helpersscripts/compare_fixtures.pytests/ported_static/🔗 Related Issues or PRs
Depends on #2552 (
--post-verificationsflag).✅ Checklist
toxchecks to avoid unnecessary CI fails:uvx tox -e static@ported_frommarker.Cute Animal Picture