System information
Erigon version: 3.4.1 (affected), 3.4.2-aebbd15b (NOT fully resolved — see updates below)
OS & Version: Fedora Linux (aarch64, AWS Graviton)
Commit hash: aebbd15 (3.4.2)
Erigon Command (with flags/config):
/usr/bin/erigon --datadir /var/lib/erigon --prune.mode=archive --private.api.addr=127.0.0.1:9090 --http --http.compression --http.addr=0.0.0.0 --http.api eth,net,web3,engine,debug,trace,erigon --ws --ws.compression --rpc.gascap 0 --rpc.returndata.limit=10000000 --rpc.blockrange.limit=10000 --authrpc.jwtsecret=/etc/jwt.hex --db.size.limit=8TB --db.pagesize=16k --torrent.download.rate=32mb --torrent.upload.rate=8mb --downloader.disable.ipv6=true --pprof --pprof.port=6060 --log.dir.verbosity=4 --nat=extip:xx.xx.xx.xx
Consensus Layer: Erigon internal (embedded)
Consensus Layer Command (with flags/config): N/A
Chain/Network: Ethereum mainnet
Status: Resolved on 3.4.2 NOT fully resolved — recurs on 3.4.2 after reorg
This issue was observed on Erigon 3.4.1 on ARM64 and appears resolved on 3.4.2. Reporting for visibility in case other ARM64 users on 3.4.1 encounter the same block.
Update (May 30): The issue recurs on v3.4.2 after approximately 12 hours. A new reorg at block 25205371 triggered the same state-leak bug, producing a gas mismatch with diff=71,016. PR #21157 (findExecutedDiffsetAtHeight) does not fully address the root cause. Chaindata rebuild is only a temporary fix. See comments below for details.
Our x86_64 nodes running v3.4.0 are not affected. We recommend staying on v3.4.0 until this is properly fixed.
Expected behaviour
Block 25202264 should be processed successfully, as it is on x86_64 nodes running the same Erigon version.
Actual behaviour
Erigon on ARM64 (AWS Graviton) fails to process blocks after reorg unwinds involving CREATE2 transactions, with a gas used mismatch.
First occurrence (v3.4.1): block 25202264
[WARN] gas used mismatch block=25202264 header=33888325 execution=33961963 diff=73638 txCount=338 receiptCount=338
Second occurrence (v3.4.2): block 25205371, ~12 hours after chaindata rebuild
[INFO] [4/6 Execution] Unwind Execution from=25205371 to=25205370
[WARN] gas used mismatch block=25205371 header=31783201 execution=31854217 diff=71016 txCount=149 receiptCount=149
The node gets stuck at the affected block — restarts do not resolve it, as the same divergence occurs deterministically. Chaindata rebuild temporarily fixes it, but the next reorg with a CREATE2 tx triggers it again.
Two x86_64 nodes running v3.4.0 process these blocks without issues and have never encountered the bug.
As noted in the comments, this is likely not ARM64-specific — it is a state-leak bug in unwindExec3 that depends on which reorg the node encounters, not the CPU architecture.
Steps to reproduce the behaviour
- Run Erigon 3.4.1 or 3.4.2 on any architecture
- Wait for a chain reorg where the unwound block contains a
CREATE2 tx
- Node halts with the gas mismatch warning and fails to advance
Backtrace
First occurrence: block 25202264 (v3.4.1) — transaction gas details (first 36 of 338 transactions)
[WARN] gas used mismatch block=25202264 header=33888325 execution=33961963 diff=73638 txCount=338 receiptCount=338
tx gas detail block=25202264 txIdx=0 txHash=0x369a3278a7a6bc1f gasUsed=122183 cumGasUsed=122183 computedCumGas=122183 status=1
tx gas detail block=25202264 txIdx=1 txHash=0x7011e9a36c5c0023 gasUsed=122052 cumGasUsed=244235 computedCumGas=244235 status=1
tx gas detail block=25202264 txIdx=2 txHash=0xaa6a5c3034789c98 gasUsed=161752 cumGasUsed=405987 computedCumGas=405987 status=1
tx gas detail block=25202264 txIdx=3 txHash=0xfe572dc99bea3715 gasUsed=122489 cumGasUsed=528476 computedCumGas=528476 status=1
tx gas detail block=25202264 txIdx=4 txHash=0x7f7fd4f92f87246d gasUsed=178458 cumGasUsed=706934 computedCumGas=706934 status=1
tx gas detail block=25202264 txIdx=5 txHash=0xea50e9ab76497b21 gasUsed=41309 cumGasUsed=748243 computedCumGas=748243 status=1
tx gas detail block=25202264 txIdx=6 txHash=0xe88a7b68b97f5591 gasUsed=169036 cumGasUsed=917279 computedCumGas=917279 status=1
tx gas detail block=25202264 txIdx=7 txHash=0x28a797e9a4082f34 gasUsed=121261 cumGasUsed=1038540 computedCumGas=1038540 status=1
tx gas detail block=25202264 txIdx=8 txHash=0x0ea3ac517d141dab gasUsed=27938 cumGasUsed=1066478 computedCumGas=1066478 status=1
tx gas detail block=25202264 txIdx=9 txHash=0xe027ea449576d268 gasUsed=49154 cumGasUsed=1115632 computedCumGas=1115632 status=1
tx gas detail block=25202264 txIdx=10 txHash=0xd78854d46bd0c453 gasUsed=156503 cumGasUsed=1272135 computedCumGas=1272135 status=1
tx gas detail block=25202264 txIdx=11 txHash=0x180fdd94113c0456 gasUsed=122716 cumGasUsed=1394851 computedCumGas=1394851 status=1
tx gas detail block=25202264 txIdx=12 txHash=0x7086cd3cc5aa8cad gasUsed=21000 cumGasUsed=1415851 computedCumGas=1415851 status=1
tx gas detail block=25202264 txIdx=13 txHash=0xefc6c2836bec347f gasUsed=45083 cumGasUsed=1460934 computedCumGas=1460934 status=1
tx gas detail block=25202264 txIdx=14 txHash=0x8d2f62a7eac1cf07 gasUsed=55660 cumGasUsed=1516594 computedCumGas=1516594 status=1
tx gas detail block=25202264 txIdx=15 txHash=0x015b11d8396915d2 gasUsed=47307 cumGasUsed=1563901 computedCumGas=1563901 status=1
tx gas detail block=25202264 txIdx=16 txHash=0x7171b72411b742f1 gasUsed=289858 cumGasUsed=1853759 computedCumGas=1853759 status=1
tx gas detail block=25202264 txIdx=17 txHash=0x98c2804192273d06 gasUsed=203190 cumGasUsed=2056949 computedCumGas=2056949 status=1
tx gas detail block=25202264 txIdx=18 txHash=0x6b022441dcc83a7c gasUsed=358740 cumGasUsed=2415689 computedCumGas=2415689 status=1
tx gas detail block=25202264 txIdx=19 txHash=0x425d3e705481d2e3 gasUsed=84363 cumGasUsed=2500052 computedCumGas=2500052 status=1
tx gas detail block=25202264 txIdx=20 txHash=0xfd84813bd9673246 gasUsed=41309 cumGasUsed=2541361 computedCumGas=2541361 status=1
tx gas detail block=25202264 txIdx=21 txHash=0x5076a4658204432c gasUsed=21000 cumGasUsed=2562361 computedCumGas=2562361 status=1
tx gas detail block=25202264 txIdx=22 txHash=0xb0d52511632291d2 gasUsed=21000 cumGasUsed=2583361 computedCumGas=2583361 status=1
tx gas detail block=25202264 txIdx=23 txHash=0x508b978e57e5acc3 gasUsed=48549 cumGasUsed=2631910 computedCumGas=2631910 status=1
tx gas detail block=25202264 txIdx=24 txHash=0x10ef532e80c8c5fa gasUsed=220644 cumGasUsed=2852554 computedCumGas=2852554 status=1
tx gas detail block=25202264 txIdx=25 txHash=0x0437d8c685ef3e2d gasUsed=48271 cumGasUsed=2900825 computedCumGas=2900825 status=1
tx gas detail block=25202264 txIdx=26 txHash=0xe096d7eedddf89ed gasUsed=21000 cumGasUsed=2921825 computedCumGas=2921825 status=1
tx gas detail block=25202264 txIdx=27 txHash=0x5abcf2671c4a0925 gasUsed=40360 cumGasUsed=2962185 computedCumGas=2962185 status=1
tx gas detail block=25202264 txIdx=28 txHash=0xe0807809f4219ffe gasUsed=46166 cumGasUsed=3008351 computedCumGas=3008351 status=1
tx gas detail block=25202264 txIdx=29 txHash=0x7f086840153882b7 gasUsed=506130 cumGasUsed=3514481 computedCumGas=3514481 status=1
tx gas detail block=25202264 txIdx=30 txHash=0xa2abaa5d4dbe210a gasUsed=63197 cumGasUsed=3577678 computedCumGas=3577678 status=1
tx gas detail block=25202264 txIdx=31 txHash=0x8d23cf637736640a gasUsed=123683 cumGasUsed=3701361 computedCumGas=3701361 status=1
tx gas detail block=25202264 txIdx=32 txHash=0xad41c404070f6d6d gasUsed=180059 cumGasUsed=3881420 computedCumGas=3881420 status=1
tx gas detail block=25202264 txIdx=33 txHash=0x6ea07b8ebd7ae648 gasUsed=48537 cumGasUsed=3929957 computedCumGas=3929957 status=1
tx gas detail block=25202264 txIdx=34 txHash=0x4fd6491ce75f287f gasUsed=63221 cumGasUsed=3993178 computedCumGas=3993178 status=1
tx gas detail block=25202264 txIdx=35 txHash=0x36eb18dd119ef4ca gasUsed=78942 cumGasUsed=4072120 computedCumGas=4072120 status=1
(Log truncated — block has 338 transactions total, first 36 shown. All individual computedCumGas values match cumGasUsed for the shown transactions.)
System information
Erigon version: 3.4.1 (affected), 3.4.2-aebbd15b (NOT fully resolved — see updates below)
OS & Version: Fedora Linux (aarch64, AWS Graviton)
Commit hash: aebbd15 (3.4.2)
Erigon Command (with flags/config):
Consensus Layer: Erigon internal (embedded)
Consensus Layer Command (with flags/config): N/A
Chain/Network: Ethereum mainnet
Status:
Resolved on 3.4.2NOT fully resolved — recurs on 3.4.2 after reorgThis issue was observed on Erigon 3.4.1 on ARM64 and appears resolved on 3.4.2. Reporting for visibility in case other ARM64 users on 3.4.1 encounter the same block.Update (May 30): The issue recurs on v3.4.2 after approximately 12 hours. A new reorg at block 25205371 triggered the same state-leak bug, producing a gas mismatch with diff=71,016. PR #21157 (
findExecutedDiffsetAtHeight) does not fully address the root cause. Chaindata rebuild is only a temporary fix. See comments below for details.Our x86_64 nodes running v3.4.0 are not affected. We recommend staying on v3.4.0 until this is properly fixed.
Expected behaviour
Block 25202264 should be processed successfully, as it is on x86_64 nodes running the same Erigon version.
Actual behaviour
Erigon on ARM64 (AWS Graviton) fails to process blocks after reorg unwinds involving
CREATE2transactions, with a gas used mismatch.First occurrence (v3.4.1): block 25202264
Second occurrence (v3.4.2): block 25205371, ~12 hours after chaindata rebuild
The node gets stuck at the affected block — restarts do not resolve it, as the same divergence occurs deterministically. Chaindata rebuild temporarily fixes it, but the next reorg with a
CREATE2tx triggers it again.Two x86_64 nodes running v3.4.0 process these blocks without issues and have never encountered the bug.
As noted in the comments, this is likely not ARM64-specific — it is a state-leak bug in
unwindExec3that depends on which reorg the node encounters, not the CPU architecture.Steps to reproduce the behaviour
CREATE2txBacktrace
First occurrence: block 25202264 (v3.4.1) — transaction gas details (first 36 of 338 transactions)
(Log truncated — block has 338 transactions total, first 36 shown. All individual
computedCumGasvalues matchcumGasUsedfor the shown transactions.)