Cannot sync chain from scratch due to block import error #11891
Description
Is there an existing issue?
- I have searched the existing issues
Experiencing problems? Have you tried our Stack Exchange first?
- This is not a support question.
Description of bug
Description
Since we've upgraded from polkadot-v0.9.19 to polkadot-v0.9.24, we cannot sync a node from scratch.
It always get stuck trying to import block 1086612 but it fails due to WASM call stack exhaustion.
If we downgrade to latest client before polkadot-v0.9.24 upgrade, the block is imported successfully.
We've upgraded to polkadot-v0.9.26 but error still persists.
Log
Here is the log from one of our collators (about 2 weeks old):
collator_20220707.txt
What Was Investigated So Far
From what I can see in the log, an EVM-related call is being executed and it ends up with call stack exhausted error.
I followed the stacktrace and my guess is that it's some contract that perhaps has infinite recursion (or calls lots of functions) and used to fail due to gas limit. I cannot confirm that but according to blockscout, 1 EVM transaction is marked as failed.
In the evm repo I found this:
Opcode::CALL => system::call(state, CallScheme::Call, handler),
Opcode::CALLCODE => system::call(state, CallScheme::CallCode, handler),
Opcode::DELEGATECALL => system::call(state, CallScheme::DelegateCall, handler),
Opcode::STATICCALL => system::call(state, CallScheme::StaticCall, handler),
hence my guess it's a recursive call (this part constantly repeats itself in the stacktrace).
My guess is that with latest upgrade(s), the callstack depth was reduced for wasmi and that's why it fails with new but works with old client. I haven't been able to verify this since I haven't managed to find where this is configured. (might be totally wrong, of course).
I did find this, although it relates to wasmtime:
https://bytemeta.vip/repo/bytecodealliance/wasmtime/issues/4214
We also have regalloc2 as dependency, not sure if it is relevant though.
Zoomed Part of the Log
The following is taken from the provided log to point out repeated calls:
2022-07-07 20:10:15 [Parachain] ⚙️ Syncing, target=#1389107 (7 peers), best: #1086611 (0x1c8b…6b8f), finalized #0 (0x9eb7…29c6), ⬇ 3.9MiB/s ⬆ 17.1kiB/s
2022-07-07 20:10:16 [Parachain] 330 storage transactions are left open by the runtime. Those will be rolled back.
2022-07-07 20:10:16 [Parachain] 330 storage transactions are left open by the runtime. Those will be rolled back.
2022-07-07 20:10:16 [Parachain] Block prepare storage changes error: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: call stack exhausted
WASM backtrace:
0: <unknown>!evm_runtime::Runtime::run::h9658b50101d9a1f1
1: 0x11a23f - <unknown>!evm::executor::stack::executor::StackExecutor<S,P>::call_inner::hdfd6092bf3da0b18
2: 0x11d364 - <unknown>!<evm::executor::stack::executor::StackExecutor<S,P> as evm_runtime::handler::Handler>::call::h69ec87fd06dc45ef
3: 0xb3968 - <unknown>!evm_runtime::eval::system::call::h87659850f0b00992
4: 0x2300be - <unknown>!evm_runtime::eval::eval::hc1a492a7d69a6d92
5: 0xe9d05 - <unknown>!evm_runtime::Runtime::run::h9658b50101d9a1f1
6: 0x11a23f - <unknown>!evm::executor::stack::executor::StackExecutor<S,P>::call_inner::hdfd6092bf3da0b18
7: 0x11d364 - <unknown>!<evm::executor::stack::executor::StackExecutor<S,P> as evm_runtime::handler::Handler>::call::h69ec87fd06dc45ef
8: 0xb3968 - <unknown>!evm_runtime::eval::system::call::h87659850f0b00992
9: 0x2300be - <unknown>!evm_runtime::eval::eval::hc1a492a7d69a6d92
10: 0xe9d05 - <unknown>!evm_runtime::Runtime::run::h9658b50101d9a1f1
11: 0x11a23f - <unknown>!evm::executor::stack::executor::StackExecutor<S,P>::call_inner::hdfd6092bf3da0b18
12: 0x11d364 - <unknown>!<evm::executor::stack::executor::StackExecutor<S,P> as evm_runtime::handler::Handler>::call::h69ec87fd06dc45ef
13: 0xb3968 - <unknown>!evm_runtime::eval::system::call::h87659850f0b00992
14: 0x2300be - <unknown>!evm_runtime::eval::eval::hc1a492a7d69a6d92
...
...
...
1634: 0x2300be - <unknown>!evm_runtime::eval::eval::hc1a492a7d69a6d92
1635: 0xe9d05 - <unknown>!evm_runtime::Runtime::run::h9658b50101d9a1f1
1636: 0x11a23f - <unknown>!evm::executor::stack::executor::StackExecutor<S,P>::call_inner::hdfd6092bf3da0b18
1637: 0x11d364 - <unknown>!<evm::executor::stack::executor::StackExecutor<S,P> as evm_runtime::handler::Handler>::call::h69ec87fd06dc45ef
1638: 0xb3968 - <unknown>!evm_runtime::eval::system::call::h87659850f0b00992
1639: 0x2300be - <unknown>!evm_runtime::eval::eval::hc1a492a7d69a6d92
1640: 0xe9d05 - <unknown>!evm_runtime::Runtime::run::h9658b50101d9a1f1
1641: 0x11a23f - <unknown>!evm::executor::stack::executor::StackExecutor<S,P>::call_inner::hdfd6092bf3da0b18
1642: 0x11d364 - <unknown>!<evm::executor::stack::executor::StackExecutor<S,P> as evm_runtime::handler::Handler>::call::h69ec87fd06dc45ef
1643: 0xb3968 - <unknown>!evm_runtime::eval::system::call::h87659850f0b00992
1644: 0x2300be - <unknown>!evm_runtime::eval::eval::hc1a492a7d69a6d92
1645: 0xe9d05 - <unknown>!evm_runtime::Runtime::run::h9658b50101d9a1f1
1646: 0x11a23f - <unknown>!evm::executor::stack::executor::StackExecutor<S,P>::call_inner::hdfd6092bf3da0b18
1647: 0x119716 - <unknown>!evm::executor::stack::executor::StackExecutor<S,P>::transact_call::habd7cd23fb2ca77f
1648: 0xdb284 - <unknown>!<pallet_evm::runner::stack::Runner<T> as pallet_evm::runner::Runner<T>>::call::hce6b1473ecdc4939
1649: 0xe5235 - <unknown>!pallet_ethereum::<impl pallet_ethereum::pallet::Pallet<T>>::apply_validated_transaction::heb98eb14d23df8cc
1650: 0xecd87 - <unknown>!<pallet_ethereum::pallet::Call<T> as frame_support::traits::dispatch::UnfilteredDispatchable>::dispatch_bypass_filter::h747abd25cb98a9d2
1651: 0x9f7ff - <unknown>!<astar_runtime::Call as frame_support::traits::dispatch::UnfilteredDispatchable>::dispatch_bypass_filter::h77ff5d2bdf975aed
1652: 0x9cac5 - <unknown>!<astar_runtime::Call as sp_runtime::traits::Dispatchable>::dispatch::h0b11be2c27055c06
1653: 0x9ca2f - <unknown>!<astar_runtime::Call as fp_self_contained::SelfContainedCall>::apply_self_contained::he16e52be80a30fbe
1654: 0x22ce7f - <unknown>!<fp_self_contained::checked_extrinsic::CheckedExtrinsic<AccountId,Call,Extra,SelfContainedSignedInfo> as sp_runtime::traits::Applyable>::apply::h1c69cda9a66cc589
1655: 0x547e5 - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::apply_extrinsic::h775d70717e8f3dba
1656: 0x52d76 - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::execute_block::h019662ea5c7bf877
1657: 0x175ca8 - <unknown>!Core_execute_block
Steps to reproduce
Issue can be reproduced by downloading latest client from here.
To be on the safe side, the issue was first reported with the v4.7.0 client so I recommend to try out with that one.
Use the following command to start the node:
astar-collator --chain astar --parachain-id 2006 --name "CannotImport" --execution wasm --prometheus-external --ws-port 9944 --port 33333 --state-cache-size 1 --rpc-port 29933 --prometheus-port 39616