EXPB Benchmark Comparison

Run: View workflow run

Scenario: nethermind-halfpath-superblocks-fix-warmup-system-tx-routing-delay0s

Benchmarks Against master

Allocations

Detailed Fix Stats

BlockProcessingBenchmark Comparison: master vs fix/warmup-system-tx-routing

Master (baseline — no warmup optimization)

Warmup PR (fix/warmup-system-tx-routing)

Delta (Warmup PR vs master)

Key Takeaways

• EmptyBlock / SingleTransfer: No meaningful change (< 3 txs, pre-warming does not trigger) — within noise.
• Blocks with 3+ transactions show 20–40% improvement in mean processing time on the warmup PR.
  • The fix skips BuyGas, IncrementNonce, PayFees, PayValue, and PayRefund during the pre-warmer’s Warmup() calls, making the concurrent warmup pass substantially cheaper.
• ContractCall_200 shows the largest improvement (-40.2%).
  • Contract calls execute the EVM during warmup, so skipping gas/nonce/fee accounting has a larger relative impact.
• Allocations drop significantly in most scenarios (up to -790 KB for ContractCall_200), because the warmup path no longer runs the full gas accounting and fee payment code paths.

Redoing tests - seems like new commits have made some regression which is weird

Do not merge - new findings are made and working on proper adjustment

@claude This PR brings significant performance imporvement but it also increased the "Get" RocksDb method OwnTime per dottrace by 25k ms - making it not super optimal solution - any ideas how to make it even better?

Claude finished @kamilchodola's task in 5m 24s —— View job

Analysis: Why RocksDB "Get" OwnTime Increased — and How to Reduce It Further

• Read the changed files and understand the warmup mechanism
• Investigate RocksDB "Get" increase root cause
• Suggest optimizations

Root Cause of the Extra RocksDB Reads

The increase in RocksDB "Get" OwnTime is expected and is actually a sign the fix is working correctly. Here's why:

Before the fix (broken path through full Execute()):

The BuyGas and IncrementNonce calls were writing the sender's account to the WorldState dirty cache (balance subtracted, nonce incremented). More critically, for EIP-1559 transactions with non-zero MaxFeePerGas, ShouldValidateGas() returns true even with SkipValidation set — meaning the balance check in BuyGas could fail and abort the transaction before EVM ran. Those aborted transactions never touched the storage-heavy EVM path, so they produced fewer RocksDB reads.

After the fix (correct path through SystemTransactionProcessor):

BuyGas is a no-op → no early abort → all transactions now run EVM fully → more contract storage slots get read → more RocksDB misses in the warmup thread. These misses populate PreBlockCaches, which is the entire point of warming. The main processing thread then hits the cache instead of RocksDB, delivering the 20–40% speedup.

The 25k ms extra cost in warmup-thread RocksDB reads is paying for useful work that saves much more than 25k ms in the main thread. The EXPB numbers confirm this — the overall latencies are similar/better even with more warmup-side reads.

Ideas for Further Reduction

1. Batch-load statically-known addresses before EVM execution (highest impact)

In WarmupSingleTransaction(), the sender is already checked via AccountExists. But the recipient and block beneficiary (coinbase) are not pre-loaded before EVM starts. Every time EVM hits a cold account, it goes: dirty-cache miss → PreBlockCaches miss → RocksDB read. Converting those to a single batch load before EVM avoids sequential round-trips:

// In WarmupSingleTransaction, before Warmup() call:
worldState.WarmUp(tx.To);          // recipient
worldState.WarmUp(header.Beneficiary); // coinbase is accessed by PayFees on main thread

The coinbase in particular is accessed by every transaction on the main thread (in PayFees), but SystemTransactionProcessor.PayFees is a no-op so it's never warmed from the transaction warmup path. AddressWarmer may handle this, but worth verifying.

2. Use RocksDB MultiGet for access lists

worldState.WarmUp(tx.AccessList) (line 275 in BlockCachePreWarmer.cs) currently reads addresses/slots sequentially. For blocks with large EIP-2930 access lists or EIP-4337 user ops, this is N individual RocksDB Get calls. Routing these through a MultiGet would batch the I/O into a single call, reducing syscall overhead significantly on high-tx-count blocks.

3. Thundering herd on SeqlockCache for popular contracts

When multiple warmup threads process transactions that all call the same popular contract (e.g., USDC, a DEX), they can all get a cache miss on the same storage slots simultaneously and all go to RocksDB for the same key. The SeqlockCache is optimistic (no locking on read), so this race isn't prevented.

A simple "in-flight" set (or first-writer-wins pattern) on PreBlockCaches would let the first thread load the slot while others skip the duplicate DB read.

4. Skip EVM for pure ETH transfers

For transactions where tx.To has no code (!WorldState.HasCode(tx.To) && !tx.IsContractCreation), EVM execution only touches sender and recipient accounts — both of which are already known before EVM runs. The warmup could skip the full Warmup() call for these and just do two targeted WarmUp(address) calls, saving EVM setup overhead and the associated reads.

5. Speculative slot pre-loading from recent block history

For high-frequency contracts (DEXs, stablecoins), the storage slots accessed are largely stable across blocks. A sliding-window bloom filter tracking which (contract, slot) pairs were accessed in the last N blocks could be used to pre-load them speculatively before transaction warmup starts — turning cold misses into pre-warmed hits without needing EVM execution to discover them.

Summary

The 25k ms RocksDB increase is the correct trade-off — the warmup is doing more useful work than before. The highest-leverage quick win is #1 (batch pre-load recipient + coinbase before EVM) and #2 (MultiGet for access lists), both of which are small, focused changes to WarmupSingleTransaction() and WorldState.WarmUp(AccessList) respectively.