Summary

This PR adds two cooperating tools:

1. eest_stateful_generator.py — a one-shot orchestrator that:

   • boots a Nethermind node from a snapshot using OverlayFS (ephemeral upper),
   • pre-seeds a few engine payloads,
   • launches a mitmdump reverse proxy for JSON-RPC,
   • runs Execution Spec Tests (EST) remotely against the proxy,
   • saves produced payloads and cleans up.

2. mitm_addon.py — a mitmproxy addon that groups intercepted eth_sendRawTransaction calls by test metadata and produces engine payloads without ever triggering a reorg (reorg is fully disabled for now).

Both tools save artifacts under payloads/ and extensive logs under /root/mitm_addon.log (proxy) and /root/nethermind.log (container logs).

Why

• Create deterministic, reproducible execution payloads while running EST suites against a local node.
• Capture payloads grouped by test/phase for later replay.
• Keep the environment ephemeral via OverlayFS to avoid mutating the base snapshot.
• Disable reorg handling for now to simplify flows and avoid accidental reorg paths during early testing.

Important engine behavior note (custom Nethermind image)

This setup uses a custom Nethermind image:
nethermindeth/nethermind:gp-hacked

That image intentionally implements a non-standard variant of engine_getPayloadV4:

• ✅ It accepts additional parameters (e.g., our generator calls it like
  engine_getPayloadV4([txrlp_list_or_None, rpc_address_or_EMPTY])), which is not per the EL Engine API spec.
• ✅ It directly returns a block with the desired payload attributes without requiring a prior engine_forkchoiceUpdatedV3 carrying payload attributes.
  In other words, the block is produced immediately on getPayload, rather than first configuring payload attributes via FCU and then calling getPayload.

What’s in this PR

• eest_stateful_generator.py

  • Snapshot bootstrap (ethpandaops) with optional refresh/skip.
  • OverlayFS mount (lower=snapshot, upper/work ephemeral, merged mounted).
  • Nethermind container startup with Engine+RPC exposed.
  • Health checks for ports and JSON-RPC.
  • Chain ID detection (CLI > map > RPC fallback).
  • Engine pre-seeding (“gas-bump” and “funding”) to create initial payload(s).
  • Writes mitm_config.json for the addon, including a dynamic finalized_block.
  • Starts mitmdump on :8549 with the addon and runs EST via uv.
  • Graceful teardown and cleanup (unless --keep).

• mitm_addon.py

  • Buffers eth_sendRawTransaction by (file_base, test_name, phase) derived from request metadata (id JSON or X-EEST-ID header).
  • Quiet-period flush (1s) or forced flush on group switch.
  • Calls engine_getPayloadV4([txrlps, "EMPTY"]), saves payload JSON to payloads/<file>/<test>/<phase>/<n>.json, then submits engine_newPayloadV4 and engine_forkchoiceUpdatedV3 with head/safe/finalized anchored.
  • JWT auth for engine requests; verbose, redacted logging.

Step-by-step: eest_stateful_generator.py runtime flow

1. Parse arguments
   Reads chain, test path, fork, RPC seed info, snapshot flags, --keep, etc.

2. Bootstrap tools & repo

   • Ensures requests is installed.
   • Clones execution-spec-tests repo if missing.
   • Ensures uv is available, pins Python 3.11 for the repo, and runs uv sync --all-extras.

3. Snapshot prep

   • If not --no-snapshot, downloads the latest Nethermind snapshot for the selected chain (via Docker/alpine) to execution-data/, unless it already exists and --refresh-snapshot isn’t set.
   • If --no-snapshot, ensures execution-data/ exists (empty).

4. JWT setup
   Writes a random engine-jwt/jwt.hex if missing.

5. OverlayFS mount

   • Prepares overlay-upper/, overlay-work/, overlay-merged/.
   • Mounts OverlayFS with lowerdir=execution-data, upperdir=overlay-upper, workdir=overlay-work into overlay-merged/ (uses sudo if not root).

6. Nethermind container

   • Removes any existing eest-nethermind.
   • Starts nethermindeth/nethermind:gp-hacked with:
     • DB at /db (overlay-merged bind),
     • Engine on :8551, RPC on :8545, JWT at /jwt/jwt.hex,
     • generous txpool and gas limit settings,
     • dev/unsecure RPC auth disabled (for local use).

7. Health checks

   • Waits for 127.0.0.1:8545 to accept TCP.
   • Polls eth_blockNumber until RPC is responsive or fails with logs and cleanup.

8. Chain ID

   • Chooses rpc_chain_id from CLI; else mapping; else eth_chainId; else defaults to 1.

9. Mitmproxy install
   Ensures mitmproxy is available.

10. Engine pre-seeding

    • Tries to generate “gas-bump” payloads (saved under payloads/gas-bump-*.json) to increase gas limit to desired 1 TeraGas.
    • Generates a “funding” payload targeting --rpc-address; captures the resulting blockHash and uses it as finalized_block. Funding long.Max ETH amount so it is enough for any testing.

11. Add-on config

    • Writes mitm_config.json with:
      • rpc_direct = http://127.0.0.1:8545
      • engine_url = http://127.0.0.1:8551
      • jwt_hex_path = engine-jwt/jwt.hex
      • finalized_block = <block hash from funding prep>

12. Start mitmdump + addon

    • Verifies mitm_addon.py is present.
    • Launches mitmdump on :8549 in reverse mode to :8545, with addon and conservative connection settings.
    • Waits for :8549 to bind.

13. Run EST

    • Runs uv run execute remote -v --fork=<fork> --rpc-seed-key=<key> --rpc-chain-id=<id> --rpc-endpoint=http://127.0.0.1:8549 <tests_path> -- -m benchmark -n 1 in the EST repo.
    • All test transactions go through the proxy and are captured/produced by the addon.

14. Teardown

    • Unless --keep:
      • Saves Nethermind logs to /root/nethermind.log.
      • Stops/removes the container.
      • Unmounts OverlayFS.
      • Removes overlay-* dirs.
    • Always attempts to terminate mitmdump.
    • Prints Done.

Artifacts produced

• payloads/:
  • gas-bump-*.json and funding-*.json (pre-seed),
  • Per-test engine payloads at payloads/<file>/<test>/<phase>/<n>.json.
• Logs:
  • /root/mitm_addon.log (addon & proxy),
  • /root/nethermind.log (container logs).

Step-by-step: mitm_addon.py runtime flow

1. Load config (MITM_ADDON_CONFIG env or mitm_config.json).
2. Start background monitor thread for quiet-period flush.
3. Intercept requests:
   • Logs all POSTs.
   • For eth_sendRawTransaction, derives (file_base, test_name, phase) from id or X-EEST-ID.
   • Buffers txs until quiet period or group switch.
4. Flush (produce payload):
   • Logs txpool summary.
   • Calls engine_getPayloadV4([txrlps, "EMPTY"]) (custom image returns block immediately).
   • Saves payload JSON under payloads/<file>/<test>/<phase>/<n>.json.
   • Calls engine_newPayloadV4 + engine_forkchoiceUpdatedV3.
   • Updates finalized anchor if in global-setup.
   • Reorg is fully disabled.
5. Shutdown gracefully stops monitor thread.

Configuration & flags

Generator

• --chain (mainnet|sepolia|holesky|goerli|ethereum; default mainnet)
• --test-path (path inside EST repo; default tests)
• --fork (e.g., Prague; default Prague)
• --rpc-endpoint (defaults to proxy http://127.0.0.1:8549)
• --rpc-seed-key (required) seed private key for EST funding
• --rpc-address (required) address to pre-fund
• --rpc-chain-id (override chain id)
• --no-snapshot, --refresh-snapshot
• --keep (preserve container/mounts/logs)

Addon

• Env MITM_ADDON_CONFIG → path to mitm_config.json:

  {
    "rpc_direct": "http://127.0.0.1:8545",
    "engine_url": "http://127.0.0.1:8551",
    "jwt_hex_path": "engine-jwt/jwt.hex",
    "finalized_block": "0x..."
  }