eth_call returns stale/incorrect data instead of error for blocks beyond Execution stage

# eth_call returns stale/incorrect data instead of error for blocks beyond Execution stage

## Summary

When a node is syncing, `eth_call` returns **stale data from the Execution stage** for blocks that have headers but haven't been executed yet, instead of returning an error.

**Two failure modes:**
1. **New contracts** (created after Execution stage): Returns `0x` (empty)
2. **Existing contracts**: Returns **incorrect data from the Execution block**, not the requested block

This causes downstream systems to cache incorrect values, leading to silent data corruption.

**Detection:** `eth_syncing` correctly reports when the node is in this unsafe state. If `eth_syncing != false`, `eth_call` may return stale data.

**Expected fix:** `eth_call` should return an error like `"state for block X not available (execution stage at Y)"` when the requested block exceeds the Execution stage.

## Environment

- **Erigon version:** 3.3.0, 3.3.2, and 3.3.7 (all affected)
- **Network:** Ethereum Mainnet
- **Sync status:** Node syncing (Execution stage behind chain tip)

## Steps to Reproduce

### 1. Find a syncing node

Check if a node is syncing and note the Execution stage progress:

```bash
# Check sync status
curl -s -X POST "http://YOUR_NODE:8545" \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"eth_syncing","params":[],"id":1}'
```

Example response showing node is syncing:
```json
{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    "currentBlock": "0x173a003",
    "highestBlock": "0x173a592",
    "stages": [...]
  }
}
```

### 2. Get the current block number reported by the node

```bash
curl -s -X POST "http://YOUR_NODE:8545" \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'
```

The node may report a block number **ahead of the Execution stage** (e.g., `0x173a52d` = 24356141).

### 3. Get the block hash for a block between Execution and Headers

```bash
# Get block hash from a synced reference node
curl -s -X POST "http://REFERENCE_NODE:8545" \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["0x173a817", false],"id":1}' | jq '.result.hash'
# Returns: "0x4ab3551819150bd8d2232f28438af0c4aa55a4b00b6a48e3da05ebb7bce1af43"
```

### 4. Call eth_call with block hash on both nodes

```bash
# On syncing node - returns WRONG data:
curl -s -X POST "http://SYNCING_NODE:8545" \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"eth_call","params":[{"to":"0xC02aaA39b223FE8D0A0e5C4F27eAD9083C756Cc2","data":"0x18160ddd"},{"blockHash":"0x4ab3551819150bd8d2232f28438af0c4aa55a4b00b6a48e3da05ebb7bce1af43"}],"id":1}'

# On reference node - returns CORRECT data:
curl -s -X POST "http://REFERENCE_NODE:8545" \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"eth_call","params":[{"to":"0xC02aaA39b223FE8D0A0e5C4F27eAD9083C756Cc2","data":"0x18160ddd"},{"blockHash":"0x4ab3551819150bd8d2232f28438af0c4aa55a4b00b6a48e3da05ebb7bce1af43"}],"id":1}'
```

### 5. Observe incorrect response

**Syncing node result (BUG):**
```json
{
  "jsonrpc": "2.0",
  "id": 1,
  "result": "0x00000000000000000000000000000000000000000001d428c73d81b04ddce7ec"
}
```
This is the state from the **Execution stage block**, not the requested block.

**Reference node result (CORRECT):**
```json
{
  "jsonrpc": "2.0",
  "id": 1,
  "result": "0x00000000000000000000000000000000000000000001d3b0b66c899f10ea4771"
}
```

**Expected behavior when state unavailable:**
```json
{
  "jsonrpc": "2.0",
  "id": 1,
  "error": {
    "code": -32000,
    "message": "state for block 0x4ab355... not available (execution stage at 24356849)"
  }
}
```

## Live Test Results (2026-01-31)

### Test Setup
- **Syncing node Node A:** Erigon 3.3.7, Execution stage at block 24356849
- **Synced node Node B:** Erigon 3.3.0, fully synced

### Test: WETH.totalSupply() at different blocks

| Block | Node A (syncing) | Node B (synced) | Match? |
|-------|------------------|---------------------|--------|
| 24356726 (BEFORE Execution) | `...d854501c182b5effab06` | `...d854501c182b5effab06` | ✓ |
| 24356786 (AFTER Execution) | `...d86b821f1c571634cd8b` | `...d7c6b655a677aa9b0058` | **✗ WRONG** |

### Critical Finding

The syncing node returns the **Execution stage value** regardless of which block is requested:

```
Request: WETH.totalSupply() with blockHash of block 24356887

Syncing node returns: 0x...d428c73d81b04ddce7ec
Reference node returns: 0x...d3b0b66c899f10ea4771  ← Correct value for block 24356887

Reference at Execution block (24356849): 0x...d428c73d81b04ddce7ec  ← Same as syncing!
```

**The syncing node ignores the requested block hash and returns data from the Execution stage.**

### Test: eth_call on newly created contract

For a Uniswap V3 pool created at block 24356141:

| Node | slot0() result |
|------|----------------|
| Node A (Execution at 24356035) | `{"result": "0x"}` |
| Node B (synced) | `{"result": "0x00000000..."}` (valid data) |

**New contracts return empty `0x` because they don't exist at the Execution stage.**

### Test: Same block hash returns DIFFERENT data on syncing vs reference node

```bash
# Block 24356887 hash: 0x4ab3551819150bd8d2232f28438af0c4aa55a4b00b6a48e3da05ebb7bce1af43

# Syncing node:
curl -s -X POST "http://SYNCING_NODE:8545" -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"eth_call","params":[{"to":"0xC02aaA39b223FE8D0A0e5C4F27eAD9083C756Cc2","data":"0x18160ddd"},{"blockHash":"0x4ab3551819150bd8d2232f28438af0c4aa55a4b00b6a48e3da05ebb7bce1af43"}],"id":1}'
# Returns: 0x...d428c73d81b04ddce7ec (WRONG - stale data)

# Reference node:
curl -s -X POST "http://REFERENCE_NODE:8545" -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"eth_call","params":[{"to":"0xC02aaA39b223FE8D0A0e5C4F27eAD9083C756Cc2","data":"0x18160ddd"},{"blockHash":"0x4ab3551819150bd8d2232f28438af0c4aa55a4b00b6a48e3da05ebb7bce1af43"}],"id":1}'
# Returns: 0x...d3b0b66c899f10ea4771 (CORRECT)
```

**An immutable block hash should always return the same data across all nodes.**

### Test: Same block number returns different data on repeated requests

Without any reorg, requesting the same block number twice returns different values as Execution advances:

```bash
# First request (Execution at block X):
{"result":"0x00000000000000000000000000000000000000000001d42c83ce9fe7dea71de3"}

# Second request seconds later (Execution advanced to block X+N):
{"result":"0x00000000000000000000000000000000000000000001d41655fa5708e3f522b6"}
```

**The node returns whatever state the Execution stage is currently at**, not the state for the requested block.

## Minimal Reproduction Script

```bash
#!/bin/bash
# test_eth_call_sync_bug.sh
# Tests if a syncing Erigon node returns wrong data for blocks beyond Execution stage
# Uses block hash to make the bug more obvious (hash is immutable)

SYNCING_NODE="${1:-http://localhost:8545}"
REFERENCE_NODE="${2:-http://localhost:8546}"  # A fully synced node for comparison

echo "=== Testing eth_call sync bug ==="
echo "Syncing node: $SYNCING_NODE"
echo "Reference node: $REFERENCE_NODE"
echo

# 1. Check if syncing and get Execution stage
SYNC=$(curl -s -X POST "$SYNCING_NODE" -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"eth_syncing","params":[],"id":1}')

EXEC_HEX=$(echo "$SYNC" | jq -r '.result.stages[] | select(.stage_name == "Execution") | .block_number')
HEADERS_HEX=$(echo "$SYNC" | jq -r '.result.stages[] | select(.stage_name == "Headers") | .block_number')

if [ -z "$EXEC_HEX" ]; then
  echo "Node is not syncing (eth_syncing = false). Test requires a syncing node."
  exit 0
fi

EXEC_DEC=$((EXEC_HEX))
HEADERS_DEC=$((HEADERS_HEX))
GAP=$((HEADERS_DEC - EXEC_DEC))

echo "Execution stage: $EXEC_DEC"
echo "Headers stage: $HEADERS_DEC"
echo "Gap: $GAP blocks"

if [ $GAP -lt 5 ]; then
  echo "Gap too small. Wait for node to fall behind."
  exit 0
fi

# 2. Pick a test block and get its hash from reference node
TEST_BLOCK=$((EXEC_DEC + GAP / 2))
TEST_HEX=$(printf "0x%x" $TEST_BLOCK)

BLOCK_HASH=$(curl -s -X POST "$REFERENCE_NODE" -H "Content-Type: application/json" \
  -d "{\"jsonrpc\":\"2.0\",\"method\":\"eth_getBlockByNumber\",\"params\":[\"$TEST_HEX\", false],\"id\":1}" \
  | jq -r '.result.hash')

echo "Test block: $TEST_BLOCK"
echo "Block hash: $BLOCK_HASH"
echo

# 3. Query both nodes with the same block hash
echo "=== WETH.totalSupply() with blockHash ==="

RESULT_SYNC=$(curl -s -X POST "$SYNCING_NODE" -H "Content-Type: application/json" \
  -d "{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"to\":\"0xC02aaA39b223FE8D0A0e5C4F27eAD9083C756Cc2\",\"data\":\"0x18160ddd\"},{\"blockHash\":\"$BLOCK_HASH\"}],\"id\":1}" \
  | jq -r '.result')

RESULT_REF=$(curl -s -X POST "$REFERENCE_NODE" -H "Content-Type: application/json" \
  -d "{\"jsonrpc\":\"2.0\",\"method\":\"eth_call\",\"params\":[{\"to\":\"0xC02aaA39b223FE8D0A0e5C4F27eAD9083C756Cc2\",\"data\":\"0x18160ddd\"},{\"blockHash\":\"$BLOCK_HASH\"}],\"id\":1}" \
  | jq -r '.result')

echo "Syncing node: $RESULT_SYNC"
echo "Reference:    $RESULT_REF"

if [ "$RESULT_SYNC" = "$RESULT_REF" ]; then
  echo "MATCH - Node may have caught up"
else
  echo ""
  echo "=== BUG CONFIRMED ==="
  echo "Same block hash returns different data!"
  echo "The syncing node ignores the requested block and returns Execution stage data."
  exit 1
fi
```

## Root Cause Analysis

### The Problem

`eth_call` does not validate that the requested block's state has been executed. It only checks:
1. Whether the header exists (downloaded via P2P)
2. Whether history has been pruned (for old blocks)

But it does **NOT** check if the Execution stage has processed that block.

### Code Path

**1. eth_call handler** (`rpc/jsonrpc/eth_call.go:65-120`):
```go
func (api *APIImpl) Call(ctx context.Context, args ethapi2.CallArgs, requestedBlock *rpc.BlockNumberOrHash, ...) (hexutil.Bytes, error) {
    // Gets header - succeeds because headers are downloaded ahead of execution
    header, _, err := headerByNumberOrHash(ctx, tx, blockNrOrHash, api)

    // Only checks pruning, NOT execution stage
    err = api.BaseAPI.checkPruneHistory(ctx, tx, header.Number.Uint64())

    // Creates state reader with incomplete data
    stateReader, err := rpchelper.CreateStateReader(...)

    // Executes - silently returns empty result
    result, err := transactions.DoCall(...)
}
```

**2. checkPruneHistory** (`rpc/jsonrpc/eth_api.go:344-368`):
```go
func (api *BaseAPI) checkPruneHistory(ctx context.Context, tx kv.Tx, block uint64) error {
    // Only checks if block is BEFORE pruned threshold (old blocks)
    prunedTo := p.History.PruneTo(latest)
    if block < prunedTo {
        return state.PrunedError
    }
    return nil  // No check for blocks AFTER execution stage!
}
```

**3. HistoryReaderV3** (`execution/state/history_reader_v3.go:81-98`):
```go
func (hr *HistoryReaderV3) ReadAccountData(address accounts.Address) (*accounts.Account, error) {
    enc, ok, err := hr.ttx.GetAsOf(kv.AccountsDomain, addressValue[:], hr.txNum)

    // Returns (nil, nil) when data not found - no error!
    if err != nil || !ok || len(enc) == 0 {
        return nil, err
    }
    // ...
}
```

**4. IntraBlockState** (`execution/state/intra_block_state.go:464-502`):
```go
func (sdb *IntraBlockState) GetCode(addr accounts.Address) ([]byte, error) {
    stateObject, err := sdb.getStateObject(addr, true)
    if stateObject == nil || stateObject.deleted {
        return nil, nil  // Returns nil without error for "non-existent" accounts
    }
    // ...
}
```

### Why This Happens

1. **Header exists** → Downloaded by P2P layer ahead of execution
2. **checkPruneHistory passes** → Block is recent, not pruned
3. **State reader created** → But temporal data doesn't exist for this block
4. **ReadAccountData returns (nil, nil)** → Data not found = nil, no error
5. **GetCode/GetBalance treat this as "account doesn't exist"** → Valid Ethereum behavior
6. **EVM executes successfully** → Empty account is valid
7. **Result is `0x`** → Empty but valid response

The bug is that **"state not synced yet"** is treated the same as **"account doesn't exist"**.

## Proposed Fix

Add execution stage validation in `eth_call` before creating the state reader:

```go
// In rpc/jsonrpc/eth_call.go, after checkPruneHistory:

// Check if block has been executed
executedBlock, err := stages.GetStageProgress(tx, stages.Execution)
if err != nil {
    return nil, err
}
if header.Number.Uint64() > executedBlock {
    return nil, fmt.Errorf("state for block %d not available (execution stage at %d)",
        header.Number.Uint64(), executedBlock)
}
```

This check should also be added to:
- `eth_getCode`
- `eth_getBalance`
- `eth_getStorageAt`
- `eth_getProof`
- `eth_estimateGas`

Alternatively, add this check in `CreateStateReader()` in `rpc/rpchelper/helper.go`.

## Impact

This bug affects any system that:
1. Uses a syncing Erigon node for RPC calls
2. Caches `eth_call` results
3. Requests data for recently created contracts

**Concrete example:** The Graph's `graph-node` caches `eth_call` results. If it calls a syncing Erigon node and receives `0x`, it caches this empty value permanently, causing persistent indexing failures even after the node is fully synced.

## Workaround

The issue can be detected using `eth_syncing`:

```bash
# Check if node is safe for eth_call
curl -s -X POST "http://NODE:8545" \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"eth_syncing","params":[],"id":1}'
```

| `eth_syncing` result | Safe for `eth_call`? |
|---------------------|---------------------|
| `{"result": false}` | ✓ Yes - fully synced |
| `{"result": {...}}` | ✗ No - may return stale data |

**For load balancers:** Exclude nodes where `eth_syncing != false` from serving `eth_call` requests.

```python
def is_node_safe(node):
    result = rpc_call(node, "eth_syncing", [])
    return result["result"] == False
```

**Verified:** When `eth_syncing` returns an object (syncing), `eth_call` returns incorrect data. When it returns `false`, data is correct.

## Erigon Logs During Bug

The following logs show the Execution stage processing blocks 24356824→24356926 while RPC requests are being served:

```
[INFO] [01-31|19:26:25.930] [4/6 Execution] DONE                     in=1m32.99s block=24356815
[INFO] [01-31|19:26:29.686] [4/6 Execution] serial starting          from=24356825 to=24356926 ...
[INFO] [01-31|19:26:50.307] [4/6 Execution] serial executed          blk=24356837 blks=12 blk/s=0.58 ...
[WARN] [01-31|19:27:28.645] [rpc] served                             method=eth_getProof err="root hash mismatch..."
[INFO] [01-31|19:28:05.012] [4/6 Execution] DONE                     in=1m35.33s block=24356837
```

During this window:
- Execution stage is at block 24356824 → 24356837
- Headers stage is at block 24356926
- RPC requests for blocks 24356838-24356926 return **stale data from block 24356837**
- An `eth_getProof` request even shows "root hash mismatch" error

The node accepts and processes RPC requests for blocks it hasn't executed yet, returning data from the last executed block instead of an error.

## Related

- This may be related to the distinction between `forkchoiceHead` and Execution stage progress
- Similar issue may exist in other RPC methods that read state (`eth_getBalance`, `eth_getCode`, `eth_getStorageAt`, `eth_getProof`)

## Severity

**Critical** - Silent data corruption with two failure modes:

1. **New contracts return `0x`**: Appears as "contract doesn't exist"
2. **Existing contracts return stale data**: Returns data from Execution stage block, not requested block

Both return valid-looking JSON-RPC responses with no error, making it **impossible for clients to detect the problem** without comparing against another node.

### Real-world Impact

- **The Graph's graph-node**: Caches `eth_call` results permanently. If it queries a syncing node, it caches wrong values forever, causing persistent indexing failures even after the node is fully synced.
- **Any caching layer**: Will cache and serve incorrect data
- **DeFi applications**: Could make incorrect trading decisions based on stale state


`eth_syncing` result	Safe for `eth_call`?
`{"result": false}`	✓ Yes - fully synced
`{"result": {...}}`	✗ No - may return stale data

Block	Node A (syncing)	Node B (synced)	Match?
24356726 (BEFORE Execution)	`...d854501c182b5effab06`	`...d854501c182b5effab06`	✓
24356786 (AFTER Execution)	`...d86b821f1c571634cd8b`	`...d7c6b655a677aa9b0058`	✗ WRONG

Node	slot0() result
Node A (Execution at 24356035)	`{"result": "0x"}`
Node B (synced)	`{"result": "0x00000000..."}` (valid data)

eth_call returns stale/incorrect data instead of error for blocks beyond Execution stage #18912

Description

eth_call returns stale/incorrect data instead of error for blocks beyond Execution stage

Summary

Environment

Steps to Reproduce

1. Find a syncing node

2. Get the current block number reported by the node

3. Get the block hash for a block between Execution and Headers

4. Call eth_call with block hash on both nodes

5. Observe incorrect response

Live Test Results (2026-01-31)

Test Setup

Test: WETH.totalSupply() at different blocks

Critical Finding

Test: eth_call on newly created contract

Test: Same block hash returns DIFFERENT data on syncing vs reference node

Test: Same block number returns different data on repeated requests

Minimal Reproduction Script

Root Cause Analysis

The Problem

Code Path

Why This Happens

Proposed Fix

Impact

Workaround

Erigon Logs During Bug

Related

Severity

Real-world Impact

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions