Skip to content

feat: Add legacy RPC routing support for historical block data migration#6

Merged
louisliu2048 merged 8 commits intodevfrom
zjg/api_legacy
Nov 7, 2025
Merged

feat: Add legacy RPC routing support for historical block data migration#6
louisliu2048 merged 8 commits intodevfrom
zjg/api_legacy

Conversation

@zjg555543
Copy link
Copy Markdown

@zjg555543 zjg555543 commented Oct 29, 2025

XLayer Legacy RPC Implementation

Based on op-geth migration RPC implementation

Parameters

Parameter Description
--rpc.legacy-url URL of the legacy Erigon RPC endpoint
--rpc.legacy-timeout Timeout for legacy RPC requests in seconds

API Implementation Status

# RPC Method Status Note
1 eth_getBlockByNumber
2 eth_getBlockByHash
3 eth_getHeaderByNumber ⚠️ Legacy doesn't support
4 eth_getHeaderByHash ⚠️ Legacy doesn't support
5 eth_getBlockReceipts
6 eth_getBlockTransactionCountByNumber
7 eth_getBlockTransactionCountByHash
8 eth_getTransactionByHash
9 eth_getTransactionReceipt
10 eth_getTransactionByBlockHashAndIndex
11 eth_getTransactionByBlockNumberAndIndex
12 eth_getRawTransactionByHash
13 eth_getRawTransactionByBlockHashAndIndex
14 eth_getRawTransactionByBlockNumberAndIndex
15 eth_getBalance
16 eth_getCode
17 eth_getStorageAt
18 eth_getTransactionCount
19 eth_getLogs
20 eth_newFilter Real-time query only
21 eth_getFilterLogs Real-time query only
22 eth_getFilterChanges Real-time query only
23 eth_uninstallFilter Real-time query only
24 eth_getBlockInternalTransactions 🔄 TODO
25 eth_getInternalTransactions 🔄 TODO

integration testing

https://github.com/okx/xlayer-toolkit/blob/zjg/reth-legacy/scripts/legacy-api/test_legacy_rpc.sh

This commit implements a complete Legacy RPC routing system to support
seamless data migration from Erigon to Reth for XLayer network.

Key Features:
- Add legacy RPC client for routing historical block requests
- Implement LegacyRpc trait for OpEthApi and EthApi
- Support hybrid eth_getLogs queries across migration boundary
- Add comprehensive integration test script for legacy RPC validation
- Remove filter-related legacy code (not suitable for historical blocks)

Core Changes:
- crates/rpc/rpc-eth-types: Add LegacyRpcClient and configuration
- crates/rpc/rpc-eth-api: Add LegacyRpc trait and helpers
- crates/rpc/rpc: Implement legacy routing in filter.rs and core.rs
- crates/optimism/rpc: Implement LegacyRpc trait for OpEthApi
- crates/node/builder: Pass legacy config to EthApiBuilder

Testing:
- Add test_legacy_rpc.sh script with 10 test phases
- Add comprehensive README for testing documentation
- Test coverage: block queries, transactions, state, logs, filters, edge cases

Docker:
- Update DockerfileOp to include curl for debugging
- Add build scripts for debug and release images

Configuration:
- XLayer Mainnet: cutoff block 42810021
- XLayer Testnet: cutoff block 12241701
- Legacy RPC timeout: 30s (configurable)

This implementation ensures backward compatibility while maintaining
performance for new block data served directly from Reth.
- Resolve conflicts in RPC configuration files
- Keep both legacy RPC config and send_raw_transaction_sync_timeout
- Update dependency: add dyn-clone alongside humantime
- Sync with latest dev branch improvements
- Add exec_legacy() helper function for unified logging
- Add elapsed_ms tracking for all legacy RPC calls
- Add detailed metrics for hybrid eth_getLogs queries
- Refactor XLayer-specific code to separate *_xlayer.rs files
- Simplify macro implementations for better maintainability

Changes:
- Legacy RPC calls now log request/response with timing
- Hybrid mode shows legacy_logs, local_logs, and total counts
- Error cases include elapsed_ms and detailed error messages
- All XLayer extensions moved to dedicated files for clean separation
@zjg555543 zjg555543 marked this pull request as ready for review November 4, 2025 10:40
@zjg555543 zjg555543 marked this pull request as draft November 5, 2025 03:01
@zjg555543 zjg555543 marked this pull request as ready for review November 5, 2025 03:01
- Add rpc_server_xlayer.rs for XLayer-specific extensions
- Auto-apply genesis block number as default for legacy_cutoff_block
- Only 2 files modified, 10 lines changed
- All XLayer logic isolated in dedicated file
@louisliu2048 louisliu2048 merged commit c6e1734 into dev Nov 7, 2025
Vui-Chee added a commit that referenced this pull request Nov 11, 2025
* dev: (458 commits)
  Ignore xlayer-e2e-test during integration testing (#32)
  feat(genesis): optimize genesis loading with built-in xlayer specs (#23)
  Mirror default timeout in `clap` args struct (#31)
  fix constraint (#30)
  support trace tx log (#16)
  Avoid cloning tx during payload execution (#29)
  Feature: import blockchain data into op-reth (#20)
  perf(cargo): Use alloy 1.1.0 and fix some warning issue (#26)
  feat: Add E2E Test Suite (#19)
  fix tx number inconsistencies issue cause by intercept bridge function (#25)
  fix build error
  feat: integrate Apollo dynamic configuration (#8)
  feat: support `eth_transactionPreExec` api (#10)
  chore: bump version
  chore: bump revm v31.0.1 (paradigmxyz#19567)
  Xiong/block bridge okb (#21)
  feat: Add legacy RPC routing support for historical block data migration (#6)
  feat(genesis): Support legacyXLayerBlock in genesis.json (#11)
  chore: Update cursor rules, add rust comprehensive cursor rules (#17)
  chore: bump v1.9.0 (paradigmxyz#19507)
  ...
louisliu2048 pushed a commit that referenced this pull request Mar 24, 2026
…ap checklist

Complete implementation of the MPT WAL-First gap checklist (10 items) plus
additional sei-db alignment fixes identified during review.

WAL subsystem (#1):
- Segment-based binary WAL (bincode, mptwal02 format) with CRC32 checksums
- Record header contains version field for O(1) scan without deserialization
- Segment-aware prune/truncate (delete whole segments, rewrite only boundary)
- Corrupted tail detection and recovery on append
- Single fsync per append (removed redundant meta.json fsync)

Snapshot rewrite (#2):
- Per-block incremental publish_generation in persist worker (keeps L3 hot)
- Periodic full rewrite via separate published-rewrite worker thread
- Pre-built segments from materializer's in-memory state (avoids disk reload)
- Staged activation: compact/prune before meta activation (atomic boundary)
- Retry mechanism for dropped rewrite jobs (pending_rewrite tracking)

Backpressure (#3):
- max_durable_lag / max_published_lag config fields
- wait_for_backpressure() blocks frontend commit when lag exceeds threshold
- 60s timeout with warning on backpressure wait

Seek-best-snapshot (#4):
- seek_best_snapshot_version() finds max(snapshot) <= target
- WAL chain validation before replay attempt
- Clear error messages with snapshot version and WAL range

WAL auto-cleanup (#5):
- Auto prune_before after each set_durable_version in both worker paths
- Floor = min(manifest.earliest, published.earliest_snapshot)

Replay parallelism (#6):
- Aggressive parallelism thresholds during replay (storage_tries_min=4)
- Batch WAL pre-fetch (64 entries per batch, amortizes lock + IO)

Temp cleanup (#7): already done by codex

account_root independent (#8):
- CommitWalEntry.account_root now explicitly passed, not copied from state_root

WAL upgrade field (#9):
- CommitWalUpgrade struct (key/value pairs) for schema migrations
- CommitWalEntry.upgrades field (empty for normal commits)

Snapshot rate limiting (#10):
- IoRateLimiter token bucket for background snapshot writes
- snapshot_write_rate_mb_per_sec config (default 0 = unlimited)

COW Arena:
- MutableTrieArena rewritten with frozen base (Arc) + overlay (HashMap)
- clone() is O(1) for frozen base, O(overlay) for mutations
- freeze() uses Arc::make_mut for in-place patch when strong_count=1
- set_committed_base drops old base before snapshot() to ensure O(overlay) freeze

Additional sei-db alignment:
- SetInitialVersion API with validation (version==0, fresh DB)
- nextVersion jump logic matching sei-db's nextVersionU32
- LoadForOverwriting: pre-open truncation (manifest + WAL + published)
- Graceful error recovery: published baseline errors are non-fatal (warn only)
- MemoryStats for in-memory node tracking
- Lazy published view refresh in apply_bundle_state
- Published rewrite timeout (configurable, default 60s)

B4.5 profile (1M accounts, 10 blocks × 5K updates):
- mpt-db per-block: ~370ms (vs reth ~1290ms) = ~3.5x faster
- account_root: ~28ms (COW freeze working correctly)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants