Skip to content

feat: integrate Apollo dynamic configuration#8

Merged
louisliu2048 merged 46 commits intodevfrom
feat/apollo
Nov 7, 2025
Merged

feat: integrate Apollo dynamic configuration#8
louisliu2048 merged 46 commits intodevfrom
feat/apollo

Conversation

@limyeechern
Copy link
Copy Markdown

@limyeechern limyeechern commented Oct 31, 2025

Apollo Dynamic Configuration Integration

Overview

Adds Apollo configuration management system to reth, enabling dynamic runtime configuration updates without restarts.
Refer to Apollo PR on op-geth at okx/op-geth#60.

Note: The design for Apollo feature on reth is slightly different from that of op-geth due to inherent client design differences.

Key Components

New Crate: reth-apollo

  • client.rs: Core Apollo client with singleton pattern

    • Thread-safe cache with TTL (moka::sync::Cache)
    • Updates via:
    1. Fixed interval background polling task
    2. Event-driven configuration updates
  • namespace.rs: Namespace and component definitions

    • Component::OpReth - Component identifier
    • Namespace enum for configured namespace
  • types.rs: Type system and error handling

    • FromJsonValue trait for type-safe config extraction
    • Supports: u64, i64, f64, bool, String, Vec<T>
  • lib.rs: Public macro

    • apollo_config_or! macro for config access

CLI Integration

  • ApolloArgs: Command-line arguments
    • --apollo.enable - Enable/disable Apollo
    • --apollo.app-id - Apollo application ID
    • --apollo.ip - Apollo meta server URL
    • --apollo.cluster - Cluster name
    • --apollo.namespace - Configuration namespace

Usage Example

The developer must explicitly use the macro function if the developer wishes to get the configuration from Apollo. If the configuration exists in the cache, it will be used, otherwise it will default to the default value.

This behavior is different from the op-geth implementation, but it adds a layer of security as the developer must explicitly attempt to use the Apollo configuration.

use reth_apollo::{Component, Namespace, apollo_cached_config};

let gpo_factor = apollo_config_or!(
    Component::OpReth.with_namespace(Namespace::JsonRpc),
    "gpo.factor",
    100u64  // default value
);

let some_config = apollo_config_or!(
    Component::OpReth.with_namespace(Namespace::JsonRpc),
    "some.config",
    node_config.config().someConfig  // default value
);

@limyeechern limyeechern marked this pull request as draft October 31, 2025 02:09
@limyeechern limyeechern requested a review from sieniven November 6, 2025 03:58
@louisliu2048 louisliu2048 merged commit 0a6bf5e into dev Nov 7, 2025
Vui-Chee added a commit that referenced this pull request Nov 11, 2025
* dev: (458 commits)
  Ignore xlayer-e2e-test during integration testing (#32)
  feat(genesis): optimize genesis loading with built-in xlayer specs (#23)
  Mirror default timeout in `clap` args struct (#31)
  fix constraint (#30)
  support trace tx log (#16)
  Avoid cloning tx during payload execution (#29)
  Feature: import blockchain data into op-reth (#20)
  perf(cargo): Use alloy 1.1.0 and fix some warning issue (#26)
  feat: Add E2E Test Suite (#19)
  fix tx number inconsistencies issue cause by intercept bridge function (#25)
  fix build error
  feat: integrate Apollo dynamic configuration (#8)
  feat: support `eth_transactionPreExec` api (#10)
  chore: bump version
  chore: bump revm v31.0.1 (paradigmxyz#19567)
  Xiong/block bridge okb (#21)
  feat: Add legacy RPC routing support for historical block data migration (#6)
  feat(genesis): Support legacyXLayerBlock in genesis.json (#11)
  chore: Update cursor rules, add rust comprehensive cursor rules (#17)
  chore: bump v1.9.0 (paradigmxyz#19507)
  ...
louisliu2048 pushed a commit that referenced this pull request Mar 24, 2026
…ap checklist

Complete implementation of the MPT WAL-First gap checklist (10 items) plus
additional sei-db alignment fixes identified during review.

WAL subsystem (#1):
- Segment-based binary WAL (bincode, mptwal02 format) with CRC32 checksums
- Record header contains version field for O(1) scan without deserialization
- Segment-aware prune/truncate (delete whole segments, rewrite only boundary)
- Corrupted tail detection and recovery on append
- Single fsync per append (removed redundant meta.json fsync)

Snapshot rewrite (#2):
- Per-block incremental publish_generation in persist worker (keeps L3 hot)
- Periodic full rewrite via separate published-rewrite worker thread
- Pre-built segments from materializer's in-memory state (avoids disk reload)
- Staged activation: compact/prune before meta activation (atomic boundary)
- Retry mechanism for dropped rewrite jobs (pending_rewrite tracking)

Backpressure (#3):
- max_durable_lag / max_published_lag config fields
- wait_for_backpressure() blocks frontend commit when lag exceeds threshold
- 60s timeout with warning on backpressure wait

Seek-best-snapshot (#4):
- seek_best_snapshot_version() finds max(snapshot) <= target
- WAL chain validation before replay attempt
- Clear error messages with snapshot version and WAL range

WAL auto-cleanup (#5):
- Auto prune_before after each set_durable_version in both worker paths
- Floor = min(manifest.earliest, published.earliest_snapshot)

Replay parallelism (#6):
- Aggressive parallelism thresholds during replay (storage_tries_min=4)
- Batch WAL pre-fetch (64 entries per batch, amortizes lock + IO)

Temp cleanup (#7): already done by codex

account_root independent (#8):
- CommitWalEntry.account_root now explicitly passed, not copied from state_root

WAL upgrade field (#9):
- CommitWalUpgrade struct (key/value pairs) for schema migrations
- CommitWalEntry.upgrades field (empty for normal commits)

Snapshot rate limiting (#10):
- IoRateLimiter token bucket for background snapshot writes
- snapshot_write_rate_mb_per_sec config (default 0 = unlimited)

COW Arena:
- MutableTrieArena rewritten with frozen base (Arc) + overlay (HashMap)
- clone() is O(1) for frozen base, O(overlay) for mutations
- freeze() uses Arc::make_mut for in-place patch when strong_count=1
- set_committed_base drops old base before snapshot() to ensure O(overlay) freeze

Additional sei-db alignment:
- SetInitialVersion API with validation (version==0, fresh DB)
- nextVersion jump logic matching sei-db's nextVersionU32
- LoadForOverwriting: pre-open truncation (manifest + WAL + published)
- Graceful error recovery: published baseline errors are non-fatal (warn only)
- MemoryStats for in-memory node tracking
- Lazy published view refresh in apply_bundle_state
- Published rewrite timeout (configurable, default 60s)

B4.5 profile (1M accounts, 10 blocks × 5K updates):
- mpt-db per-block: ~370ms (vs reth ~1290ms) = ~3.5x faster
- account_root: ~28ms (COW freeze working correctly)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants