…ap checklist

Complete implementation of the MPT WAL-First gap checklist (10 items) plus
additional sei-db alignment fixes identified during review.

WAL subsystem (#1):
- Segment-based binary WAL (bincode, mptwal02 format) with CRC32 checksums
- Record header contains version field for O(1) scan without deserialization
- Segment-aware prune/truncate (delete whole segments, rewrite only boundary)
- Corrupted tail detection and recovery on append
- Single fsync per append (removed redundant meta.json fsync)

Snapshot rewrite (#2):
- Per-block incremental publish_generation in persist worker (keeps L3 hot)
- Periodic full rewrite via separate published-rewrite worker thread
- Pre-built segments from materializer's in-memory state (avoids disk reload)
- Staged activation: compact/prune before meta activation (atomic boundary)
- Retry mechanism for dropped rewrite jobs (pending_rewrite tracking)

Backpressure (#3):
- max_durable_lag / max_published_lag config fields
- wait_for_backpressure() blocks frontend commit when lag exceeds threshold
- 60s timeout with warning on backpressure wait

Seek-best-snapshot (#4):
- seek_best_snapshot_version() finds max(snapshot) <= target
- WAL chain validation before replay attempt
- Clear error messages with snapshot version and WAL range

WAL auto-cleanup (#5):
- Auto prune_before after each set_durable_version in both worker paths
- Floor = min(manifest.earliest, published.earliest_snapshot)

Replay parallelism (#6):
- Aggressive parallelism thresholds during replay (storage_tries_min=4)
- Batch WAL pre-fetch (64 entries per batch, amortizes lock + IO)

Temp cleanup (#7): already done by codex

account_root independent (#8):
- CommitWalEntry.account_root now explicitly passed, not copied from state_root

WAL upgrade field (#9):
- CommitWalUpgrade struct (key/value pairs) for schema migrations
- CommitWalEntry.upgrades field (empty for normal commits)

Snapshot rate limiting (#10):
- IoRateLimiter token bucket for background snapshot writes
- snapshot_write_rate_mb_per_sec config (default 0 = unlimited)

COW Arena:
- MutableTrieArena rewritten with frozen base (Arc) + overlay (HashMap)
- clone() is O(1) for frozen base, O(overlay) for mutations
- freeze() uses Arc::make_mut for in-place patch when strong_count=1
- set_committed_base drops old base before snapshot() to ensure O(overlay) freeze

Additional sei-db alignment:
- SetInitialVersion API with validation (version==0, fresh DB)
- nextVersion jump logic matching sei-db's nextVersionU32
- LoadForOverwriting: pre-open truncation (manifest + WAL + published)
- Graceful error recovery: published baseline errors are non-fatal (warn only)
- MemoryStats for in-memory node tracking
- Lazy published view refresh in apply_bundle_state
- Published rewrite timeout (configurable, default 60s)

B4.5 profile (1M accounts, 10 blocks × 5K updates):
- mpt-db per-block: ~370ms (vs reth ~1290ms) = ~3.5x faster
- account_root: ~28ms (COW freeze working correctly)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…hildren reuse

Observability fixes (commit_store.rs)

  Introduce OverlayOutcome enum with three distinct cases:
  - Stolen{shrank, reused_bytes}: capacity transferred from base (lazy or clone).
  - FreshClone: fell back to base.clone() — real fresh heap allocation.
  - ExistingWorking: pre-materialised working trie reused, no steal needed.

  Previously overlay_reuse_misses conflated FreshClone and ExistingWorking,
  making the stats untrustworthy for performance analysis.  CommitProfile now
  reports overlay_stolen / overlay_fresh_clone / overlay_existing_working /
  overlay_shrink_events / overlay_reused_capacity_bytes separately.

  Fix reset ordering: counters are now reset at the start of
  apply_bundle_state_inner (not commit_inner_with_mode), so both the apply
  phase and the commit phase steals accumulate into the same per-block total.

  Fix lazy-materialisation hit: take_working_for_version returns
  OverlayOutcome::Stolen when the lazy path steals from base, matching the
  clone_base_with_steal path.  Previously the lazy steal was always reported
  as hit=false.

pending_lazy_children reuse (storage_cow.rs, arena.rs)

  Root cause of the previous jemalloc SIGKILL: is_overlay_reusable() did not
  check pending_lazy_children.is_empty().  Values in that map can be
  CowChildRef::Lazy(CowLazyNodeRef::Inline(Vec<u8>)) — heap allocations from
  the calling thread.  Swapping a non-empty map across rayon thread boundaries
  triggers cross-thread deallocation which trips jemalloc guard pages.

  Fix: add pending_lazy_children.is_empty() to is_overlay_reusable().
  set_committed_base always calls clear_pending_lazy() first, so the check
  passes in the normal flow.  steal_overlay_capacity_from now includes a
  debug_assert + conditional swap of pending_lazy_children, safely transferring
  the cleared-but-capacity-holding map to the working trie.

  Completes Guardrail #1 (all major overlay containers now covered).

shrink return value (arena.rs, storage_cow.rs)

  shrink_overlay_if_oversized now returns bool indicating whether a shrink
  occurred, propagated through steal_overlay_capacity_from for accurate
  overlay_shrink_events counting.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>