-
Notifications
You must be signed in to change notification settings - Fork 182
feat: add --index-backfill-epochs to forest-tool api serve
#6068
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
WalkthroughMade the Changes
Sequence Diagram(s)sequenceDiagram
participant CLI as forest-tool (CLI)
participant Server as OfflineServer
participant DB as DB
participant Snap as SnapshotHandler
participant Index as BackfillIndexer
CLI->>Server: start_offline_server(snapshot_files, chain?, height, index_backfill_epochs, ...)
alt chain not provided
Server->>DB: infer chain from DB (genesis/head)
DB-->>Server: inferred_chain
Server->>Server: validate provided chain discriminant if present
end
Server->>Snap: handle_snapshots(snapshot_files, chain?)
Snap-->>Server: loaded_snapshot(s)
Server->>DB: compute head_epoch / validate tipsets
Server->>Index: if index_backfill_epochs > 0 -> backfill range (head_epoch ... head_epoch+1-index_backfill_epochs)
Index-->>DB: write indexed data
Server-->>CLI: RPC server started (port ...)
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related PRs
Suggested reviewers
📜 Recent review detailsConfiguration used: CodeRabbit UI Review profile: CHILL Plan: Pro 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (2)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (7)
Pre-merge checks (2 passed, 1 warning)❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✨ Finishing touches
🧪 Generate unit tests
Comment |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 0
🧹 Nitpick comments (9)
CHANGELOG.md (1)
34-35: Nit: link PR instead of issue for 6068 and keep style consistent.Other entries often link PRs; switch to /pull/6068 for both additions to match convention.
Apply:
- - [#6068](https://github.com/ChainSafe/forest/issues/6068) Added `--index-backfill-epochs` to `forest-tool api serve`. + - [#6068](https://github.com/ChainSafe/forest/pull/6068) Added `--index-backfill-epochs` to `forest-tool api serve`. @@ - - [#6068](https://github.com/ChainSafe/forest/issues/6068) Made `--chain` optional in `forest-tool api serve` by inferring network from the snapshots. + - [#6068](https://github.com/ChainSafe/forest/pull/6068) Made `--chain` optional in `forest-tool api serve` by inferring network from the snapshots.Also applies to: 38-39
scripts/tests/api_compare/docker-compose.yml (1)
138-138: Confirm healthcheck budget with backfill.With
--index-backfill-epochs 50, startup does work more before serving. Ensure the 10mstart_periodis enough on CI runners; otherwise bump it or reduce epochs for this job.documentation/src/offline-forest.md (1)
48-48: Docs: mention the new--index-backfill-epochsflag and optional--chain.Add a brief note in Usage showing
--index-backfill-epochsand that--chaincan be omitted when serving snapshots (still required for devnets with just a genesis).src/tool/offline_server/server.rs (4)
58-66: Chain inference: better error for devnet and explicit logging.Discriminant-only matching avoids false negatives for devnets, but the error message can mislead when both sides are
Devnet(_). Consider improving the message and logging the inferred source (genesis CID) to aid debugging.Apply:
- anyhow::ensure!( - discriminant(&inferred_chain) == discriminant(&chain), - "chain mismatch, specified: {chain}, actual: {inferred_chain}", - ); + anyhow::ensure!( + discriminant(&inferred_chain) == discriminant(&chain), + "chain mismatch (variant differs), specified: {chain}, inferred from genesis: {inferred_chain}", + ); + tracing::info!("Inferred chain from genesis: {inferred_chain}");Also applies to: 68-82
107-114: Saturateto_epochto avoid underflow on large values.Casting
usizetoi64and subtracting can underflow; saturate at 0 for safety.Apply:
- backfill_db( - &state_manager, - &head_ts, - head_ts.epoch() + 1 - index_backfill_epochs as i64, - ) + // Backfill last N epochs inclusive of head; clamp lower bound at 0. + let to_epoch = (head_ts.epoch().saturating_add(1)) + .saturating_sub(index_backfill_epochs as i64) + .max(0); + backfill_db(&state_manager, &head_ts, to_epoch) .await?;
126-133: Typing/logic polish in validation window.
- Fix comment typo.
- Use saturating math to handle very negative
height.Apply:
- // Validate tipsets since the {height} EPOCH when `height >= 0`, - // or valiadte the last {-height} EPOCH(s) when `height < 0` + // Validate tipsets since the {height} EPOCH when `height >= 0`, + // or validate the last {-height} EPOCH(s) when `height < 0` - let validate_until_epoch = if height > 0 { - height - } else { - head_ts.epoch() + height + 1 - }; + let validate_until_epoch = if height > 0 { + height + } else { + head_ts.epoch().saturating_add(height + 1) + };
214-232: Devnet path: clarify snapshot vs. genesis handling.Returning only
genesiswhen no snapshots are supplied is fine, butManyCar::read_only_filesexpects CAR/ForestCAR files. If a plain genesis CAR is passed here, ensure it’s supported (or wrap it via the existing transcode/import flow) to avoid runtime load errors.src/tool/subcommands/api_cmd.rs (2)
66-67: Clarify that --chain is optional and validated against snapshots.Improve the help text to avoid user confusion and document the mismatch check performed downstream.
- /// Filecoin network chain + /// Filecoin network chain. Optional; inferred from snapshots when omitted. + /// If provided, it must match the chain inferred from the snapshots. #[arg(long)] chain: Option<NetworkChain>,
78-81: Add range validation and clarify semantics for --index-backfill-epochs.Prevent unrealistic values (which later cast to i64) and document that 0 disables backfill. This fails fast at parse-time, matching your preference for early failures.
- /// Backfill index for the given EPOCH(s) - #[arg(long, default_value_t = 0)] + /// Backfill index for the given EPOCH(s). 0 disables backfill. + #[arg(long, default_value_t = 0, value_parser = clap::value_parser!(usize).range(0..=i64::MAX as usize))] index_backfill_epochs: usize,
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (6)
CHANGELOG.md(1 hunks)documentation/src/offline-forest.md(1 hunks)scripts/tests/api_compare/docker-compose.yml(1 hunks)src/daemon/db_util.rs(1 hunks)src/tool/offline_server/server.rs(6 hunks)src/tool/subcommands/api_cmd.rs(4 hunks)
🧰 Additional context used
🧠 Learnings (3)
📓 Common learnings
Learnt from: hanabi1224
PR: ChainSafe/forest#5930
File: build.rs:64-77
Timestamp: 2025-08-13T09:43:20.301Z
Learning: hanabi1224 prefers hard compile-time errors in build scripts rather than runtime safeguards or collision detection, believing it's better to fail fast and fix root causes of issues like malformed snapshot names.
📚 Learning: 2025-08-07T09:37:03.079Z
Learnt from: hanabi1224
PR: ChainSafe/forest#5886
File: scripts/tests/calibnet_eth_mapping_check.sh:49-49
Timestamp: 2025-08-07T09:37:03.079Z
Learning: In Forest project scripts, `forest_wait_api` (called within `forest_init`) ensures basic RPC service readiness, while `forest_wait_for_healthcheck_ready` performs additional comprehensive checks including DB backfill completion. When using `--backfill-db` flag, basic RPC operations can proceed after `forest_init`, but operations requiring complete DB backfill should wait for `forest_wait_for_healthcheck_ready`.
Applied to files:
scripts/tests/api_compare/docker-compose.yml
📚 Learning: 2025-08-25T13:35:24.230Z
Learnt from: hanabi1224
PR: ChainSafe/forest#5969
File: src/tool/subcommands/snapshot_cmd.rs:412-412
Timestamp: 2025-08-25T13:35:24.230Z
Learning: In src/tool/subcommands/snapshot_cmd.rs, the +1 in `last_epoch = ts.epoch() - epochs as i64 + 1` fixes an off-by-1 bug where specifying --check-stateroots=N would validate N+1 epochs instead of N epochs, causing out-of-bounds errors when the snapshot contains only N recent state roots.
Applied to files:
src/daemon/db_util.rs
🧬 Code graph analysis (2)
src/tool/subcommands/api_cmd.rs (1)
src/tool/offline_server/server.rs (1)
start_offline_server(44-181)
src/tool/offline_server/server.rs (2)
src/blocks/tipset.rs (2)
chain(390-397)genesis(415-452)src/networks/mod.rs (1)
from_genesis_or_devnet_placeholder(123-125)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (7)
- GitHub Check: tests
- GitHub Check: tests-release
- GitHub Check: Build forest binaries on Linux AMD64
- GitHub Check: Build Ubuntu
- GitHub Check: Build MacOS
- GitHub Check: cargo-publish-dry-run
- GitHub Check: All lint checks
🔇 Additional comments (4)
src/daemon/db_util.rs (1)
339-339: Iterator guard looks good; inclusive window is correct.
take_while(|ts| ts.epoch() >= to_epoch)matches the inclusive semantics and avoids the manualbreak. Nice cleanup.src/tool/subcommands/api_cmd.rs (3)
24-24: Import cleanup is correct.Dropping
ensurealigns with code using onlybail!/with_context. No issues.
266-267: Plumbing the new field looks good.
index_backfill_epochsis correctly threaded through the Serve variant.
270-279: Add a preflight guard when no snapshots are provided and auto-download is off.Return a clear CLI error early instead of deferring to deeper code paths.
} => { + if snapshot_files.is_empty() && !auto_download_snapshot { + bail!("Provide at least one snapshot file or pass --auto-download-snapshot."); + } start_offline_server( snapshot_files, chain, port, auto_download_snapshot, height, index_backfill_epochs, genesis, save_token, )
- Verify there are no lingering call sites using the old start_offline_server signature (search the repo for start_offline_server\s*().
Summary of changes
Changes introduced in this pull request:
--index-backfill-epochstoforest-tool api serve--chainoptional by inferring network from the snapshot filesReference issue to close (if applicable)
Closes
Other information and links
Change checklist
Summary by CodeRabbit
New Features
Documentation
Tests