Skip to content

Conversation

@hanabi1224
Copy link
Contributor

@hanabi1224 hanabi1224 commented Sep 11, 2025

Summary of changes

Changes introduced in this pull request:

  • add --index-backfill-epochs to forest-tool api serve
  • make --chain optional by inferring network from the snapshot files

Reference issue to close (if applicable)

Closes

Other information and links

Change checklist

  • I have performed a self-review of my own code,
  • I have made corresponding changes to the documentation. All new code adheres to the team's documentation standards,
  • I have added tests that prove my fix is effective or that my feature works (if possible),
  • I have made sure the CHANGELOG is up-to-date. All user-facing changes should be reflected in this document.

Summary by CodeRabbit

  • New Features

    • Added --index-backfill-epochs to control how many epochs are backfilled when serving the offline API (default 0).
    • Made --chain optional; the network will be inferred from provided snapshots when possible.
  • Documentation

    • Updated offline serving examples to remove the required --chain flag when using snapshots.
    • Documented the new --index-backfill-epochs option and devnet genesis handling when no snapshots are supplied.
  • Tests

    • Updated test configs to drop explicit --chain and include --index-backfill-epochs.

@coderabbitai
Copy link
Contributor

coderabbitai bot commented Sep 11, 2025

Walkthrough

Made the --chain argument optional for forest-tool api serve by inferring the network from snapshots or DB; added a new --index-backfill-epochs flag and threaded that option into offline server backfill logic and related call sites. Also refactored a backfill loop to use iterator termination.

Changes

Cohort / File(s) Summary
CLI command changes
src/tool/subcommands/api_cmd.rs
Made chain an Option<NetworkChain>, added index_backfill_epochs: usize CLI flag (default 0), removed devnet-specific ensure, and pass index_backfill_epochs to start_offline_server.
Offline server API & logic
src/tool/offline_server/server.rs
start_offline_server signature changed to accept chain: Option<NetworkChain> and index_backfill_epochs: usize. Added chain inference/validation from DB/genesis, adjusted snapshot handling to accept optional chain, and implemented backfill range controlled by index_backfill_epochs.
Backfill loop refactor
src/daemon/db_util.rs
Replaced explicit loop break with `.take_while(
Docs / examples
documentation/src/offline-forest.md
Removed explicit --chain calibnet from example invocation, showing snapshot-only invocation.
Test infra / docker-compose
scripts/tests/api_compare/docker-compose.yml
Adjusted offline RPC service command: removed --chain calibnet, added --index-backfill-epochs 50, reordered flags.
Changelog
CHANGELOG.md
Documented new --index-backfill-epochs flag and the --chain becoming optional via snapshot inference.

Sequence Diagram(s)

sequenceDiagram
    participant CLI as forest-tool (CLI)
    participant Server as OfflineServer
    participant DB as DB
    participant Snap as SnapshotHandler
    participant Index as BackfillIndexer

    CLI->>Server: start_offline_server(snapshot_files, chain?, height, index_backfill_epochs, ...)
    alt chain not provided
        Server->>DB: infer chain from DB (genesis/head)
        DB-->>Server: inferred_chain
        Server->>Server: validate provided chain discriminant if present
    end
    Server->>Snap: handle_snapshots(snapshot_files, chain?)
    Snap-->>Server: loaded_snapshot(s)
    Server->>DB: compute head_epoch / validate tipsets
    Server->>Index: if index_backfill_epochs > 0 -> backfill range (head_epoch ... head_epoch+1-index_backfill_epochs)
    Index-->>DB: write indexed data
    Server-->>CLI: RPC server started (port ...)
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

  • Remove noisy prints #5903 — Modifies DB backfill logic and related backfill invocations; strongly related to the backfill iterator/refactor in this PR.

Suggested reviewers

  • elmattic
  • LesnyRumcajs
  • akaladarshi

📜 Recent review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 83eed1e and dcf09aa.

📒 Files selected for processing (2)
  • CHANGELOG.md (1 hunks)
  • src/tool/subcommands/api_cmd.rs (4 hunks)
🚧 Files skipped from review as they are similar to previous changes (2)
  • src/tool/subcommands/api_cmd.rs
  • CHANGELOG.md
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (7)
  • GitHub Check: cargo-publish-dry-run
  • GitHub Check: Build MacOS
  • GitHub Check: Build Ubuntu
  • GitHub Check: All lint checks
  • GitHub Check: Build forest binaries on Linux AMD64
  • GitHub Check: tests-release
  • GitHub Check: tests

Pre-merge checks (2 passed, 1 warning)

❌ Failed checks (1 warning)
Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. You can run @coderabbitai generate docstrings to improve docstring coverage.
✅ Passed checks (2 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title Check ✅ Passed The title succinctly and accurately describes a primary user-visible change: the addition of the --index-backfill-epochs flag to forest-tool api serve. It is concise, clear, and free of noise; although the PR also makes --chain optional, the title still reflects a significant and central change.
✨ Finishing touches
  • 📝 Generate Docstrings
🧪 Generate unit tests
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch hm/index-backfill-epochs

Comment @coderabbitai help to get the list of available commands and usage tips.

@hanabi1224 hanabi1224 added the RPC requires calibnet RPC checks to run on CI label Sep 11, 2025
@hanabi1224 hanabi1224 marked this pull request as ready for review September 11, 2025 10:56
@hanabi1224 hanabi1224 requested a review from a team as a code owner September 11, 2025 10:56
@hanabi1224 hanabi1224 requested review from LesnyRumcajs and akaladarshi and removed request for a team September 11, 2025 10:56
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (9)
CHANGELOG.md (1)

34-35: Nit: link PR instead of issue for 6068 and keep style consistent.

Other entries often link PRs; switch to /pull/6068 for both additions to match convention.

Apply:

- - [#6068](https://github.com/ChainSafe/forest/issues/6068) Added `--index-backfill-epochs` to `forest-tool api serve`.
+ - [#6068](https://github.com/ChainSafe/forest/pull/6068) Added `--index-backfill-epochs` to `forest-tool api serve`.
@@
- - [#6068](https://github.com/ChainSafe/forest/issues/6068) Made `--chain` optional in `forest-tool api serve` by inferring network from the snapshots.
+ - [#6068](https://github.com/ChainSafe/forest/pull/6068) Made `--chain` optional in `forest-tool api serve` by inferring network from the snapshots.

Also applies to: 38-39

scripts/tests/api_compare/docker-compose.yml (1)

138-138: Confirm healthcheck budget with backfill.

With --index-backfill-epochs 50, startup does work more before serving. Ensure the 10m start_period is enough on CI runners; otherwise bump it or reduce epochs for this job.

documentation/src/offline-forest.md (1)

48-48: Docs: mention the new --index-backfill-epochs flag and optional --chain.

Add a brief note in Usage showing --index-backfill-epochs and that --chain can be omitted when serving snapshots (still required for devnets with just a genesis).

src/tool/offline_server/server.rs (4)

58-66: Chain inference: better error for devnet and explicit logging.

Discriminant-only matching avoids false negatives for devnets, but the error message can mislead when both sides are Devnet(_). Consider improving the message and logging the inferred source (genesis CID) to aid debugging.

Apply:

- anyhow::ensure!(
-     discriminant(&inferred_chain) == discriminant(&chain),
-     "chain mismatch, specified: {chain}, actual: {inferred_chain}",
- );
+ anyhow::ensure!(
+     discriminant(&inferred_chain) == discriminant(&chain),
+     "chain mismatch (variant differs), specified: {chain}, inferred from genesis: {inferred_chain}",
+ );
+ tracing::info!("Inferred chain from genesis: {inferred_chain}");

Also applies to: 68-82


107-114: Saturate to_epoch to avoid underflow on large values.

Casting usize to i64 and subtracting can underflow; saturate at 0 for safety.

Apply:

-        backfill_db(
-            &state_manager,
-            &head_ts,
-            head_ts.epoch() + 1 - index_backfill_epochs as i64,
-        )
+        // Backfill last N epochs inclusive of head; clamp lower bound at 0.
+        let to_epoch = (head_ts.epoch().saturating_add(1))
+            .saturating_sub(index_backfill_epochs as i64)
+            .max(0);
+        backfill_db(&state_manager, &head_ts, to_epoch)
         .await?;

126-133: Typing/logic polish in validation window.

  • Fix comment typo.
  • Use saturating math to handle very negative height.

Apply:

-    // Validate tipsets since the {height} EPOCH when `height >= 0`,
-    // or valiadte the last {-height} EPOCH(s) when `height < 0`
+    // Validate tipsets since the {height} EPOCH when `height >= 0`,
+    // or validate the last {-height} EPOCH(s) when `height < 0`
-    let validate_until_epoch = if height > 0 {
-        height
-    } else {
-        head_ts.epoch() + height + 1
-    };
+    let validate_until_epoch = if height > 0 {
+        height
+    } else {
+        head_ts.epoch().saturating_add(height + 1)
+    };

214-232: Devnet path: clarify snapshot vs. genesis handling.

Returning only genesis when no snapshots are supplied is fine, but ManyCar::read_only_files expects CAR/ForestCAR files. If a plain genesis CAR is passed here, ensure it’s supported (or wrap it via the existing transcode/import flow) to avoid runtime load errors.

src/tool/subcommands/api_cmd.rs (2)

66-67: Clarify that --chain is optional and validated against snapshots.

Improve the help text to avoid user confusion and document the mismatch check performed downstream.

-        /// Filecoin network chain
+        /// Filecoin network chain. Optional; inferred from snapshots when omitted.
+        /// If provided, it must match the chain inferred from the snapshots.
         #[arg(long)]
         chain: Option<NetworkChain>,

78-81: Add range validation and clarify semantics for --index-backfill-epochs.

Prevent unrealistic values (which later cast to i64) and document that 0 disables backfill. This fails fast at parse-time, matching your preference for early failures.

-        /// Backfill index for the given EPOCH(s)
-        #[arg(long, default_value_t = 0)]
+        /// Backfill index for the given EPOCH(s). 0 disables backfill.
+        #[arg(long, default_value_t = 0, value_parser = clap::value_parser!(usize).range(0..=i64::MAX as usize))]
         index_backfill_epochs: usize,
📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 4b4411a and 83eed1e.

📒 Files selected for processing (6)
  • CHANGELOG.md (1 hunks)
  • documentation/src/offline-forest.md (1 hunks)
  • scripts/tests/api_compare/docker-compose.yml (1 hunks)
  • src/daemon/db_util.rs (1 hunks)
  • src/tool/offline_server/server.rs (6 hunks)
  • src/tool/subcommands/api_cmd.rs (4 hunks)
🧰 Additional context used
🧠 Learnings (3)
📓 Common learnings
Learnt from: hanabi1224
PR: ChainSafe/forest#5930
File: build.rs:64-77
Timestamp: 2025-08-13T09:43:20.301Z
Learning: hanabi1224 prefers hard compile-time errors in build scripts rather than runtime safeguards or collision detection, believing it's better to fail fast and fix root causes of issues like malformed snapshot names.
📚 Learning: 2025-08-07T09:37:03.079Z
Learnt from: hanabi1224
PR: ChainSafe/forest#5886
File: scripts/tests/calibnet_eth_mapping_check.sh:49-49
Timestamp: 2025-08-07T09:37:03.079Z
Learning: In Forest project scripts, `forest_wait_api` (called within `forest_init`) ensures basic RPC service readiness, while `forest_wait_for_healthcheck_ready` performs additional comprehensive checks including DB backfill completion. When using `--backfill-db` flag, basic RPC operations can proceed after `forest_init`, but operations requiring complete DB backfill should wait for `forest_wait_for_healthcheck_ready`.

Applied to files:

  • scripts/tests/api_compare/docker-compose.yml
📚 Learning: 2025-08-25T13:35:24.230Z
Learnt from: hanabi1224
PR: ChainSafe/forest#5969
File: src/tool/subcommands/snapshot_cmd.rs:412-412
Timestamp: 2025-08-25T13:35:24.230Z
Learning: In src/tool/subcommands/snapshot_cmd.rs, the +1 in `last_epoch = ts.epoch() - epochs as i64 + 1` fixes an off-by-1 bug where specifying --check-stateroots=N would validate N+1 epochs instead of N epochs, causing out-of-bounds errors when the snapshot contains only N recent state roots.

Applied to files:

  • src/daemon/db_util.rs
🧬 Code graph analysis (2)
src/tool/subcommands/api_cmd.rs (1)
src/tool/offline_server/server.rs (1)
  • start_offline_server (44-181)
src/tool/offline_server/server.rs (2)
src/blocks/tipset.rs (2)
  • chain (390-397)
  • genesis (415-452)
src/networks/mod.rs (1)
  • from_genesis_or_devnet_placeholder (123-125)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (7)
  • GitHub Check: tests
  • GitHub Check: tests-release
  • GitHub Check: Build forest binaries on Linux AMD64
  • GitHub Check: Build Ubuntu
  • GitHub Check: Build MacOS
  • GitHub Check: cargo-publish-dry-run
  • GitHub Check: All lint checks
🔇 Additional comments (4)
src/daemon/db_util.rs (1)

339-339: Iterator guard looks good; inclusive window is correct.

take_while(|ts| ts.epoch() >= to_epoch) matches the inclusive semantics and avoids the manual break. Nice cleanup.

src/tool/subcommands/api_cmd.rs (3)

24-24: Import cleanup is correct.

Dropping ensure aligns with code using only bail!/with_context. No issues.


266-267: Plumbing the new field looks good.

index_backfill_epochs is correctly threaded through the Serve variant.


270-279: Add a preflight guard when no snapshots are provided and auto-download is off.

Return a clear CLI error early instead of deferring to deeper code paths.

             } => {
+                if snapshot_files.is_empty() && !auto_download_snapshot {
+                    bail!("Provide at least one snapshot file or pass --auto-download-snapshot.");
+                }
                 start_offline_server(
                     snapshot_files,
                     chain,
                     port,
                     auto_download_snapshot,
                     height,
                     index_backfill_epochs,
                     genesis,
                     save_token,
                 )
  • Verify there are no lingering call sites using the old start_offline_server signature (search the repo for start_offline_server\s*().

@hanabi1224 hanabi1224 removed the RPC requires calibnet RPC checks to run on CI label Sep 11, 2025
@hanabi1224 hanabi1224 added the RPC requires calibnet RPC checks to run on CI label Sep 11, 2025
@LesnyRumcajs
Copy link
Member

@akaladarshi

@hanabi1224 hanabi1224 added this pull request to the merge queue Sep 16, 2025
Merged via the queue into main with commit 02861dd Sep 16, 2025
71 of 72 checks passed
@hanabi1224 hanabi1224 deleted the hm/index-backfill-epochs branch September 16, 2025 08:03
@coderabbitai coderabbitai bot mentioned this pull request Sep 17, 2025
4 tasks
@coderabbitai coderabbitai bot mentioned this pull request Sep 29, 2025
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

RPC requires calibnet RPC checks to run on CI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants