Skip to content

Conversation

@hanabi1224
Copy link
Contributor

@hanabi1224 hanabi1224 commented Aug 1, 2025

Summary of changes

(On top of #5835)

Changes introduced in this pull request:

  • implement F3 snapshot import and export with go-f3 FFI
  • add forest-tool archive merge-f3 subcommand for merging a v1 Filecoin snapshot and an F3 snapshot into a v2 Filecoin snapshot. (used in CI test before we start publishing v2 snapshots)
  • add CI tests

Reference issue to close (if applicable)

Closes #5809

Other information and links

Change checklist

  • I have performed a self-review of my own code,
  • I have made corresponding changes to the documentation. All new code adheres to the team's documentation standards,
  • I have added tests that prove my fix is effective or that my feature works (if possible),
  • I have made sure the CHANGELOG is up-to-date. All user-facing changes should be reflected in this document.

Summary by CodeRabbit

  • New Features
    • Added RPC method to export the latest F3 snapshot to a file (F3.ExportLatestSnapshot).
    • V2 chain export now embeds the latest F3 snapshot when available.
    • forest-tool: new “archive merge-f3” to combine a v1 Filecoin snapshot with an F3 snapshot into v2; “archive metadata” now prints F3 header info.
    • Automatic F3 snapshot import during snapshot-based initialization; improved F3 sidecar startup.
  • Documentation
    • Added CLI docs for “archive metadata” and “archive merge-f3”.
  • Chores
    • Renamed CI workflow and included it in integration status.

@coderabbitai
Copy link
Contributor

coderabbitai bot commented Aug 1, 2025

Walkthrough

Adds F3 snapshot import/export support across sidecar, RPC, daemon, DB, and tooling. Introduces F3 snapshot CID/header handling, merges v1 Filecoin + F3 into v2 snapshots, and wires export of latest F3 snapshot into chain export. Updates CI, scripts, and docs accordingly.

Changes

Cohort / File(s) Summary
CI and test scripts
.github/workflows/forest.yml, scripts/tests/calibnet_export_f3_check.sh, scripts/tests/calibnet_eth_mapping_check.sh, scripts/tests/harness.sh
Rename job, switch to F3 export check script, aggregate status. Add F3 export validation script. Add F3-aware snapshot download/import/init helpers and readiness wait function. Replace direct healthcheck CLI with helper.
f3-sidecar (Go) runtime + FFI
f3-sidecar/api.go, f3-sidecar/import.go, f3-sidecar/run.go, f3-sidecar/utils.go, f3-sidecar/main.go, f3-sidecar/ffi_gen.go, f3-sidecar/ffi_impl.go, f3-sidecar/go.mod
Add latest F3 snapshot export API, snapshot import flow, helper utilities, new CLI flag, refactors for datastore/network waits, and FFI method import_snap. Bump go-f3 dependency and add go-datastore.
RPC: F3 methods and chain export
src/rpc/methods/f3.rs, src/rpc/mod.rs, src/rpc/methods/chain.rs, src/rpc/methods/f3/types.rs
Add RPC F3.ExportLatestSnapshot returning CID; update F3 methods to unwrap LotusJson; integrate F3 snapshot export into chain export v2; introduce tuple-serde F3PowerEntry with HasLotusJson mapping.
F3 module and snapshot handling (Rust)
src/f3/mod.rs, src/f3/go_ffi.rs, src/f3/snapshot.rs, src/f3/snapshot/tests.rs
Add get_f3_root, run signature cleanup, and import_f3_snapshot. Implement F3 snapshot CID computation and header decoding/Display with tests. Extend Go FFI trait with import_snap.
DB CAR readers
src/db/car/forest.rs, src/db/car/plain.rs, src/db/car/any.rs
Add streaming get_reader for ForestCar, PlainCar, and AnyCar to read blocks by CID without full materialization; update tests.
Daemon wiring
src/daemon/mod.rs, src/daemon/db_util.rs, src/daemon/context.rs, src/cli_shared/cli/client.rs
Pass rpc_v1_endpoint, f3_root, and chain_config into snapshot import; start F3 sidecar with unified endpoint/root; propagate errors. Add chain_config accessor and rpc_v1_endpoint helper.
Chain export and formatting
src/chain/mod.rs, src/chain/snapshot_format.rs
Use get_f3_snapshot_cid and length via seek for F3 data in export_v2; adjust Display alignment of snapshot metadata.
Tooling and docs
src/tool/subcommands/archive_cmd.rs, src/tool/subcommands/api_cmd/test_snapshots_ignored.txt, docs/docs/users/reference/cli.sh, CHANGELOG.md
Add forest-tool archive merge-f3 and enhanced metadata printing; ignore new RPC in API tests; document new subcommands; update changelog.

Sequence Diagram(s)

sequenceDiagram
  participant User
  participant RPC_Client as Forest CLI/RPC client
  participant Node as Forest Node
  participant F3 as F3 Sidecar
  participant Export as Export v2

  User->>RPC_Client: Call Forest.ChainExport(path, v2)
  RPC_Client->>Node: JSON-RPC Forest.ChainExport
  Node->>F3: RPC F3.ExportLatestSnapshot(tmp_f3_path)
  F3-->>Node: CID(f3), write tmp file
  Node->>Export: export_v2(output, head, (CID(f3), reader(tmp_f3)))
  Export-->>Node: Write CAR v2 with metadata+F3 block+chain blocks
  Node-->>RPC_Client: Ok
  RPC_Client-->>User: Success
Loading
sequenceDiagram
  participant User
  participant Tool as forest-tool
  participant FS as Filesystem

  User->>Tool: archive merge-f3 v1.car[.zst] f3.bin out.forest.car.zst
  Tool->>Tool: Read v1 head roots
  Tool->>Tool: Compute CID(f3), build metadata
  Tool->>FS: Encode frames (metadata, f3, blocks) -> tmp
  FS-->>Tool: tmp written
  Tool->>FS: Persist tmp -> out.forest.car.zst
  Tool-->>User: Done
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Assessment against linked issues

Objective Addressed Explanation
Implement F3 snapshot import and export APIs in go-f3 (#5809)

Assessment against linked issues: Out-of-scope changes

Code Change Explanation
Adjusted Display formatting/spacing for FilecoinSnapshotMetadata (src/chain/snapshot_format.rs, multiple lines) Cosmetic formatting not tied to implementing F3 import/export APIs.
Replace direct healthcheck CLI call with helper function (scripts/tests/calibnet_eth_mapping_check.sh, multiple lines) Test harness readiness change unrelated to F3 import/export API implementation.

Possibly related PRs

Suggested labels

RPC

Suggested reviewers

  • akaladarshi
  • LesnyRumcajs
  • sudo-shashank

📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 196c884 and c2621bc.

📒 Files selected for processing (1)
  • src/db/car/forest.rs (3 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
  • src/db/car/forest.rs
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (11)
  • GitHub Check: All lint checks
  • GitHub Check: cargo-publish-dry-run
  • GitHub Check: Build Ubuntu
  • GitHub Check: Build MacOS
  • GitHub Check: Check
  • GitHub Check: Build forest binaries on Linux AMD64
  • GitHub Check: Deploy to Cloudflare Pages
  • GitHub Check: tests-release
  • GitHub Check: tests
  • GitHub Check: Analyze (go)
  • GitHub Check: Analyze (rust)
✨ Finishing Touches
  • 📝 Generate Docstrings
🧪 Generate unit tests
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch hm/export-f3-data

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR/Issue comments)

Type @coderabbitai help to get the list of available commands.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Status, Documentation and Community

  • Visit our Status Page to check the current availability of CodeRabbit.
  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

♻️ Duplicate comments (1)
src/f3/snapshot.rs (1)

21-27: Add documentation for the F3SnapshotHeader struct and its fields.

The struct and its fields lack documentation explaining their purpose and usage. This would improve code maintainability and help other developers understand the F3 snapshot format.

Consider adding documentation like:

 /// Defined in <https://github.com/filecoin-project/FIPs/blob/98e33b9fa306959aa0131519eb4cc155522b2081/FRCs/frc-0108.md#f3snapshotheader>
+/// Represents the header of an F3 snapshot containing versioning and power table information.
 #[derive(Debug, Clone, Eq, PartialEq, Serialize_tuple, Deserialize_tuple)]
 pub struct F3SnapshotHeader {
+    /// Version of the F3 snapshot format
     pub version: u64,
+    /// The first F3 instance number in the snapshot
     pub first_instance: u64,
+    /// The latest F3 instance number in the snapshot
     pub latest_instance: u64,
+    /// Initial power table entries for F3 consensus
     pub initial_power_table: Vec<F3PowerEntry>,
 }
🧹 Nitpick comments (6)
src/cli_shared/cli/client.rs (1)

106-110: Consider making the protocol configurable or document the HTTP-only limitation.

The method hardcodes the HTTP protocol. If RPC endpoints might need HTTPS support in the future, consider making the protocol configurable or at least document that this method only supports HTTP endpoints.

src/tool/subcommands/archive_cmd.rs (1)

676-687: Consider adding error context for frame creation failures.

The frame creation uses anyhow::Ok directly. Consider adding more context to potential errors to aid debugging:

 let snap_meta_frame = {
     let mut encoder =
         crate::db::car::forest::new_encoder(DEFAULT_FOREST_CAR_COMPRESSION_LEVEL)?;
     snap_meta_block.write(&mut encoder)?;
-    anyhow::Ok((
+    Ok::<_, anyhow::Error>((
         vec![snap_meta_block.cid],
         crate::db::car::forest::finalize_frame(
             DEFAULT_FOREST_CAR_COMPRESSION_LEVEL,
             &mut encoder,
-        )?,
+        ).context("Failed to finalize metadata frame")?,
     ))
 };
src/daemon/mod.rs (4)

143-146: New args to import_chain_as_forest_car look correct; minor readability nit

Passing rpc_v1_endpoint, f3_root, and chain_config into the import is aligned with the new API. For clarity and to avoid taking a reference to a temporary, consider binding f3_root before the call.

-        let (car_db_path, ts) = import_chain_as_forest_car(
+        let f3_root = crate::f3::get_f3_root(config);
+        let (car_db_path, ts) = import_chain_as_forest_car(
             path,
             &ctx.db_meta_data.get_forest_car_db_dir(),
             config.client.import_mode,
-            config.client.rpc_v1_endpoint()?,
-            &crate::f3::get_f3_root(config),
-            ctx.chain_config(),
+            config.client.rpc_v1_endpoint()?,
+            &f3_root,
+            ctx.chain_config(),
             &snapshot_tracker,
         )
         .await?;

121-132: Prefer a consistent way to access ChainConfig throughout the file

This block uses state_manager.chain_config() while the call below uses ctx.chain_config(). Pick one approach for consistency across the module.

Also applies to: 146-146


405-409: Nit: trailing space and clarify warning message

There’s a trailing space in the warn! string. Consider tightening the message to make it actionable (e.g., mention how to enable RPC).

-            tracing::warn!("F3 sidecar is enabled but not run because RPC is disabled. ")
+            tracing::warn!("F3 sidecar is enabled but not started because RPC is disabled. Enable RPC to start the sidecar.")

416-445: Consider supervising the sidecar thread (optional)

spawn_blocking’s JoinHandle is dropped; if the sidecar exits early, we won’t notice. If practical, attach basic supervision/logging or integrate it into the services JoinSet so failures propagate.

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between ca4e52d and 5243383.

📒 Files selected for processing (7)
  • scripts/tests/harness.sh (3 hunks)
  • src/cli_shared/cli/client.rs (1 hunks)
  • src/daemon/db_util.rs (5 hunks)
  • src/daemon/mod.rs (5 hunks)
  • src/f3/mod.rs (3 hunks)
  • src/f3/snapshot.rs (1 hunks)
  • src/tool/subcommands/archive_cmd.rs (7 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
  • scripts/tests/harness.sh
🧰 Additional context used
🧠 Learnings (1)
📚 Learning: 2025-08-06T14:36:08.654Z
Learnt from: hanabi1224
PR: ChainSafe/forest#5886
File: src/daemon/db_util.rs:244-259
Timestamp: 2025-08-06T14:36:08.654Z
Learning: The `import_f3_snapshot` function already prints success messages internally, so additional success logging at the call site is not needed.

Applied to files:

  • src/daemon/db_util.rs
🧬 Code Graph Analysis (5)
src/daemon/db_util.rs (5)
src/daemon/context.rs (1)
  • chain_config (65-67)
src/rpc/mod.rs (1)
  • chain_config (474-476)
src/networks/mod.rs (2)
  • chain_config (715-720)
  • devnet (342-366)
src/db/car/forest.rs (3)
  • try_from (242-244)
  • new (107-125)
  • std (110-110)
src/f3/mod.rs (1)
  • import_f3_snapshot (120-144)
src/f3/snapshot.rs (4)
src/rpc/methods/f3.rs (1)
  • fvm_ipld_encoding (895-895)
src/rpc/methods/f3/types.rs (1)
  • fvm_ipld_encoding (365-365)
src/utils/encoding/fallback_de_ipld_dagcbor.rs (2)
  • tuple (479-494)
  • from_slice (86-95)
src/chain/snapshot_format.rs (1)
  • fmt (76-92)
src/daemon/mod.rs (3)
src/f3/mod.rs (2)
  • get_f3_root (26-35)
  • is_sidecar_ffi_enabled (147-162)
build.rs (1)
  • is_sidecar_ffi_enabled (40-46)
src/rpc/methods/f3.rs (1)
  • get_f3_rpc_endpoint (936-942)
src/f3/mod.rs (3)
build.rs (1)
  • is_sidecar_ffi_enabled (40-46)
f3-sidecar/ffi_gen.go (1)
  • GoF3NodeImpl (29-29)
src/f3/go_ffi.rs (2)
  • run (12-20)
  • import_snap (22-22)
src/tool/subcommands/archive_cmd.rs (3)
src/f3/snapshot.rs (2)
  • decode_from_snapshot (30-42)
  • get_f3_snapshot_cid (13-18)
src/utils/db/car_stream.rs (2)
  • new (244-247)
  • write (56-58)
src/chain/snapshot_format.rs (1)
  • new_v2 (70-72)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (11)
  • GitHub Check: Check
  • GitHub Check: tests-release
  • GitHub Check: tests
  • GitHub Check: Build MacOS
  • GitHub Check: cargo-publish-dry-run
  • GitHub Check: Build Ubuntu
  • GitHub Check: Build forest binaries on Linux AMD64
  • GitHub Check: All lint checks
  • GitHub Check: Deploy to Cloudflare Pages
  • GitHub Check: Analyze (rust)
  • GitHub Check: Analyze (go)
🔇 Additional comments (9)
src/f3/snapshot.rs (1)

30-42: Good implementation of size validation to prevent DoS attacks.

The implementation properly validates the header size against a 100 MiB limit before allocation, preventing potential memory exhaustion attacks from malformed snapshots.

src/f3/mod.rs (2)

91-117: Good fix for the misleading underscore prefixes.

The removal of underscore prefixes and use of explicit parameter names is clearer and more accurate since these parameters are actually used in the FFI-enabled builds.


119-144: Consider the implications of soft failure for F3 snapshot import.

The function logs F3 import errors but continues execution (returns Ok(())). While this prevents F3 issues from blocking the main chain import, it might hide critical consensus-related problems. Consider whether F3 import failures should be surfaced more prominently or at least tracked via metrics.

src/tool/subcommands/archive_cmd.rs (1)

641-732: Well-implemented merge functionality with proper atomic writes.

The implementation correctly:

  • Validates the input is v1 format before processing
  • Uses atomic file operations with tempfile to prevent partial writes
  • Properly handles the F3 snapshot CID computation
  • Provides good user feedback with progress indication
src/daemon/db_util.rs (2)

134-136: Good use of strongly typed Url parameter.

Using Url type instead of a plain string provides compile-time validation and better type safety.


238-259: Improved temporary file handling addresses previous concerns.

The implementation correctly:

  • Uses tempfile crate which automatically cleans up on drop
  • Calls sync_all() to ensure data is written to disk
  • Maintains the soft-failure approach consistent with the F3 module

The temporary file will be automatically cleaned up even if an error occurs, addressing the resource management concern from the previous review.

src/daemon/mod.rs (3)

398-402: Good change: return anyhow::Result and early Ok(()) on no-op

Returning Result enables error propagation (e.g., from rpc_v1_endpoint parsing) and makes startup sequencing more deterministic.


581-582: Good: propagate maybe_start_f3_service errors

This ensures invalid RPC endpoints or other setup errors surface during startup.


411-445: Gate lease manager initialization behind FFI check
Move the global F3_LEASE_MANAGER.set(...) inside the is_sidecar_ffi_enabled guard so that the manager is only initialized when the sidecar FFI is actually enabled. Although the sidecar startup itself checks is_sidecar_ffi_enabled, the lease manager must be gated as well to avoid leaving the system in a “set” state when FFI is disabled.

• File: src/daemon/mod.rs, around lines 412–445
• Wrap the spawn_blocking (and .set(...)) in an if !crate::f3::is_sidecar_ffi_enabled(...) { return Ok(()); } block

Suggested diff:

     if !opts.halt_after_import && !opts.stateless {
         let rpc_endpoint = config.client.rpc_v1_endpoint()?;
         let state_manager = &ctx.state_manager;
         let p2p_peer_id = ctx.p2p_peer_id;
         let admin_jwt = ctx.admin_jwt.clone();

+        if !crate::f3::is_sidecar_ffi_enabled(state_manager.chain_config()) {
+            tracing::info!("F3 sidecar is disabled; skipping lease manager initialization.");
+            return Ok(());
+        }

         tokio::task::spawn_blocking({
             crate::rpc::f3::F3_LEASE_MANAGER
                 .set(crate::rpc::f3::F3LeaseManager::new(
                     state_manager.chain_config().network.clone(),
                     p2p_peer_id,
                 ))
                 .expect("F3 lease manager should not have been initialized before");
             // …
         });
     }

Please also verify whether run_f3_sidecar_if_enabled expects a full HTTP URL (as returned by rpc_v1_endpoint()) or only a bare host:port. If it requires just host:port, switch to using config.client.rpc_address.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

♻️ Duplicate comments (1)
src/daemon/db_util.rs (1)

250-257: Non-fatal behavior acknowledged; consider surfacing a soft-result for observability

You intentionally log and proceed on F3 import failure. Consider returning a soft status (e.g., Result<(), anyhow::Error> alongside the main result, or emitting a distinct metric) so callers/CI can detect partial-success scenarios without failing the whole import.

🧹 Nitpick comments (3)
src/daemon/db_util.rs (3)

236-259: Avoid unnecessary I/O when F3 is disabled in config

Currently you always build and write a temp file before import_f3_snapshot, which itself may no-op if F3 is disabled. Guard the copy-and-import with chain_config.f3_enabled to save I/O.

-    if let Some(f3_cid) = forest_car.metadata().as_ref().and_then(|m| m.f3_data) {
+    if chain_config.f3_enabled {
+        if let Some(f3_cid) = forest_car.metadata().as_ref().and_then(|m| m.f3_data) {
             // existing copy and import logic...
-    }
+        }
+    } else {
+        tracing::debug!("F3 disabled in chain config; skipping F3 import");
+    }

If you prefer exact gating consistent with the FFI feature flags, consider exposing a pub fn is_sidecar_ffi_enabled(&ChainConfig) -> bool in crate::f3 and using that here.


250-255: Nit: prefer lossless-to-lossy conversion for paths over display().to_string()

display().to_string() can be surprising on non-UTF8 paths. to_string_lossy() is more explicit about lossy conversion when passing to FFI.

-            rpc_endpoint.to_string(),
-            f3_root.display().to_string(),
-            temp_f3_snap.path().display().to_string(),
+            rpc_endpoint.to_string(),
+            f3_root.as_os_str().to_string_lossy().into_owned(),
+            temp_f3_snap.path().as_os_str().to_string_lossy().into_owned(),

488-491: Tests updated for new params: LGTM

Using Url and a placeholder f3_root with ChainConfig::devnet() is appropriate. Consider adding a unit/integration test that covers:

  • Snapshot with F3 metadata + f3_enabled=false to confirm we skip gracefully.
  • Snapshot with F3 metadata + f3_enabled=true where sidecar is enabled (behind a feature/integration flag).

I can draft such tests behind a cfg flag if helpful.

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 5243383 and 196c884.

📒 Files selected for processing (1)
  • src/daemon/db_util.rs (5 hunks)
🧰 Additional context used
🧠 Learnings (1)
📚 Learning: 2025-08-06T14:36:08.654Z
Learnt from: hanabi1224
PR: ChainSafe/forest#5886
File: src/daemon/db_util.rs:244-259
Timestamp: 2025-08-06T14:36:08.654Z
Learning: The `import_f3_snapshot` function already prints success messages internally, so additional success logging at the call site is not needed.

Applied to files:

  • src/daemon/db_util.rs
🧬 Code Graph Analysis (1)
src/daemon/db_util.rs (2)
src/networks/mod.rs (2)
  • chain_config (715-720)
  • devnet (342-366)
src/f3/mod.rs (1)
  • import_f3_snapshot (120-144)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (11)
  • GitHub Check: cargo-publish-dry-run
  • GitHub Check: Build Ubuntu
  • GitHub Check: Build MacOS
  • GitHub Check: Build forest binaries on Linux AMD64
  • GitHub Check: Check
  • GitHub Check: tests-release
  • GitHub Check: tests
  • GitHub Check: Deploy to Cloudflare Pages
  • GitHub Check: All lint checks
  • GitHub Check: Analyze (rust)
  • GitHub Check: Analyze (go)
🔇 Additional comments (4)
src/daemon/db_util.rs (4)

10-10: Importing ChainConfig is appropriate for F3 gating context

Strong typing here aligns with new F3 snapshot import flow.


21-21: OsStr import is correct for filename handling

Needed for OsStr::to_str usage when scanning CAR files.


261-261: Good change: source heaviest tipset from the loaded ForestCar

This removes any ambiguity with path-derived context and aligns with the newly persisted CAR.


134-137: Incorrect scope–no matching function signature found
The parameters rpc_endpoint: Url, f3_root: &Path, chain_config: &ChainConfig, snapshot_progress_tracker: &SnapshotProgressTracker do not appear in src/daemon/db_util.rs. The only nearby function is import_chain_as_forest_car(from_path: &Path, forest_car_db_dir: &Path, import_mode: ImportMode). No call sites refer to a signature with the four new parameters.
Please disregard this comment or correct the target function.

Likely an incorrect or invalid review comment.

Comment on lines +236 to +259
let forest_car = ForestCar::try_from(forest_car_db_path.as_path())?;

if let Some(f3_cid) = forest_car.metadata().as_ref().and_then(|m| m.f3_data) {
let mut f3_data = forest_car
.get_reader(f3_cid)?
.with_context(|| format!("f3 data not found, cid: {f3_cid}"))?;
let mut temp_f3_snap = tempfile::Builder::new()
.suffix(".f3snap.bin")
.tempfile_in(forest_car_db_dir)?;
{
let f = temp_f3_snap.as_file_mut();
std::io::copy(&mut f3_data, f)?;
f.sync_all()?;
}
if let Err(e) = crate::f3::import_f3_snapshot(
chain_config,
rpc_endpoint.to_string(),
f3_root.display().to_string(),
temp_f3_snap.path().display().to_string(),
) {
// Do not make it a hard error if anything is wrong with F3 snapshot
tracing::error!("Failed to import F3 snapshot: {e}");
}
}
Copy link
Contributor

@coderabbitai coderabbitai bot Aug 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Inconsistent non-fatal semantics: missing F3 data currently aborts the whole import

You intend to make F3 issues non-fatal, but if the F3 CID is present and the reader is missing, the current with_context(...)? causes a hard error. Prefer warning and skipping the F3 step instead.

Apply this refactor to skip F3 import when the data is missing while keeping the rest unchanged:

-    if let Some(f3_cid) = forest_car.metadata().as_ref().and_then(|m| m.f3_data) {
-        let mut f3_data = forest_car
-            .get_reader(f3_cid)?
-            .with_context(|| format!("f3 data not found, cid: {f3_cid}"))?;
-        let mut temp_f3_snap = tempfile::Builder::new()
-            .suffix(".f3snap.bin")
-            .tempfile_in(forest_car_db_dir)?;
-        {
-            let f = temp_f3_snap.as_file_mut();
-            std::io::copy(&mut f3_data, f)?;
-            f.sync_all()?;
-        }
-        if let Err(e) = crate::f3::import_f3_snapshot(
-            chain_config,
-            rpc_endpoint.to_string(),
-            f3_root.display().to_string(),
-            temp_f3_snap.path().display().to_string(),
-        ) {
-            // Do not make it a hard error if anything is wrong with F3 snapshot
-            tracing::error!("Failed to import F3 snapshot: {e}");
-        }
-    }
+    if let Some(f3_cid) = forest_car.metadata().as_ref().and_then(|m| m.f3_data) {
+        if let Some(mut f3_data) = forest_car.get_reader(f3_cid)? {
+            let mut temp_f3_snap = tempfile::Builder::new()
+                .suffix(".f3snap.bin")
+                .tempfile_in(forest_car_db_dir)?;
+            {
+                let f = temp_f3_snap.as_file_mut();
+                std::io::copy(&mut f3_data, f)?;
+                f.sync_all()?;
+            }
+            if let Err(e) = crate::f3::import_f3_snapshot(
+                chain_config,
+                rpc_endpoint.to_string(),
+                f3_root.display().to_string(),
+                temp_f3_snap.path().display().to_string(),
+            ) {
+                // Do not make it a hard error if anything is wrong with F3 snapshot
+                tracing::error!("Failed to import F3 snapshot: {e}");
+            }
+        } else {
+            tracing::warn!("F3 data not found, cid: {f3_cid} — skipping F3 import");
+        }
+    }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
let forest_car = ForestCar::try_from(forest_car_db_path.as_path())?;
if let Some(f3_cid) = forest_car.metadata().as_ref().and_then(|m| m.f3_data) {
let mut f3_data = forest_car
.get_reader(f3_cid)?
.with_context(|| format!("f3 data not found, cid: {f3_cid}"))?;
let mut temp_f3_snap = tempfile::Builder::new()
.suffix(".f3snap.bin")
.tempfile_in(forest_car_db_dir)?;
{
let f = temp_f3_snap.as_file_mut();
std::io::copy(&mut f3_data, f)?;
f.sync_all()?;
}
if let Err(e) = crate::f3::import_f3_snapshot(
chain_config,
rpc_endpoint.to_string(),
f3_root.display().to_string(),
temp_f3_snap.path().display().to_string(),
) {
// Do not make it a hard error if anything is wrong with F3 snapshot
tracing::error!("Failed to import F3 snapshot: {e}");
}
}
let forest_car = ForestCar::try_from(forest_car_db_path.as_path())?;
if let Some(f3_cid) = forest_car.metadata().as_ref().and_then(|m| m.f3_data) {
if let Some(mut f3_data) = forest_car.get_reader(f3_cid)? {
let mut temp_f3_snap = tempfile::Builder::new()
.suffix(".f3snap.bin")
.tempfile_in(forest_car_db_dir)?;
{
let f = temp_f3_snap.as_file_mut();
std::io::copy(&mut f3_data, f)?;
f.sync_all()?;
}
if let Err(e) = crate::f3::import_f3_snapshot(
chain_config,
rpc_endpoint.to_string(),
f3_root.display().to_string(),
temp_f3_snap.path().display().to_string(),
) {
// Do not make it a hard error if anything is wrong with F3 snapshot
tracing::error!("Failed to import F3 snapshot: {e}");
}
} else {
tracing::warn!("F3 data not found, cid: {f3_cid} — skipping F3 import");
}
}

Copy link
Contributor Author

@hanabi1224 hanabi1224 Aug 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We try to forgive bad F3 data during import which could be caused by error in external Go go-f3 package, but when the F3 CID is present in the header while F3 data is missing, it indicates the snapshot is bad.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hanabi1224, I understand your reasoning now. You're absolutely right - if the F3 CID is present in the snapshot metadata but the corresponding F3 data is missing, this indicates snapshot corruption rather than just an F3-specific issue. Failing fast with a hard error in this case is the correct approach to maintain data integrity.

The distinction is clear:

  • Missing F3 data when CID is present → snapshot corruption → hard error (current behavior)
  • F3 import failure → F3-specific issue → log error and continue (line 256)

Thank you for the clarification. The current implementation correctly handles both scenarios.


✏️ Learnings added
Learnt from: hanabi1224
PR: ChainSafe/forest#5886
File: src/daemon/db_util.rs:236-259
Timestamp: 2025-08-11T14:00:47.037Z
Learning: In Forest's snapshot import (`src/daemon/db_util.rs`), when F3 CID is present in snapshot metadata but the actual F3 data is missing, this should cause a hard error as it indicates snapshot corruption. Only errors during the F3 import process itself (after data is successfully retrieved) should be non-fatal and logged.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.

Copy link
Collaborator

@akaladarshi akaladarshi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@hanabi1224 hanabi1224 added this pull request to the merge queue Aug 13, 2025
Merged via the queue into main with commit 091b114 Aug 13, 2025
55 checks passed
@hanabi1224 hanabi1224 deleted the hm/export-f3-data branch August 13, 2025 13:14
@coderabbitai coderabbitai bot mentioned this pull request Oct 14, 2025
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement F3 snapshot import and export APIs in go-f3

4 participants