Skip to content

Conversation

@hanabi1224
Copy link
Contributor

@hanabi1224 hanabi1224 commented Sep 5, 2025

Summary of changes

Changes introduced in this pull request:

	Command being timed: "forest --chain calibnet --encrypt-keystore false --halt-after-import --height=-200 --no-gc --import-snapshot https://forest-snapshots.fra1.cdn.digitaloceanspaces.com/debug/filecoin_calibnet_height_16900.car.zst"
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:55.22
	Maximum resident set size (kbytes): 1149436
	
	Command being timed: "forest --chain calibnet --encrypt-keystore false --halt-after-import --height=-200 --no-gc --import-snapshot https://forest-snapshots.fra1.cdn.digitaloceanspaces.com/debug/filecoin_calibnet_height_322454.car.zst"
	Elapsed (wall clock) time (h:mm:ss or m:ss): 1:07.64
	Maximum resident set size (kbytes): 666968
	
	Command being timed: "forest --chain calibnet --encrypt-keystore false --halt-after-import --height=-200 --no-gc --import-snapshot https://forest-snapshots.fra1.cdn.digitaloceanspaces.com/debug/filecoin_calibnet_height_489194.car.zst"
	Elapsed (wall clock) time (h:mm:ss or m:ss): 1:27.85
	Maximum resident set size (kbytes): 1115788
	
	Command being timed: "forest --chain calibnet --encrypt-keystore false --halt-after-import --height=-200 --no-gc --import-snapshot https://forest-snapshots.fra1.cdn.digitaloceanspaces.com/debug/filecoin_calibnet_height_492314.car.zst"
	Elapsed (wall clock) time (h:mm:ss or m:ss): 1:02.93
	Maximum resident set size (kbytes): 566232

Reference issue to close (if applicable)

Closes

Other information and links

Change checklist

  • I have performed a self-review of my own code,
  • I have made corresponding changes to the documentation. All new code adheres to the team's documentation standards,
  • I have added tests that prove my fix is effective or that my feature works (if possible),
  • I have made sure the CHANGELOG is up-to-date. All user-facing changes should be reflected in this document.

Summary by CodeRabbit

  • Tests
    • Test scripts now run under an external resource-measurement tool (/usr/bin/time -v) and no longer use built-in peak-RSS tracking.
  • Documentation
    • Removed the deprecated CLI option --track-peak-rss from user docs.
  • Chores
    • Removed internal peak memory-tracking support and an associated dependency, simplifying runtime and builds.

@coderabbitai
Copy link
Contributor

coderabbitai bot commented Sep 5, 2025

Walkthrough

Removed internal peak-RSS memory-tracking code and CLI flag, deleted the MemStatsTracker module and monitoring re-exports, removed the memory-stats dependency, updated docs and changelog, and replaced internal tracking in tests and helper scripts with external measurement by prefixing invocations with /usr/bin/time -v.

Changes

Cohort / File(s) Summary
CLI surface
src/cli_shared/cli/mod.rs
Removed the pub track_peak_rss: bool CLI option and the --track-peak-rss flag.
Daemon startup
src/daemon/mod.rs
Removed creation/invocation of the MemStatsTracker service and related imports; consolidated imports.
Monitoring module removed
src/utils/monitoring/mod.rs, src/utils/monitoring/mem_tracker.rs, src/utils/mod.rs
Deleted the mem_tracker implementation and its re-exports; removed pub mod monitoring from utils.
Build deps
Cargo.toml
Removed the memory-stats = "1" dependency.
Docs & changelog
docs/docs/users/reference/cli.md, CHANGELOG.md
Removed CLI docs for --track-peak-rss; updated changelog to note removal and recommend external tools.
Scripts / tests
scripts/tests/calibnet_migration_regression_tests.sh, scripts/tests/calibnet_db_migration.sh, scripts/tests/harness.sh
Removed --track-peak-rss usage and wrapped Forest invocations with /usr/bin/time -v; preserved other args and backgrounding behavior.

Sequence Diagram(s)

sequenceDiagram
    autonumber
    participant User
    participant Forest as forest
    participant MemTracker as MemStatsTracker
    participant TimeTool as /usr/bin/time -v

    rect #f0f9ff
    Note over User,Forest: Old flow — internal peak RSS tracking
    User->>Forest: start forest --track-peak-rss ...
    Forest->>MemTracker: spawn tracker
    MemTracker-->>MemTracker: sample memory periodically
    MemTracker-->>Forest: emit peak on drop
    end

    rect #fff7f0
    Note over User,TimeTool: New flow — external resource measurement
    User->>TimeTool: /usr/bin/time -v forest ...
    TimeTool->>Forest: exec forest
    TimeTool-->>User: verbose resource report on exit
    end
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Suggested reviewers

  • akaladarshi
  • LesnyRumcajs

Pre-merge checks (2 passed, 1 warning)

❌ Failed checks (1 warning)
Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. You can run @coderabbitai generate docstrings to improve docstring coverage.
✅ Passed checks (2 passed)
Check name Status Explanation
Title Check ✅ Passed The title ‘test: track peak rss for CI state migration test’ clearly describes the addition of peak RSS measurement in the CI tests, which is indeed part of the changeset, but it does not capture the broader refactoring work such as removal of the internal memory-tracking feature, CLI flag, and related modules. Because it refers to a real aspect of the change while omitting other significant refactoring, it is partially related to the pull request.
Description Check ✅ Passed

📜 Recent review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between ce414c2 and 6eac0bf.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (4)
  • CHANGELOG.md (1 hunks)
  • Cargo.toml (0 hunks)
  • docs/docs/users/reference/cli.md (0 hunks)
  • scripts/tests/harness.sh (1 hunks)
💤 Files with no reviewable changes (2)
  • Cargo.toml
  • docs/docs/users/reference/cli.md
🚧 Files skipped from review as they are similar to previous changes (2)
  • scripts/tests/harness.sh
  • CHANGELOG.md
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (7)
  • GitHub Check: Build forest binaries on Linux AMD64
  • GitHub Check: tests
  • GitHub Check: All lint checks
  • GitHub Check: tests-release
  • GitHub Check: cargo-publish-dry-run
  • GitHub Check: Build Ubuntu
  • GitHub Check: Build MacOS
✨ Finishing Touches
  • 📝 Generate Docstrings
🧪 Generate unit tests
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch hm/track-peak-rss-for-migration-ci-tests

Comment @coderabbitai help to get the list of available commands and usage tips.

@hanabi1224 hanabi1224 marked this pull request as ready for review September 5, 2025 09:48
@hanabi1224 hanabi1224 requested a review from a team as a code owner September 5, 2025 09:48
@hanabi1224 hanabi1224 requested review from LesnyRumcajs and akaladarshi and removed request for a team September 5, 2025 09:48
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (3)
scripts/tests/calibnet_migration_regression_tests.sh (3)

10-10: Make command composition safer using an array (optional)

Using a string for a long command is brittle (word-splitting, future args with spaces). An array is safer.

Apply this diff for the definition:

- M I G R A T I O N _ T E S T ="$FOREST_PATH --chain calibnet --encrypt-keystore false --track-peak-rss --halt-after-import --height=-200 --no-gc --import-snapshot"
+ MIGRATION_TEST=( "$FOREST_PATH" --chain calibnet --encrypt-keystore false --track-peak-rss --halt-after-import --height=-200 --no-gc --import-snapshot )

Then invoke with:

# replace: $MIGRATION_TEST "URL"
"${MIGRATION_TEST[@]}" "URL"

10-10: Optional: gate the flag via env for local runs

If you want CI to always track RSS but keep local runs quiet, gate the flag via env without changing call sites.

Minimal change:

-MIGRATION_TEST="$FOREST_PATH --chain calibnet --encrypt-keystore false --track-peak-rss --halt-after-import --height=-200 --no-gc --import-snapshot"
+MIGRATION_TEST="$FOREST_PATH --chain calibnet --encrypt-keystore false ${TRACK_PEAK_RSS:+--track-peak-rss} --halt-after-import --height=-200 --no-gc --import-snapshot"

Usage:

  • In CI: export TRACK_PEAK_RSS=1
  • Locally (default): omit to skip the flag

10-10: Surface a one-line metric for CI consumption (nice-to-have)

Consider extracting and emitting a single “Peak RSS” line per run to aid trend tracking in job logs.

Example wrapper (no behavior change to the test result):

# After each invocation:
# ${MIGRATION_TEST[@]} "$URL" | tee /dev/stderr | rg -i 'Peak physical memory usage' || true

This keeps full logs while surfacing the metric.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

💡 Knowledge Base configuration:

  • MCP integration is disabled by default for public repositories
  • Jira integration is disabled by default for public repositories
  • Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

📥 Commits

Reviewing files that changed from the base of the PR and between 7dc92b4 and 836062f.

📒 Files selected for processing (1)
  • scripts/tests/calibnet_migration_regression_tests.sh (1 hunks)
🧰 Additional context used
🧠 Learnings (1)
📓 Common learnings
Learnt from: hanabi1224
PR: ChainSafe/forest#5930
File: build.rs:64-77
Timestamp: 2025-08-13T09:43:20.301Z
Learning: hanabi1224 prefers hard compile-time errors in build scripts rather than runtime safeguards or collision detection, believing it's better to fail fast and fix root causes of issues like malformed snapshot names.
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)
  • GitHub Check: Build forest binaries on Linux AMD64
  • GitHub Check: tests-release
  • GitHub Check: tests
🔇 Additional comments (1)
scripts/tests/calibnet_migration_regression_tests.sh (1)

10-10: LGTM: enabling peak RSS tracking for migration tests

Adding --track-peak-rss to MIGRATION_TEST is straightforward and aligns with the PR goal.


FOREST_PATH="forest"
MIGRATION_TEST="$FOREST_PATH --chain calibnet --encrypt-keystore false --halt-after-import --height=-200 --no-gc --import-snapshot"
MIGRATION_TEST="$FOREST_PATH --chain calibnet --encrypt-keystore false --track-peak-rss --halt-after-import --height=-200 --no-gc --import-snapshot"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Verification agent

🧩 Analysis chain

Verify CLI flag availability to avoid CI breakage

Please confirm the forest binary in CI exposes --track-peak-rss; otherwise set -e will fail the job on unknown-flag errors.

Run from repo root:


🏁 Script executed:

#!/bin/bash
# Check the CLI defines the flag (clap/arg names, help text, etc.)
rg -nC3 -i --type=rust 'track-peak-rss|peak.*rss|--track-peak-rss'

# If the binary is available in PATH in CI, also inspect help output (non-fatal if missing):
command -v forest >/dev/null && forest --help | rg -n -i 'track-peak-rss|peak.*rss' || true

Length of output: 1484


Ensure CI uses an updated forest binary with --track-peak-rss support
The code’s CLI derive in cli_shared/cli/mod.rs already defines the track_peak_rss flag (kebab-case --track-peak-rss), but the forest binary on PATH (used in CI) does not advertise that flag in its --help output. Update your CI setup to point $FOREST_PATH at a build of forest that includes this flag (or add a version check) to prevent unknown-flag errors and pipeline failures.

🤖 Prompt for AI Agents
In scripts/tests/calibnet_migration_regression_tests.sh around line 10, the CI
invokes $FOREST_PATH with the --track-peak-rss flag which the binary on PATH in
CI may not support; update CI to point $FOREST_PATH at a build of forest that
includes the cli_shared/cli/mod.rs track_peak_rss flag (rebuild forest from the
current repo/commit and export its path), or add a pre-check in CI that runs
`$FOREST_PATH --help` (or `--version`) to assert support for `--track-peak-rss`
and fail with a clear message if absent so the pipeline doesn’t pass an unknown
flag to the binary.

LesnyRumcajs
LesnyRumcajs previously approved these changes Sep 5, 2025
@hanabi1224 hanabi1224 enabled auto-merge September 5, 2025 10:00
@hanabi1224 hanabi1224 marked this pull request as draft September 5, 2025 10:04
auto-merge was automatically disabled September 5, 2025 10:04

Pull request was converted to draft

LesnyRumcajs
LesnyRumcajs previously approved these changes Sep 5, 2025
Copy link
Member

@LesnyRumcajs LesnyRumcajs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thank you!

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (4)
scripts/tests/calibnet_db_migration.sh (3)

45-45: Stream time -v to a log file for later parsing (keeps console output too).

This makes peak RSS easy to grep from CI artifacts.

-/usr/bin/time -v forest --chain calibnet --log-dir "$LOG_DIRECTORY" --halt-after-import --config "${CONFIG_FILE}"
+/usr/bin/time -v forest --chain calibnet --log-dir "$LOG_DIRECTORY" --halt-after-import --config "${CONFIG_FILE}" 2> >(tee -a "$LOG_DIRECTORY/peak_rss_migration_import.log" >&2)

48-48: Do the same for the long-running sync phase.

Captures the backgrounded run’s peak RSS reliably in logs.

-/usr/bin/time -v forest --chain calibnet --log-dir "$LOG_DIRECTORY" --save-token ./admin_token --config "${CONFIG_FILE}" &
+/usr/bin/time -v forest --chain calibnet --log-dir "$LOG_DIRECTORY" --save-token ./admin_token --config "${CONFIG_FILE}" 2> >(tee -a "$LOG_DIRECTORY/peak_rss_migration_sync.log" >&2) &

45-48: Optional: add a fallback if /usr/bin/time is unavailable.

CI is Linux, but a small guard improves portability.

Example outside this hunk:

time_wrap() { if command -v /usr/bin/time >/dev/null; then /usr/bin/time -v "$@"; else "$@"; fi; }
# then replace: /usr/bin/time -v forest ...  =>  time_wrap forest ...
scripts/tests/harness.sh (1)

91-91: Capture time -v output to logs and keep console visibility.

Eases automated parsing of peak RSS from CI logs.

-  /usr/bin/time -v $FOREST_PATH --chain calibnet --encrypt-keystore false --log-dir "$LOG_DIRECTORY" &
+  /usr/bin/time -v "$FOREST_PATH" --chain calibnet --encrypt-keystore false --log-dir "$LOG_DIRECTORY" 2> >(tee -a "$LOG_DIRECTORY/peak_rss_node_run.log" >&2) &
📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

💡 Knowledge Base configuration:

  • MCP integration is disabled by default for public repositories
  • Jira integration is disabled by default for public repositories
  • Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

📥 Commits

Reviewing files that changed from the base of the PR and between 836062f and ce414c2.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (11)
  • CHANGELOG.md (1 hunks)
  • Cargo.toml (0 hunks)
  • docs/docs/users/reference/cli.md (0 hunks)
  • scripts/tests/calibnet_db_migration.sh (1 hunks)
  • scripts/tests/calibnet_migration_regression_tests.sh (1 hunks)
  • scripts/tests/harness.sh (1 hunks)
  • src/cli_shared/cli/mod.rs (0 hunks)
  • src/daemon/mod.rs (1 hunks)
  • src/utils/mod.rs (0 hunks)
  • src/utils/monitoring/mem_tracker.rs (0 hunks)
  • src/utils/monitoring/mod.rs (0 hunks)
💤 Files with no reviewable changes (6)
  • src/cli_shared/cli/mod.rs
  • src/utils/mod.rs
  • docs/docs/users/reference/cli.md
  • src/utils/monitoring/mod.rs
  • Cargo.toml
  • src/utils/monitoring/mem_tracker.rs
✅ Files skipped from review due to trivial changes (1)
  • CHANGELOG.md
🚧 Files skipped from review as they are similar to previous changes (1)
  • scripts/tests/calibnet_migration_regression_tests.sh
🧰 Additional context used
🧠 Learnings (2)
📓 Common learnings
Learnt from: hanabi1224
PR: ChainSafe/forest#5930
File: build.rs:64-77
Timestamp: 2025-08-13T09:43:20.301Z
Learning: hanabi1224 prefers hard compile-time errors in build scripts rather than runtime safeguards or collision detection, believing it's better to fail fast and fix root causes of issues like malformed snapshot names.
📚 Learning: 2025-08-08T12:10:45.218Z
Learnt from: hanabi1224
PR: ChainSafe/forest#5867
File: src/ipld/util.rs:553-558
Timestamp: 2025-08-08T12:10:45.218Z
Learning: Forest project targets Rust stable >=1.89; features stabilized in 1.88 like let-chains are acceptable in this codebase.

Applied to files:

  • src/daemon/mod.rs
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)
  • GitHub Check: Build forest binaries on Linux AMD64
  • GitHub Check: tests
  • GitHub Check: tests-release
🔇 Additional comments (2)
src/daemon/mod.rs (2)

31-31: LGTM: imports updated after removing internal mem tracking.

Consolidated utils imports; no lingering references here.


31-31: Cleanup verified – no leftover feature flags or memory tracker symbols
Repo-wide search found no instances of --track-peak-rss, MemStatsTracker, or monitoring modules; /usr/bin/time -v wrappers exist only in test scripts as intended.

@hanabi1224 hanabi1224 enabled auto-merge September 5, 2025 11:35
@hanabi1224 hanabi1224 added this pull request to the merge queue Sep 9, 2025
Merged via the queue into main with commit 6248118 Sep 9, 2025
50 checks passed
@hanabi1224 hanabi1224 deleted the hm/track-peak-rss-for-migration-ci-tests branch September 9, 2025 11:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants