feat: benchmark existing runtime binary and select policy, presets #411

chungquantin · 2025-02-12T18:26:44Z

Scenarios

Context: The pop bench pallet command is run at the root directory of the parachain project

Stage 1: Locating and building a `runtime` binary

Scenario 1: If runtime_path (Path to the runtime, hard-coded to "runtime" folder) is not a Rust project, list all directories in the runtime_path (similar to pop-node which has multiple runtimes inside the runtime folder). After user select the runtime, we check if the runtime binary exists or not. If not, we build the runtime with runtime-benchmarks feature.

list all directories -> select runtime -> check if binary exists -> build runtime with `runtime-benchmarks` feature.

Scenario 2: If runtime_path is a Rust project, located inside the runtime folder, which means it is a runtime codebase (Note: There could be a chance that this is a workspace?). We will throw No runtime found if the workspace is not valid. We check if the runtime binary exists or not. If not, we build the runtime with runtime-benchmarks feature.

check if binary exists -> build runtime with `runtime-benchmarks` feature.

Scenario 3: If no runtime found, we throws No runtime found. (See diagram).

Stage 2: Select genesis builder policy and input the preset.

After the runtime binary is ready, we prompts user to select the genesis builder (There are two options currently: none and runtime, more details can be found here). As we only support the runtime, no chain spec is provided so other genesis builder policies are not valid.

none is selected: We do nothing.
runtime is selected: The runtime must be configured with sp-genesis-builder). We prompt the user to select the available genesis builder presets (presets will be registered on the runtime using BuildGenesisConfig runtime API).

By calling sc-chain-spec::GenesisConfigBuilderRuntimeCaller get_preset_names method, we list all genesis builder presets and expect the user to select one of them. If there is no builder preset found, the CLI throws No preset found (can be reproduced with pop-node).

If --runtime= is provided, we don't locate the runtime.
If --genesis-builder= is provided, we don't prompt user to select the genesis builder policy.
If --genesis-config-preset= is provided, we don't prompt user to input the genesis config preset.

How to test the PR?

Stage 1 | Scenario 1: `runtime` folder is the runtime.

Go to ./base-parachain or any parachain template.
Run pop bench pallet --pallet=pallet_timestamp --extrinsic= (No runtime provided)

Stage 1 | Scenario 2: `runtime` contains multiple runtime folders.

Go to ./pop-node or any parachain that has multiple runtimes
Run pop bench pallet --pallet=pallet_timestamp --extrinsic= (No runtime provided)

Stage 2

`genesis-builder = runtime` with available presets.

Go to ./base-parachain or any parachain template.
Run pop bench pallet --pallet=pallet_timestamp --extrinsic= (No runtime provided)
You will be prompted to select the preset.

`genesis-builder = runtime` with unavailable presets.

Go to ./pop-node or any parachain template.
Run pop bench pallet --pallet=pallet_timestamp --extrinsic= (No runtime provided)
Err thrown because there is no available genesis builder presets.

Changelog

Select runtime, genesis builder policy and genesis config preset.
Add new method build_project to build any Rust projects with features and target, required for the tests of this PR. The method is shared with build_parachain.
Add new method build_binary_path to locate the binary path of the provided Rust project. The method is shared with binary_path method.
Implement new feature to automatically locate existing <runtime-name>.wasm to benchmark. The path of the runtime WASM blob follows:

./target/<PROFILE>/wbuild/<name-of-runtime>/<name_of_runtime>.wasm

Add unit tests for new methods. To test the build_project method for the runtime, we build the mock project with wasm32-unknown-unknown as target and locates the .wasm file.

[sc-2931]

codecov · 2025-02-12T18:52:31Z

Codecov Report

Attention: Patch coverage is 75.90674% with 93 lines in your changes missing coverage. Please review.

Please upload report for BASE (chungquantin/feat-pop_bench@5a33cb5). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
crates/pop-cli/src/commands/bench/mod.rs	75.90%	30 Missing and 30 partials ⚠️
crates/pop-parachains/src/build.rs	75.55%	4 Missing and 18 partials ⚠️
crates/pop-parachains/src/bench.rs	76.59%	1 Missing and 10 partials ⚠️

@@                      Coverage Diff                       @@
##             chungquantin/feat-pop_bench     #411   +/-   ##
==============================================================
  Coverage                               ?   75.35%           
==============================================================
  Files                                  ?       67           
  Lines                                  ?    14399           
  Branches                               ?    14399           
==============================================================
  Hits                                   ?    10850           
  Misses                                 ?     2144           
  Partials                               ?     1405

Files with missing lines	Coverage Δ
crates/pop-parachains/src/bench.rs	`78.00% <76.59%> (ø)`
crates/pop-parachains/src/build.rs	`86.53% <75.55%> (ø)`
crates/pop-cli/src/commands/bench/mod.rs	`76.33% <75.90%> (ø)`

AlexD10S

The code looks very clean and well-structured, great job!

However, I encountered a couple of issues during manual testing:

If the node has already been built without the --runtime-benchmarks feature, it fails to build again because it detects the existing runtime. It throws an error (from benchmarking-cli), yet still displays a success message:

⚙  Benchmarking and generating weight file....
│  
[2025-02-15T07:41:13Z INFO  pallet_collator_selection::pallet] assembling new collators for new session 0 at #0
[2025-02-15T07:41:13Z INFO  pallet_collator_selection::pallet] assembling new collators for new session 1 at #0
[2025-02-15T07:41:13Z INFO  polkadot_sdk_frame::benchmark::pallet] Loading WASM from file
└  Failed to run benchmarking: Invalid input: Could not call runtime API to Did not find the benchmarking metadata. This could mean that you either did not build the node correctly with the `--features runtime-benchmarks` flag, or the chain spec that you are using was not created by a node that was compiled with the flag: Other: Exported method Benchmark_benchmark_metadata is not found
└  Benchmark completed successfully!

Probably we can't verify whether the runtime has been build with --runtime-benchmarks at this point right? It would be ideal if we could, but if not, we should at least prevent the misleading success message. I can see here, it shows a message with the error but continue the execution: https://github.com/r0gue-io/pop-cli/blob/chugnquantin/feat-build_runtime/crates/pop-cli/src/commands/bench/mod.rs#L68

The second issue relates to locating the WASM file, see my comment below.

Out of scope for this PR, but once the benchmarking logic is merged into main, we could consider adding a flag in pop build to handle building with the --runtime-benchmarks feature.

crates/pop-cli/src/commands/mod.rs

crates/pop-parachains/src/build.rs

crates/pop-cli/src/commands/bench/mod.rs

chungquantin added 11 commits February 10, 2025 15:51

feat: add bench subcommand

051d5ed

Merge branch 'r0gue-io:main' into chungquantin/feat-pop_bench

23ba99f

feat: integrates frame-benchmarking-cli

5987931

refactor: PalletCmd run with spec

0d5f5ef

feat: set RUST_LOG=info for benchmarking display

f6a8f06

feat: add CLI messages and tests

a3139e4

feat: add benchmark runtime wasm test

45c6ade

chore: fix comment

53258ac

fix: operator cannot be applied to type

8f82fde

chore: display error and refactor test file

fa2db46

feat: auto detect wasm blob and build runtime

907696a

chungquantin added 5 commits February 13, 2025 12:15

feat: build runtime wasm blob

7011eab

refactor: build binary path method

28c637a

fix: runtime path test

dc63358

refactor: feature gating for benchmarking feature

6f17808

chore: remove unused argument

557cdaa

chungquantin changed the title ~~feat: auto detect wasm blob and build runtime~~ feat: benchmark existing <runtime-name>.wasm or trigger build if needed Feb 13, 2025

chungquantin requested a review from AlexD10S February 13, 2025 06:36

chungquantin added the ready-for-final-review The PR is ready for final review label Feb 13, 2025

chungquantin self-assigned this Feb 13, 2025

chungquantin added 2 commits February 13, 2025 13:39

fix: comment

2306420

chore: rename bench file

667e997

chungquantin mentioned this pull request Feb 13, 2025

feat: pop bench pallet logic implementation #407

Merged

6 tasks

AlexD10S requested changes Feb 15, 2025

View reviewed changes

crates/pop-cli/src/commands/mod.rs Show resolved Hide resolved

crates/pop-parachains/src/build.rs Show resolved Hide resolved

crates/pop-cli/src/commands/bench/mod.rs Outdated Show resolved Hide resolved

chungquantin added 5 commits February 17, 2025 12:03

chore: rename bench file

646d6f5

feat: auto detect wasm blob and build runtime

5863fd3

feat: build runtime wasm blob

b09800e

refactor: build binary path method

299b60e

feat: list and select runtimes

f879739

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: benchmark existing runtime binary and select policy, presets #411

feat: benchmark existing runtime binary and select policy, presets #411

Uh oh!

chungquantin commented Feb 12, 2025 •

edited

Loading

Uh oh!

codecov bot commented Feb 12, 2025 •

edited

Loading

Uh oh!

AlexD10S left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

feat: benchmark existing runtime binary and select policy, presets #411

feat: benchmark existing runtime binary and select policy, presets #411

Uh oh!

Conversation

chungquantin commented Feb 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Scenarios

Stage 1: Locating and building a runtime binary

Stage 2: Select genesis builder policy and input the preset.

How to test the PR?

Stage 1 | Scenario 1: runtime folder is the runtime.

Stage 1 | Scenario 2: runtime contains multiple runtime folders.

Stage 2

genesis-builder = runtime with available presets.

genesis-builder = runtime with unavailable presets.

Changelog

Uh oh!

codecov bot commented Feb 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

AlexD10S left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

chungquantin commented Feb 12, 2025 •

edited

Loading

Stage 1: Locating and building a `runtime` binary

Stage 1 | Scenario 1: `runtime` folder is the runtime.

Stage 1 | Scenario 2: `runtime` contains multiple runtime folders.

`genesis-builder = runtime` with available presets.

`genesis-builder = runtime` with unavailable presets.

codecov bot commented Feb 12, 2025 •

edited

Loading