Updated ASV benchmark to include all P0 envs#461
Conversation
Signed-off-by: Kenny Vilella <kvilella@nvidia.com>
Signed-off-by: Kenny Vilella <kvilella@nvidia.com>
|
Note Currently processing new changes in this PR. This may take a few minutes, please wait... 📒 Files selected for processing (3)
📝 WalkthroughWalkthroughSix new benchmark files are introduced under Additionally, corresponding KPI benchmark files are added under Changes
Sequence Diagram(s)sequenceDiagram
participant ASV as ASV Benchmark Framework
participant Warp as Warp Library
participant Example as Example Simulation (Newton)
participant Device as Device (CUDA/CPU)
ASV->>Warp: wp.init()
ASV->>Example: Initialize Example(robot, num_envs, ...)
Note right of Example: For simulation benchmarks
loop For each simulation frame (e.g., 50 or 100)
ASV->>Example: step()
end
ASV->>Device: synchronize()
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~15 minutes Possibly related PRs
✨ Finishing Touches
🧪 Generate unit tests
🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
Documentation and Community
|
|
It looks like that the time_load benchmarks are taken a lot of time. |
|
As I hinted at over Slack, please don't modify the files I have in |
There was a problem hiding this comment.
Actionable comments posted: 2
📜 Review details
Configuration used: .coderabbit.yml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (5)
asv/benchmarks/examples/example_ant.py(1 hunks)asv/benchmarks/examples/example_cartpole.py(5 hunks)asv/benchmarks/examples/example_g1.py(1 hunks)asv/benchmarks/examples/example_h1.py(1 hunks)asv/benchmarks/examples/example_humanoid.py(3 hunks)
🧠 Learnings (2)
asv/benchmarks/examples/example_cartpole.py (1)
Learnt from: Kenny-Vilella
PR: #398
File: newton/examples/example_mujoco.py:352-352
Timestamp: 2025-07-14T03:57:29.670Z
Learning: The use_mujoco option in newton/examples/example_mujoco.py is currently unsupported and causes crashes. The code automatically disables this option with a warning message when users attempt to enable it. This is intentionally kept as a placeholder for future implementation.
asv/benchmarks/examples/example_humanoid.py (1)
Learnt from: Kenny-Vilella
PR: #398
File: newton/examples/example_mujoco.py:352-352
Timestamp: 2025-07-14T03:57:29.670Z
Learning: The use_mujoco option in newton/examples/example_mujoco.py is currently unsupported and causes crashes. The code automatically disables this option with a warning message when users attempt to enable it. This is intentionally kept as a placeholder for future implementation.
🧰 Additional context used
🧠 Learnings (2)
asv/benchmarks/examples/example_cartpole.py (1)
Learnt from: Kenny-Vilella
PR: #398
File: newton/examples/example_mujoco.py:352-352
Timestamp: 2025-07-14T03:57:29.670Z
Learning: The use_mujoco option in newton/examples/example_mujoco.py is currently unsupported and causes crashes. The code automatically disables this option with a warning message when users attempt to enable it. This is intentionally kept as a placeholder for future implementation.
asv/benchmarks/examples/example_humanoid.py (1)
Learnt from: Kenny-Vilella
PR: #398
File: newton/examples/example_mujoco.py:352-352
Timestamp: 2025-07-14T03:57:29.670Z
Learning: The use_mujoco option in newton/examples/example_mujoco.py is currently unsupported and causes crashes. The code automatically disables this option with a warning message when users attempt to enable it. This is intentionally kept as a placeholder for future implementation.
⏰ Context from checks skipped due to timeout of 900000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)
- GitHub Check: Run GPU Unit Tests on AWS EC2 (Pull Request)
- GitHub Check: run-asv-benchmarks
🔇 Additional comments (15)
asv/benchmarks/examples/example_humanoid.py (4)
22-22: LGTM: Clean migration to unified module approach.The import change from robot-specific module to the unified
example_mujocomodule with explicit robot parameter is a good architectural improvement for maintainability.
25-36: Well-implemented initialization benchmark.The
InitializeModelclass correctly:
- Uses parameterized environment counts for comprehensive testing
- Disables CUDA graph to exclude kernel compilation overhead from timing measurements
- Properly initializes Warp in setup method
55-64: Proper subprocess configuration for load benchmarking.The command line arguments correctly specify the robot type and required parameters for consistent benchmarking across different robot models.
77-86: Comprehensive simulation benchmark setup.The configuration includes all necessary parameters for realistic simulation benchmarking:
- Increased frames to 200 for better measurement accuracy
- Randomized actuation for varied workload testing
- CUDA graph enabled for performance optimization
asv/benchmarks/examples/example_h1.py (2)
25-36: Consistent benchmark implementation.The
InitializeModelclass follows the established pattern correctly with appropriate parameter ranges and CUDA graph disabled to exclude compilation overhead.
52-64: Proper command line argument structure.The subprocess command correctly specifies the H1 robot and all necessary parameters for load benchmarking.
asv/benchmarks/examples/example_g1.py (3)
25-36: Excellent consistency with established benchmark pattern.The implementation perfectly matches the pattern established in other robot benchmark files, ensuring consistent measurement methodology across different robot models.
52-64: Correct subprocess configuration for G1 robot.The command line arguments properly specify the G1 robot while maintaining consistency with other benchmark files.
76-87: Well-configured simulation benchmark.The setup includes all necessary parameters for comprehensive performance testing with the G1 robot model.
asv/benchmarks/examples/example_ant.py (3)
25-36: Consistent and well-structured benchmark implementation.The
InitializeModelclass correctly follows the established pattern with appropriate environment count parameters and CUDA graph disabled for accurate timing measurements.
48-69: Proper load benchmark configuration.The
MuJoCoSolverLoadclass correctly implements the subprocess approach with appropriate cache clearing and CUDA device validation.
76-87: Comprehensive simulation benchmark setup.The configuration includes all necessary parameters for realistic ant robot simulation benchmarking with proper Warp initialization and device synchronization.
asv/benchmarks/examples/example_cartpole.py (3)
22-22: LGTM: Successful migration to unified module approach.The import change to
example_mujocois consistent with the overall architectural improvement across all benchmark files.
66-71: Proper subprocess command structure.The command line arguments correctly specify the cartpole robot and required parameters for consistent benchmarking.
84-98: Good benchmark configuration updates.The changes include:
- Reduced repeat count to 5 (consistent with other files)
- Added
wp.init()call for proper initialization- Comprehensive parameter setup with randomization and CUDA graph enabled
Yes, for the benchmarks in |
Ah my apologies, will change it.
Got it ! |
Signed-off-by: Kenny Vilella <kvilella@nvidia.com>
Signed-off-by: Kenny Vilella <kvilella@nvidia.com>
Signed-off-by: Kenny Vilella <kvilella@nvidia.com>
There was a problem hiding this comment.
Actionable comments posted: 2
📜 Review details
Configuration used: .coderabbit.yml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (5)
asv/benchmarks/envs/example_ant.py(1 hunks)asv/benchmarks/envs/example_cartpole.py(1 hunks)asv/benchmarks/envs/example_g1.py(1 hunks)asv/benchmarks/envs/example_h1.py(1 hunks)asv/benchmarks/envs/example_humanoid.py(1 hunks)
🧬 Code Graph Analysis (5)
asv/benchmarks/envs/example_cartpole.py (4)
asv/benchmarks/envs/example_h1.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)asv/benchmarks/envs/example_g1.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)asv/benchmarks/envs/example_humanoid.py (6)
InitializeModel(23-33)setup(28-29)setup(40-50)time_initialize_model(31-33)MuJoCoSolverSimulate(36-56)time_simulate(53-56)asv/benchmarks/envs/example_ant.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)
asv/benchmarks/envs/example_humanoid.py (4)
asv/benchmarks/envs/example_h1.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)asv/benchmarks/envs/example_cartpole.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)asv/benchmarks/envs/example_g1.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)asv/benchmarks/envs/example_ant.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)
asv/benchmarks/envs/example_g1.py (4)
asv/benchmarks/envs/example_h1.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)asv/benchmarks/envs/example_cartpole.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)asv/benchmarks/envs/example_humanoid.py (6)
InitializeModel(23-33)setup(28-29)setup(40-50)time_initialize_model(31-33)MuJoCoSolverSimulate(36-56)time_simulate(53-56)asv/benchmarks/envs/example_ant.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)
asv/benchmarks/envs/example_h1.py (4)
asv/benchmarks/envs/example_cartpole.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)asv/benchmarks/envs/example_g1.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)asv/benchmarks/envs/example_humanoid.py (6)
InitializeModel(23-33)setup(28-29)setup(40-50)time_initialize_model(31-33)MuJoCoSolverSimulate(36-56)time_simulate(53-56)asv/benchmarks/envs/example_ant.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)
asv/benchmarks/envs/example_ant.py (4)
asv/benchmarks/envs/example_h1.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)asv/benchmarks/envs/example_cartpole.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)asv/benchmarks/envs/example_g1.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)asv/benchmarks/envs/example_humanoid.py (6)
InitializeModel(23-33)setup(28-29)setup(40-50)time_initialize_model(31-33)MuJoCoSolverSimulate(36-56)time_simulate(53-56)
🧰 Additional context used
🧬 Code Graph Analysis (5)
asv/benchmarks/envs/example_cartpole.py (4)
asv/benchmarks/envs/example_h1.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)asv/benchmarks/envs/example_g1.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)asv/benchmarks/envs/example_humanoid.py (6)
InitializeModel(23-33)setup(28-29)setup(40-50)time_initialize_model(31-33)MuJoCoSolverSimulate(36-56)time_simulate(53-56)asv/benchmarks/envs/example_ant.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)
asv/benchmarks/envs/example_humanoid.py (4)
asv/benchmarks/envs/example_h1.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)asv/benchmarks/envs/example_cartpole.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)asv/benchmarks/envs/example_g1.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)asv/benchmarks/envs/example_ant.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)
asv/benchmarks/envs/example_g1.py (4)
asv/benchmarks/envs/example_h1.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)asv/benchmarks/envs/example_cartpole.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)asv/benchmarks/envs/example_humanoid.py (6)
InitializeModel(23-33)setup(28-29)setup(40-50)time_initialize_model(31-33)MuJoCoSolverSimulate(36-56)time_simulate(53-56)asv/benchmarks/envs/example_ant.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)
asv/benchmarks/envs/example_h1.py (4)
asv/benchmarks/envs/example_cartpole.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)asv/benchmarks/envs/example_g1.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)asv/benchmarks/envs/example_humanoid.py (6)
InitializeModel(23-33)setup(28-29)setup(40-50)time_initialize_model(31-33)MuJoCoSolverSimulate(36-56)time_simulate(53-56)asv/benchmarks/envs/example_ant.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)
asv/benchmarks/envs/example_ant.py (4)
asv/benchmarks/envs/example_h1.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)asv/benchmarks/envs/example_cartpole.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)asv/benchmarks/envs/example_g1.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)asv/benchmarks/envs/example_humanoid.py (6)
InitializeModel(23-33)setup(28-29)setup(40-50)time_initialize_model(31-33)MuJoCoSolverSimulate(36-56)time_simulate(53-56)
🔇 Additional comments (5)
asv/benchmarks/envs/example_cartpole.py (1)
40-42: Missing wp.init() call in setup method.The
MuJoCoSolverSimulateclass is missing thewp.init()call in its setup method, which is present in other environment benchmarks (example_ant.py,example_g1.py,example_h1.py). This inconsistency could lead to initialization issues.def setup(self): + wp.init() self.num_frames = 200Likely an incorrect or invalid review comment.
asv/benchmarks/envs/example_g1.py (1)
23-57: LGTM! Benchmark implementation follows established patterns.The benchmark classes are correctly implemented with proper Warp initialization, consistent parameter configuration, and appropriate CUDA handling. The implementation aligns well with the other environment benchmarks in the suite.
asv/benchmarks/envs/example_humanoid.py (1)
31-33: Benchmark implementation is well-structured.The initialization benchmark correctly uses
use_cuda_graph=Falseto exclude kernel compilation overhead, and the simulation benchmark properly handles CUDA device availability checks and synchronization.asv/benchmarks/envs/example_h1.py (1)
23-57: LGTM! Benchmark implementation is consistent and well-structured.The benchmark classes correctly follow the established patterns with proper Warp initialization, appropriate parameter configuration, and correct CUDA device handling. The implementation aligns perfectly with the benchmarking framework requirements.
asv/benchmarks/envs/example_ant.py (1)
23-57: LGTM! Benchmark implementation follows established patterns correctly.The benchmark classes are properly implemented with consistent Warp initialization, appropriate parameter settings, and correct CUDA device handling. The implementation aligns well with the benchmarking framework standards.
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
asv/benchmarks/envs/example_cartpole.py (1)
31-33: Inconsistent initialization approach compared to other benchmark files.This file uses
wp.ScopedDevice("cpu")and omits theuse_cuda_graph=Falseparameter, while all other benchmark files (example_ant.py, example_g1.py, example_h1.py, example_humanoid.py) explicitly setuse_cuda_graph=Falsewithout the scoped device context. Consider aligning with the established pattern for consistency.def time_initialize_model(self, num_envs): - with wp.ScopedDevice("cpu"): - _example = Example(stage_path=None, robot="cartpole", headless=True, num_envs=num_envs) + # use_cuda_graph is False to exclude kernel compilation + _example = Example(stage_path=None, robot="cartpole", headless=True, num_envs=num_envs, use_cuda_graph=False)
📜 Review details
Configuration used: .coderabbit.yml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (5)
asv/benchmarks/envs/example_ant.py(1 hunks)asv/benchmarks/envs/example_cartpole.py(1 hunks)asv/benchmarks/envs/example_g1.py(1 hunks)asv/benchmarks/envs/example_h1.py(1 hunks)asv/benchmarks/envs/example_humanoid.py(1 hunks)
🧬 Code Graph Analysis (4)
asv/benchmarks/envs/example_humanoid.py (4)
asv/benchmarks/envs/example_cartpole.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)asv/benchmarks/envs/example_ant.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)asv/benchmarks/envs/example_g1.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)asv/benchmarks/envs/example_h1.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)
asv/benchmarks/envs/example_ant.py (4)
asv/benchmarks/envs/example_cartpole.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)asv/benchmarks/envs/example_g1.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)asv/benchmarks/envs/example_h1.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)asv/benchmarks/envs/example_humanoid.py (6)
InitializeModel(23-33)setup(28-29)setup(40-50)time_initialize_model(31-33)MuJoCoSolverSimulate(36-56)time_simulate(53-56)
asv/benchmarks/envs/example_cartpole.py (4)
asv/benchmarks/envs/example_ant.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)asv/benchmarks/envs/example_g1.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)asv/benchmarks/envs/example_h1.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)asv/benchmarks/envs/example_humanoid.py (6)
InitializeModel(23-33)setup(28-29)setup(40-50)time_initialize_model(31-33)MuJoCoSolverSimulate(36-56)time_simulate(53-56)
asv/benchmarks/envs/example_h1.py (4)
asv/benchmarks/envs/example_cartpole.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)asv/benchmarks/envs/example_ant.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)asv/benchmarks/envs/example_g1.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)asv/benchmarks/envs/example_humanoid.py (6)
InitializeModel(23-33)setup(28-29)setup(40-50)time_initialize_model(31-33)MuJoCoSolverSimulate(36-56)time_simulate(53-56)
🧰 Additional context used
🧬 Code Graph Analysis (4)
asv/benchmarks/envs/example_humanoid.py (4)
asv/benchmarks/envs/example_cartpole.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)asv/benchmarks/envs/example_ant.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)asv/benchmarks/envs/example_g1.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)asv/benchmarks/envs/example_h1.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)
asv/benchmarks/envs/example_ant.py (4)
asv/benchmarks/envs/example_cartpole.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)asv/benchmarks/envs/example_g1.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)asv/benchmarks/envs/example_h1.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)asv/benchmarks/envs/example_humanoid.py (6)
InitializeModel(23-33)setup(28-29)setup(40-50)time_initialize_model(31-33)MuJoCoSolverSimulate(36-56)time_simulate(53-56)
asv/benchmarks/envs/example_cartpole.py (4)
asv/benchmarks/envs/example_ant.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)asv/benchmarks/envs/example_g1.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)asv/benchmarks/envs/example_h1.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)asv/benchmarks/envs/example_humanoid.py (6)
InitializeModel(23-33)setup(28-29)setup(40-50)time_initialize_model(31-33)MuJoCoSolverSimulate(36-56)time_simulate(53-56)
asv/benchmarks/envs/example_h1.py (4)
asv/benchmarks/envs/example_cartpole.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)asv/benchmarks/envs/example_ant.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)asv/benchmarks/envs/example_g1.py (6)
InitializeModel(23-33)setup(28-29)setup(40-51)time_initialize_model(31-33)MuJoCoSolverSimulate(36-57)time_simulate(54-57)asv/benchmarks/envs/example_humanoid.py (6)
InitializeModel(23-33)setup(28-29)setup(40-50)time_initialize_model(31-33)MuJoCoSolverSimulate(36-56)time_simulate(53-56)
🔇 Additional comments (6)
asv/benchmarks/envs/example_humanoid.py (2)
23-34: Consider adding consistency improvements for initialization benchmarking.The initialization benchmark implementation is consistent with the pattern, though other files show minor variations in approach (e.g.,
example_cartpole.pyuseswp.ScopedDevice("cpu")). The current implementation correctly excludes kernel compilation time by settinguse_cuda_graph=False.
52-56: Simulation benchmark implementation follows established pattern.The simulation timing methodology is correctly implemented with proper device synchronization and CUDA availability checking.
asv/benchmarks/envs/example_ant.py (1)
23-57: Benchmark implementation follows established pattern correctly.The implementation is consistent with other benchmark files in the PR, properly handling Warp initialization, CUDA graph usage, and device synchronization. The ant robot benchmarking setup is correctly configured.
asv/benchmarks/envs/example_cartpole.py (1)
36-57: Simulation benchmark implementation is correct.The
MuJoCoSolverSimulateclass follows the established pattern correctly with proper Warp initialization, CUDA graph usage, and device synchronization.asv/benchmarks/envs/example_g1.py (1)
23-57: Well-implemented benchmark following established pattern.The G1 robot benchmark implementation correctly follows the established pattern with proper Warp initialization, appropriate CUDA graph usage settings, and clear explanatory comments. The implementation is consistent with the other benchmark files.
asv/benchmarks/envs/example_h1.py (1)
23-57: Correct benchmark implementation following established conventions.The H1 robot benchmark implementation properly follows the established pattern with appropriate Warp initialization, correct CUDA graph usage configuration, and proper device synchronization. The code is well-structured and consistent.
|
(just taking notes) currently on my computer, I got these times for how long it takes to run the new benchmarks: |
|
Before I forget to provide an update @Kenny-Vilella, this is where I have things. I am focusing on I changed the benchmark to be: class MuJoCoSolverSimulate:
params = [4, 8, 16]
param_names = ["num_envs"]
repeat = 3
number = 1
def setup(self, num_envs):
wp.init()
self.num_frames = 50
self.example = Example(
stage_path=None,
robot="g1",
randomize=True,
headless=True,
actuation="random",
num_envs=num_envs,
use_cuda_graph=True,
)
@skip_benchmark_if(wp.get_cuda_device_count() == 0)
def track_simulate(self, num_envs):
steps = self.num_frames * self.example.sim_substeps * self.example.num_envs
start_time = time.time()
for _ in range(self.num_frames):
self.example.step()
wp.synchronize_device()
end_time = time.time()
return (end_time - start_time) * 1000 / steps
track_simulate.unit = "ms/env-step"And I get reasonably stable results on my computer like: For the |
|
@shi-eric Thanks for taking a look. |
|
Some information about example_g1 with 8192 envs using the script you provided earlier on horde (L40). |
Okay, so it seems that we might not use 8192 envs initially for the benchmark. Instead we want to use a lower number that still exhibits the qualitative trends we want to use to measure progress so developers don't need to wait so long. |
|
It looks like that a nightly (or weekly?) run to follow KPI progress + a lighter run for PRs would be a good solution. |
Okay, how about adding a The The current This shouldn't take a long time, let me know if you want to break this up into two pull requests (one set for KPI runs and one set for CI/CD runs) and if I can help with either. |
|
It makes sense, but we will probably needs a I can start doing it. Also, I noticed that there are |
Signed-off-by: Kenny Vilella <kvilella@nvidia.com>
Signed-off-by: Kenny Vilella <kvilella@nvidia.com>
|
OK so I created two folders:
All of them follows the example you have given earlier in this thread. I mainly changed the value of |
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
asv/benchmarks/KPI/example_humanoid.py (1)
48-48: Consider standardizing frame count across KPI benchmarks.This benchmark uses 50 frames while other KPI benchmarks (ant, cartpole, g1, h1) use 100 frames. Consider standardizing to ensure consistent measurement methodology across robot types.
- self.num_frames = 50 + self.num_frames = 100
📜 Review details
Configuration used: .coderabbit.yml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (10)
asv/benchmarks/KPI/example_ant.py(1 hunks)asv/benchmarks/KPI/example_cartpole.py(1 hunks)asv/benchmarks/KPI/example_g1.py(1 hunks)asv/benchmarks/KPI/example_h1.py(1 hunks)asv/benchmarks/KPI/example_humanoid.py(1 hunks)asv/benchmarks/envs/example_ant.py(1 hunks)asv/benchmarks/envs/example_cartpole.py(1 hunks)asv/benchmarks/envs/example_g1.py(1 hunks)asv/benchmarks/envs/example_h1.py(1 hunks)asv/benchmarks/envs/example_humanoid.py(1 hunks)
🚧 Files skipped from review as they are similar to previous changes (5)
- asv/benchmarks/envs/example_humanoid.py
- asv/benchmarks/envs/example_g1.py
- asv/benchmarks/envs/example_cartpole.py
- asv/benchmarks/envs/example_h1.py
- asv/benchmarks/envs/example_ant.py
🧰 Additional context used
🧠 Learnings (5)
asv/benchmarks/KPI/example_cartpole.py (1)
Learnt from: shi-eric
PR: #461
File: asv/benchmarks/envs/example_humanoid.py:40-41
Timestamp: 2025-07-23T14:36:42.135Z
Learning: In Warp benchmarks, explicit wp.init() calls are not needed in most circumstances since the first Warp API call that requires initialization will automatically call wp.init(). Explicit wp.init() in setup() methods is helpful when the ASV benchmark is measuring a Warp API call, as wp.init() has non-trivial overhead that should be excluded from the benchmark timing.
asv/benchmarks/KPI/example_ant.py (1)
Learnt from: shi-eric
PR: #461
File: asv/benchmarks/envs/example_humanoid.py:40-41
Timestamp: 2025-07-23T14:36:42.135Z
Learning: In Warp benchmarks, explicit wp.init() calls are not needed in most circumstances since the first Warp API call that requires initialization will automatically call wp.init(). Explicit wp.init() in setup() methods is helpful when the ASV benchmark is measuring a Warp API call, as wp.init() has non-trivial overhead that should be excluded from the benchmark timing.
asv/benchmarks/KPI/example_g1.py (1)
Learnt from: shi-eric
PR: #461
File: asv/benchmarks/envs/example_humanoid.py:40-41
Timestamp: 2025-07-23T14:36:42.135Z
Learning: In Warp benchmarks, explicit wp.init() calls are not needed in most circumstances since the first Warp API call that requires initialization will automatically call wp.init(). Explicit wp.init() in setup() methods is helpful when the ASV benchmark is measuring a Warp API call, as wp.init() has non-trivial overhead that should be excluded from the benchmark timing.
asv/benchmarks/KPI/example_humanoid.py (1)
Learnt from: shi-eric
PR: #461
File: asv/benchmarks/envs/example_humanoid.py:40-41
Timestamp: 2025-07-23T14:36:42.135Z
Learning: In Warp benchmarks, explicit wp.init() calls are not needed in most circumstances since the first Warp API call that requires initialization will automatically call wp.init(). Explicit wp.init() in setup() methods is helpful when the ASV benchmark is measuring a Warp API call, as wp.init() has non-trivial overhead that should be excluded from the benchmark timing.
asv/benchmarks/KPI/example_h1.py (1)
Learnt from: shi-eric
PR: #461
File: asv/benchmarks/envs/example_humanoid.py:40-41
Timestamp: 2025-07-23T14:36:42.135Z
Learning: In Warp benchmarks, explicit wp.init() calls are not needed in most circumstances since the first Warp API call that requires initialization will automatically call wp.init(). Explicit wp.init() in setup() methods is helpful when the ASV benchmark is measuring a Warp API call, as wp.init() has non-trivial overhead that should be excluded from the benchmark timing.
🧬 Code Graph Analysis (4)
asv/benchmarks/KPI/example_cartpole.py (3)
asv/benchmarks/KPI/example_humanoid.py (6)
InitializeModelKPI(25-38)setup(33-34)setup(47-57)time_initialize_model(36-38)MuJoCoSolverSimulateKPI(41-70)track_simulate(60-68)asv/benchmarks/KPI/example_h1.py (6)
InitializeModelKPI(25-38)setup(33-34)setup(47-58)time_initialize_model(36-38)MuJoCoSolverSimulateKPI(41-71)track_simulate(61-69)asv/benchmarks/envs/example_cartpole.py (4)
setup(33-34)setup(50-61)time_initialize_model(36-38)track_simulate(64-72)
asv/benchmarks/KPI/example_ant.py (3)
asv/benchmarks/KPI/example_cartpole.py (6)
InitializeModelKPI(25-38)setup(33-34)setup(47-58)time_initialize_model(36-38)MuJoCoSolverSimulateKPI(41-71)track_simulate(61-69)asv/benchmarks/KPI/example_humanoid.py (6)
InitializeModelKPI(25-38)setup(33-34)setup(47-57)time_initialize_model(36-38)MuJoCoSolverSimulateKPI(41-70)track_simulate(60-68)asv/benchmarks/envs/example_ant.py (4)
setup(33-34)setup(50-61)time_initialize_model(36-38)track_simulate(64-72)
asv/benchmarks/KPI/example_g1.py (5)
asv/benchmarks/KPI/example_cartpole.py (6)
InitializeModelKPI(25-38)setup(33-34)setup(47-58)time_initialize_model(36-38)MuJoCoSolverSimulateKPI(41-71)track_simulate(61-69)asv/benchmarks/KPI/example_ant.py (6)
InitializeModelKPI(25-38)setup(33-34)setup(47-58)time_initialize_model(36-38)MuJoCoSolverSimulateKPI(41-71)track_simulate(61-69)asv/benchmarks/KPI/example_humanoid.py (6)
InitializeModelKPI(25-38)setup(33-34)setup(47-57)time_initialize_model(36-38)MuJoCoSolverSimulateKPI(41-70)track_simulate(60-68)asv/benchmarks/KPI/example_h1.py (6)
InitializeModelKPI(25-38)setup(33-34)setup(47-58)time_initialize_model(36-38)MuJoCoSolverSimulateKPI(41-71)track_simulate(61-69)asv/benchmarks/envs/example_g1.py (4)
setup(33-34)setup(50-61)time_initialize_model(36-38)track_simulate(64-72)
asv/benchmarks/KPI/example_h1.py (2)
asv/benchmarks/KPI/example_cartpole.py (6)
InitializeModelKPI(25-38)setup(33-34)setup(47-58)time_initialize_model(36-38)MuJoCoSolverSimulateKPI(41-71)track_simulate(61-69)asv/benchmarks/envs/example_h1.py (4)
setup(33-34)setup(50-61)time_initialize_model(36-38)track_simulate(64-72)
⏰ Context from checks skipped due to timeout of 900000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: Run GPU Unit Tests on AWS EC2 (Pull Request)
🔇 Additional comments (7)
asv/benchmarks/KPI/example_ant.py (2)
25-71: LGTM: Consistent KPI benchmark implementation.The benchmark classes follow the established pattern with appropriate ASV parameters, proper CUDA graph usage, and correct timing methodology. The initialization benchmark excludes kernel compilation overhead and the simulation benchmark includes proper device synchronization.
47-58: Add missing wp.init() call in setup method.The
setup()method is missing thewp.init()call, which is inconsistent with other KPI benchmark files. Based on the retrieved learning, explicitwp.init()calls in setup methods help exclude Warp initialization overhead from benchmark timing.def setup(self): + wp.init() self.num_frames = 100 self.example = Example(⛔ Skipped due to learnings
Learnt from: shi-eric PR: newton-physics/newton#461 File: asv/benchmarks/envs/example_humanoid.py:40-41 Timestamp: 2025-07-23T14:36:42.135Z Learning: In Warp benchmarks, explicit wp.init() calls are not needed in most circumstances since the first Warp API call that requires initialization will automatically call wp.init(). Explicit wp.init() in setup() methods is helpful when the ASV benchmark is measuring a Warp API call, as wp.init() has non-trivial overhead that should be excluded from the benchmark timing.asv/benchmarks/KPI/example_cartpole.py (2)
37-37: Consider consistency with other KPI benchmarks regarding CPU scoping.This file uses
wp.ScopedDevice("cpu")in the initialization benchmark, while other KPI benchmark files don't use this scoping. Verify if this CPU scoping is intentionally specific to cartpole or should be standardized across all KPI benchmarks.
41-71: LGTM: Proper simulation benchmark implementation.The simulation benchmark correctly initializes Warp, uses appropriate CUDA graph settings, includes device synchronization, and follows the established timing methodology.
asv/benchmarks/KPI/example_humanoid.py (1)
25-38: LGTM: Correct initialization benchmark implementation.The initialization benchmark properly initializes Warp in setup, uses
use_cuda_graph=Falseto exclude kernel compilation overhead, and follows the established pattern.asv/benchmarks/KPI/example_g1.py (1)
25-71: LGTM: Well-implemented KPI benchmark.This benchmark file correctly follows the established pattern with proper Warp initialization, appropriate CUDA graph usage, consistent frame count (100), and correct timing methodology including device synchronization.
asv/benchmarks/KPI/example_h1.py (1)
25-71: LGTM: Consistent and correct KPI benchmark implementation.This benchmark file properly implements the KPI pattern with correct Warp initialization, appropriate CUDA graph settings, standard frame count (100), and proper timing methodology with device synchronization.
|
Just following along - @Kenny-Vilella I think it makes sense to report these initialization findings/problematic benchmarks in #55 such that they can be looked at. |
Signed-off-by: Kenny Vilella <kvilella@nvidia.com>
Thanks! I did some restructuring today of the class G1:
params = [4096, 8192]
param_names = ["num_envs"]
num_frames = 50
robot = "g1"
timeout = 1200
@skip_benchmark_if(wp.get_cuda_device_count() == 0)
def track_simulate(self, num_envs):
samples = 4
builder = Example.create_model_builder(self.robot, num_envs, randomize=True, seed=123)
total_time = 0.0
for _iter in range(samples):
example = Example(
stage_path=None,
robot=self.robot,
randomize=True,
headless=True,
actuation="random",
num_envs=num_envs,
use_cuda_graph=True,
builder=builder,
)
wp.synchronize_device()
start_time = time.time()
for _ in range(self.num_frames):
example.step()
wp.synchronize_device()
total_time += time.time() - start_time
return total_time * 1000 / (self.num_frames * example.sim_substeps * num_envs * samples)
track_simulate.unit = "ms/env-step" |
|
I guess you can either create a new pull request with my commits or create a pull request to this branch. |
|
I think we can get better statistics by taking the median of all |
Median vs average seems to be a complicated choice. |
|
Merged (with changes) in #475 |
# Description <!-- Thank you for your interest in sending a pull request. Please make sure to check the contribution guidelines. Link: https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html --> Fixes a bug where the XR kit file settings were split by app.python kit settings which caused the AVP retargeting to perform incorrectly. The new XR settings have been shifted to not split the settings. Fixes # (issue) <!-- As a practice, it is recommended to open an issue to have discussions on the proposed pull request. This makes it easier for the community to keep track of what is being developed or added, and if a given feature is demanded by more than one party. --> ## Type of change <!-- As you go through the list, delete the ones that are not applicable. --> - Bug fix (non-breaking change which fixes an issue) ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [ ] I have added my name to the `CONTRIBUTORS.md` or my name already exists there <!-- As you go through the checklist above, you can mark something as done by putting an x character in it For example, - [x] I have done this task - [ ] I have not done this task -->
Description
InitializeModelbenchmark for the humanoidNewton Migration Guide
Please ensure the migration guide for warp.sim users is up-to-date with the changes made in this MR.
docs/migration.rstis up-to dateBefore your PR is "Ready for review"
newton/tests/test_examples.py)pre-commit run -aSummary by CodeRabbit