Updates docs for using nurec background in locomanipulation sdg by dengyuchenkit · Pull Request #5301 · isaac-sim/IsaacLab

dengyuchenkit · 2026-04-16T23:29:14Z

Updates docs for using nurec background in locomanipulation sdg

Type of change

Documentation update

Checklist

I have read and understood the contribution guidelines
I have run the pre-commit checks with ./isaaclab.sh --format
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
I have updated the changelog and the corresponding version in the extension's config/extension.toml file
I have added my name to the CONTRIBUTORS.md or my name already exists there

greptile-apps · 2026-04-16T23:32:10Z

Greptile Summary

This PR adds documentation for using a NuRec (Neural Volume Rendering) 3D Gaussian model USD asset as a background in the locomanipulation SDG pipeline, covering download, requirements, dataset generation with background, and video export steps.

Confidence Score: 4/5

Documentation-only PR; safe to merge after clarifying the dataset filename discrepancy and the unexplained downloaded file.

All findings are P2, but the dataset filename mismatch (annotated vs. generated) in the background command could cause a user following the documented flow to fail immediately, so it warrants author attention before merge.

docs/source/overview/imitation-learning/humanoids_imitation.rst — the "Add a background USD asset" section around lines 724–771.

Important Files Changed

Filename	Overview
docs/source/overview/imitation-learning/humanoids_imitation.rst	Adds a new "Add a background USD asset during data generation" section covering NuRec/3D Gaussian model USD backgrounds; has a dataset filename inconsistency and an unexplained downloaded file.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Generate manipulation dataset
generated_dataset_g1_locomanip.hdf5] --> B[Download NuRec USD asset
stage.usdz + occupancy_map.yaml]
    B --> C[Run generate_data.py
with --background_usd_path
+ --background_occupancy_yaml_file]
    C --> D[HDF5 dataset with
camera observations]
    D --> E[Convert to MP4
hdf5_to_mp4.py]
    style B fill:#ffe0b2
    style C fill:#e1f5fe

Comments Outside Diff (1)

docs/source/overview/imitation-learning/humanoids_imitation.rst, line 757-761 (link)

Dataset inconsistency vs. preceding navigation command

The background command passes dataset_annotated_g1_locomanip.hdf5 (the raw annotated dataset) while the earlier equivalent navigation command at line 566 uses generated_dataset_g1_locomanip.hdf5 (the full 1000-demo generated dataset). The note above this command (line 745) directs users to "Generate the manipulation dataset", which produces the generated dataset—not the annotated one. A user who follows that flow will have generated_dataset_g1_locomanip.hdf5 on disk, not dataset_annotated_g1_locomanip.hdf5, making the command fail or require an unexplained extra step. Either align the filename with what the prior section produces, or explicitly note why the annotated dataset is preferred here.

_{Reviews (1): Last reviewed commit: "Update nurec in locomanipulation sdg doc" | Re-trigger Greptile}

greptile-apps · 2026-04-16T23:32:17Z

+
+   hf download nvidia/PhysicalAI-Robotics-NuRec \
+      hand_hold-voyager-babyboom/stage.usdz \
+      hand_hold-voyager-babyboom/occupancy_map.png \
+      hand_hold-voyager-babyboom/occupancy_map.yaml \
+      --repo-type dataset \
+      --local-dir <PATH_TO_USD_ASSET>


occupancy_map.png downloaded but never used

The hf download command fetches occupancy_map.png alongside stage.usdz and occupancy_map.yaml, but occupancy_map.png is never referenced in any subsequent command or requirement bullet. Its purpose (e.g. a visual reference for the user, or input to a separate tool) is unexplained. Consider either removing it from the download command or adding a brief note describing when it is used.

isaaclab-review-bot

🤖 Isaac Lab Review Bot

Summary

This PR updates NuRec/3D Gaussian documentation for locomanipulation SDG and adds a PhysX manager change to call attach_stage() for CPU pipelines after warmup. The documentation changes are solid—typo fixes and improved clarity. However, the code change introduces a critical regression test failure that must be addressed.

Architecture Impact

The PhysxManager change affects CPU pipeline initialization. The added attach_stage() call for CPU will cause the existing regression test test_warmup_attach_stage_not_called_for_cpu to fail. This test explicitly asserts that attach_stage.call_count == 0 for CPU pipelines.

Implementation Verdict

Significant concerns — The code change will break an existing regression test. The test must either be updated (if the new behavior is correct) or the change must be reverted. Without this resolution, CI will fail.

Test Coverage

🔴 Critical: The existing regression test test_warmup_attach_stage_not_called_for_cpu (lines 1256-1300 in source/isaaclab_physx/test/assets/test_rigid_object.py) explicitly asserts that attach_stage() is never called for CPU pipelines. This PR adds an attach_stage() call for CPU (at line 681), which will cause this test to fail.

If calling attach_stage() AFTER MBP init is indeed safe and necessary for resolving kinematic bodies, the test should be updated to:

Allow attach_stage() to be called once (change assertion to call_count == 1)
Verify the call happens AFTER force_load_physics_from_usd() (the dangerous case is calling it BEFORE)
Update the test docstring to reflect the new understanding

CI Status

Only labeler check has run ✅. Full test suite has not run yet—this test failure will surface when it does.

Findings

🔴 Critical: source/isaaclab_physx/isaaclab_physx/physics/physx_manager.py:681 — Will break existing regression test
The new attach_stage() call for CPU will cause test_warmup_attach_stage_not_called_for_cpu to fail. The test was added in PR #4675 specifically to guard against this. Either:

The test needs to be updated if post-warmup attach_stage() is safe
The change needs to be reverted if it could cause MBP corruption
A new test should verify the ordering (attach_stage AFTER force_load, not BEFORE)

🟡 Warning: source/isaaclab_physx/isaaclab_physx/physics/physx_manager.py:678-680 — Missing context in comment
The comment says this is needed "so the tensor plugin can resolve kinematic bodies" but does not explain what was broken before or link to an issue. Add context about the specific bug/behavior this fixes (e.g., "Fixes #XXXX" or a brief description of the symptom).

🔵 Improvement: docs/source/overview/imitation-learning/humanoids_imitation.rst — Documentation quality is good
The rewritten NuRec section is clearer and more actionable than the original. The typo fixes are all correct (determimnistically → deterministically, IssacLab → IsaacLab, flexibtility → flexibility, make → mark).

isaaclab-review-bot · 2026-04-21T02:50:54Z

-        # Calling attach_stage() + force_load_physics_from_usd() together causes a
+        # For CPU, calling attach_stage() before force_load_physics_from_usd() causes a
        # double-initialization that corrupts the CPU broadphase (MBP) collision setup,
-        # causing objects to fall through surfaces non-deterministically.


🔴 Critical: This new attach_stage() call for CPU will cause the existing regression test test_warmup_attach_stage_not_called_for_cpu to fail.

The test at line 1296 in source/isaaclab_physx/test/assets/test_rigid_object.py explicitly asserts:

assert spy.attach_stage.call_count == 0, (...)

If calling attach_stage() AFTER MBP initialization is safe (unlike calling it BEFORE), the test needs to be updated to reflect this new understanding.

Suggested action: Update the regression test to verify the correct ordering rather than forbidding the call entirely.

jaybdub · 2026-04-23T02:30:36Z

LGTM

isaaclab-review-bot

🤖 Isaac Lab Review Bot

Summary

Previous concerns have been addressed. The physx_manager.py code change has been removed from this PR — it is now a pure documentation update with typo fixes and improved NuRec/3DGS documentation.

Implementation Verdict

Ship it — This is now a clean documentation-only PR with no code changes.

Findings

The previous critical finding about the attach_stage() call breaking regression tests is no longer applicable — that code change has been removed from the PR.

All documentation changes are correct:

Typo fixes (deterministically, IsaacLab, flexibility, mark)
Added section anchors for cross-references
Clearer 3D Gaussian Splatting documentation with step-by-step instructions

🔵 Minor: docs/source/overview/imitation-learning/humanoids_imitation.rst:856 — Missing newline at EOF
The file ends without a trailing newline, which may cause a minor diff noise in future commits. Non-blocking.

FabricFrameView lacks _prim_path; reach through _usd_view.

isaaclab-review-bot

🤖 Isaac Lab Review Bot

Summary

This follow-up adds two bug fixes to the previously documentation-only PR: fixing warp tensor access patterns in generate_data.py and handling FabricFrameView composition in scene_utils.py.

Implementation Verdict

Ship it — The new code changes are correct bug fixes.

Findings

Previous documentation concerns were addressed (trailing newline is now present).

The two new code changes are valid fixes:

generate_data.py:331-332 — Correctly accesses .torch property on warp tensors before passing to torch.as_tensor(). This matches the pattern used elsewhere in the same file (e.g., lines 335-336 for root_pos_w.torch on robot).
scene_utils.py:108-110 — Correctly handles FabricFrameView which wraps UsdFrameView. The _prim_path attribute lives on the inner USD view, so using getattr(xform_prim, "_usd_view", xform_prim) with fallback is the right approach.

Both fixes are well-commented and follow existing codebase patterns.

jaybdub · 2026-05-05T17:19:58Z

LGTM

isaaclab-review-bot

🤖 Isaac Lab Review Bot

Summary

No new commits since the previous review. The PR remains in good shape with valid bug fixes for warp tensor access and FabricFrameView handling, plus documentation improvements for 3D Gaussian Splatting integration.

Implementation Verdict

Ship it — Previous review findings still apply; no new issues introduced.

dengyuchenkit requested review from Mayankm96, jtigue-bdai and kellyguo11 as code owners April 16, 2026 23:29

github-actions Bot added the documentation Improvements or additions to documentation label Apr 16, 2026

greptile-apps Bot reviewed Apr 16, 2026

View reviewed changes

dengyuchenkit force-pushed the ydeng/update_docs branch from 07ae91b to 3b229f7 Compare April 20, 2026 23:03

github-actions Bot added the isaac-lab Related to Isaac Lab team label Apr 20, 2026

dengyuchenkit force-pushed the ydeng/update_docs branch from 3b229f7 to 2e17234 Compare April 21, 2026 00:47

isaaclab-review-bot Bot reviewed Apr 21, 2026

View reviewed changes

dengyuchenkit force-pushed the ydeng/update_docs branch 4 times, most recently from adefe32 to afcfcfa Compare April 22, 2026 23:10

dengyuchenkit force-pushed the ydeng/update_docs branch from e7ded88 to 33a899f Compare May 1, 2026 20:33

isaaclab-review-bot Bot reviewed May 1, 2026

View reviewed changes

update docs: load 3d gaussian background in locomanipulation sdg

338b351

dengyuchenkit force-pushed the ydeng/update_docs branch from 33a899f to d822d81 Compare May 1, 2026 20:54

Address comments

9e9ce19

dengyuchenkit force-pushed the ydeng/update_docs branch from d822d81 to 9e9ce19 Compare May 1, 2026 21:08

Fix FrameView rebuild for FabricFrameView backend

dba8928

FabricFrameView lacks _prim_path; reach through _usd_view.

dengyuchenkit requested a review from xyao-nv as a code owner May 1, 2026 22:09

github-actions Bot added the isaac-mimic Related to Isaac Mimic team label May 1, 2026

Use ProxyArray.torch in generate_data

9765942

isaaclab-review-bot Bot reviewed May 1, 2026

View reviewed changes

kellyguo11 approved these changes May 2, 2026

View reviewed changes

peterd-NV approved these changes May 5, 2026

View reviewed changes

Merge branch 'develop' into ydeng/update_docs

0b9c1ba

isaaclab-review-bot Bot reviewed May 5, 2026

View reviewed changes

dengyuchenkit and others added 3 commits May 6, 2026 10:26

Merge branch 'develop' into ydeng/update_docs

5d4f028

Merge branch 'develop' into ydeng/update_docs

a893341

Merge branch 'develop' into ydeng/update_docs

5d18282

kellyguo11 merged commit 2388178 into isaac-sim:develop May 7, 2026
33 of 34 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Updates docs for using nurec background in locomanipulation sdg#5301

Updates docs for using nurec background in locomanipulation sdg#5301
kellyguo11 merged 8 commits into
isaac-sim:developfrom
dengyuchenkit:ydeng/update_docs

dengyuchenkit commented Apr 16, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Apr 16, 2026 •

edited

Loading

Comments Outside Diff (1)

Uh oh!

greptile-apps Bot Apr 16, 2026

Uh oh!

isaaclab-review-bot Bot left a comment

Uh oh!

isaaclab-review-bot Bot Apr 21, 2026

Uh oh!

jaybdub commented Apr 23, 2026

Uh oh!

isaaclab-review-bot Bot left a comment

Uh oh!

isaaclab-review-bot Bot left a comment

Uh oh!

jaybdub commented May 5, 2026

Uh oh!

isaaclab-review-bot Bot left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

dengyuchenkit commented Apr 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Type of change

Checklist

Uh oh!

greptile-apps Bot commented Apr 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 4/5

Important Files Changed

Flowchart

Comments Outside Diff (1)

Uh oh!

greptile-apps Bot Apr 16, 2026

Choose a reason for hiding this comment

Uh oh!

isaaclab-review-bot Bot left a comment

Choose a reason for hiding this comment

🤖 Isaac Lab Review Bot

Summary

Architecture Impact

Implementation Verdict

Test Coverage

CI Status

Findings

Uh oh!

isaaclab-review-bot Bot Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

jaybdub commented Apr 23, 2026

Uh oh!

isaaclab-review-bot Bot left a comment

Choose a reason for hiding this comment

🤖 Isaac Lab Review Bot

Summary

Implementation Verdict

Findings

Uh oh!

isaaclab-review-bot Bot left a comment

Choose a reason for hiding this comment

🤖 Isaac Lab Review Bot

Summary

Implementation Verdict

Findings

Uh oh!

jaybdub commented May 5, 2026

Uh oh!

isaaclab-review-bot Bot left a comment

Choose a reason for hiding this comment

🤖 Isaac Lab Review Bot

Summary

Implementation Verdict

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dengyuchenkit commented Apr 16, 2026 •

edited

Loading

greptile-apps Bot commented Apr 16, 2026 •

edited

Loading