Skip to content

Update Visualizer Integration Tests + Some Visualizer Fixes#5733

Merged
kellyguo11 merged 9 commits into
isaac-sim:developfrom
matthewtrepte:mtrepte/update_viz_integration_test
May 28, 2026
Merged

Update Visualizer Integration Tests + Some Visualizer Fixes#5733
kellyguo11 merged 9 commits into
isaac-sim:developfrom
matthewtrepte:mtrepte/update_viz_integration_test

Conversation

@matthewtrepte

@matthewtrepte matthewtrepte commented May 21, 2026

Copy link
Copy Markdown
Contributor

Description

Improve the visualizer integration test

  • Unskip all tests
  • Add retry loop for pause/replay step to reduce flakyness
  • Separate out visualizer tests into separate files and kit processes
  • Fix camera pose of Kit Viz + Newton_MJWarp case
  • Add debug image capture mode (disabled by default)
  • Add simple tiled image check

Other visualizer fixes

  • Pin pyarrow to 22.0.0 to fix Rerun on Windows
  • Make tiled camera view instructions in the visualizer docs more explicit
  • Fix Newton Pyglet issue on Windows
  • Change tiled camera panel in Newton Viewer from "Image Logger (1)" to "Tiled Visualizer Camera"

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)

Screenshots

Please attach before and after screenshots of the change if applicable.

Checklist

  • I have read and understood the contribution guidelines
  • I have run the pre-commit checks with ./isaaclab.sh --format
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • I have updated the changelog and the corresponding version in the extension's config/extension.toml file
  • I have added my name to the CONTRIBUTORS.md or my name already exists there

@github-actions github-actions Bot added documentation Improvements or additions to documentation isaac-lab Related to Isaac Lab team labels May 21, 2026

@isaaclab-review-bot isaaclab-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Isaac Lab Code Review

Thank you for this substantial PR improving the visualizer integration tests! I've conducted a thorough review focusing on test quality, Isaac Lab conventions, and potential issues.

✅ What's Working Well

  1. Comprehensive Test Refactoring: The new test_visualizer_integration.py is significantly more robust than the previous implementation, with better structure and more thorough coverage of play/pause/resume cycles.

  2. Multi-Backend Testing: The parameterized tests covering both PhysX and Newton MJWarp backends are well-designed and follow pytest best practices.

  3. Debug Infrastructure: The debug image capture system (_WRITE_VIS_DEBUG_FRAMES, _save_visualizer_debug_image) is a valuable addition for diagnosing flaky tests.

  4. Clean Config Reorganization (new in 6055e74): The VisualizerCfg fields are now logically grouped with section comments (Primary interactive camera, Tiled camera settings, Partial visualization, Markers, Live Plots, Internal). This improves readability significantly.

  5. Improved UI Naming: The Newton visualizer's ImageLogger controls renamed to "Tiled Camera View" with a dedicated _draw_tiled_camera_view_controls method provides a clearer user experience.

  6. Proper Test Separation (new in 6055e74): Splitting tiled camera tests into dedicated test_*_tiled_camera_motion functions is a much cleaner architecture than mixing assertions in the main motion tests.

✅ All Issues Resolved (dc7e30f)

The latest commit addresses all previously flagged items:

  1. _WRITE_VIS_DEBUG_FRAMES = False — ✅ Now set to False by default in visualizer_integration_utils.py

  2. Pyglet headless configuration — ✅ Handled automatically via NewtonVisualizerCfg(headless=True) which triggers runtime_headless detection in newton_visualizer.py

  3. Rerun import guard — ✅ Now uses deferred imports inside _get_visualizer_cfg() with proper from isaaclab_visualizers.rerun import ... pattern

📋 Changes in This Update (395ee63dc7e30f)

File Change
visualizer_integration_utils.py ✅ New shared module with all test helpers; _WRITE_VIS_DEBUG_FRAMES = False
test_visualizer_integration_{newton,physx}.py ✅ Backend-specific test entry points
test_visualizer_tiled_integration_{newton,physx}.py ✅ Dedicated tiled camera test entry points
newton_visualizer.py ✅ Cross-platform headless detection (sys.platform not in (\"win32\", \"darwin\"))
test_visualizer_cartpole_integration.py 🗑️ Removed (replaced by new modular structure)

🏁 Summary

All blockers resolved! The PR is ready for merge.

  • Set _WRITE_VIS_DEBUG_FRAMES = False
  • Pyglet headless handled via cfg
  • Deferred rerun imports

Excellent refactoring work — the new modular test structure with separate backend/tiled test files is much cleaner.


📝 Update Review (dc7e30f)

This update includes additional substantial changes beyond the visualizer test improvements:

✅ New Changes Reviewed

1. Newton Sensors Stale Data Fix (Issue #4970) — Excellent fix!

  • Added timestamp guards to all Newton sensor kernels (ContactSensor, Imu, Pva, JointWrenchSensor)
  • Prevents reading stale pre-reset physics data when scene.reset(env_ids) is called without a subsequent physics step
  • Comprehensive regression tests added for each sensor type
  • PhysX sensors also updated with matching timestamp guards

2. OVRTX Renderer Cleanup

  • Removed legacy OVRTX 0.2.x code paths (good cleanup for forward compatibility)
  • Removed _IS_OVRTX_0_3_0_OR_NEWER checks and legacy kernels
  • Simplified create_scene_partition_attributes() signature

3. PhysX Scene-Data Rigid Body View Fix

  • Changed from wildcard patterns to exact prim paths
  • Avoids spurious warnings for assets with joint prims sharing body names

4. PPISP Camera Discovery Fix

  • Fixed auto_camera_ppisp_cfg() to continue scanning when generated RenderProducts lack PPISP child
  • Added test case for this edge case

5. Kitless Launch Handling

  • Fixed explicit --viz none handling
  • Properly detects visualizer_explicit=True with visualizer=None case

6. Visualizer Test Refactoring

  • Tests properly split into separate files per physics backend
  • Tiled camera tests in dedicated files
  • Shared utilities in visualizer_integration_utils.py

7. Documentation Updates

  • Removed "experimental" statement from isaaclab_ov package docs
  • Added comprehensive changelogs for all fixes

✅ Code Quality

All changes follow Isaac Lab conventions:

  • Proper changelog entries in changelog.d/
  • Clear docstrings referencing issue numbers
  • Comprehensive test coverage for all fixes
  • Clean separation of concerns

No new issues identified. The PR continues to be ready for merge.


Automated review by Isaac Lab Review Bot | Updated for commit dc7e30f


📝 Update Review (90c5a6e)

The latest commits add a substantial pyproject.toml migration across the entire Isaac Lab codebase. This is a clean modernization effort.

✅ Migration Changes Reviewed

1. Build System Migration

  • All setup.py files removed across 18+ subpackages
  • Package metadata moved to pyproject.toml [project] sections
  • Build requirements simplified: removed toml dependency from build-system
  • Uses setuptools>=70.0,<82.0.0 consistently

2. Runtime Dependency Updates

  • Replaced third-party toml with stdlib tomllib (Python 3.11+)
  • SimulationContext and tests now use tomllib for reading .kit preset files
  • __version__ now uses importlib.metadata.version() pattern consistently

3. Test Dependencies Refactored

  • Test packages (pytest, pytest-mock, junitparser, flatdict, flaky, coverage>=7.6.1) moved from base deps to [test] optional extra
  • Docker images updated to install pytest explicitly
  • CI action (.github/actions/run-tests/action.yml) now installs test deps before running

4. Other Fixes

  • PhysX Rigid Body View: Restored wildcard patterns (/World/envs/env_*) for scene-data rigid body view — fixes Newton visualizers not updating live PhysX transforms
  • License Exception: Updated typing_extensions license from PSF-2.0 to Python Software Foundation License
  • Changelog CLI: Added version sync check between extension.toml and pyproject.toml

5. Minor Code Cleanup

  • install.py references updated from setup.py to pyproject.toml
  • Observation manager test fixed for whitespace sensitivity in table rendering
  • Benchmark utils updated to use pyproject.toml as repo marker
  • isaaclab_rl extra names normalized to PEP 685 hyphenated forms (rsl-rl, rl-games)

✅ Quality Assessment

All changes follow Isaac Lab conventions:

  • Proper changelog entries in changelog.d/
  • Consistent patterns applied across all packages
  • Good use of importlib.metadata for version detection with PackageNotFoundError fallback
  • Tests updated to match new structure

No issues found. This is a well-executed modernization.


Automated review by Isaac Lab Review Bot | Updated for commit 90c5a6e


📝 Update Review (5b41c70)

Documentation-only update adding known issue documentation for Newton + Digit robots.

✅ Changes Reviewed

1. Newton Closed-Loop Articulation Limitation

  • Added detailed known issue to docs/source/refs/issues.rst explaining why Digit robots don't work on Newton
  • The root cause (duplicate link slots in ArticulationView for bodies that are children of multiple joints) is well-documented
  • Workaround provided: use physx preset until upstream fix lands

2. Environment Table Updates

  • Updated docs/source/overview/environments.rst to remove newton_mjwarp from supported physics presets for Digit environments
  • Affects: Isaac-Velocity-Flat-Digit-v0, Isaac-Velocity-Rough-Digit-v0, Isaac-Tracking-LocoManip-Digit-v0

3. Changelog Skip Marker

  • Added changelog.d/octi-docs-digit-newton-known-limitation.skip appropriately for docs-only change

✅ Quality Assessment

  • Documentation is accurate and technically detailed
  • Follows Isaac Lab documentation conventions
  • Properly uses changelog skip marker for docs-only changes

No issues found. PR remains ready for merge.


Automated review by Isaac Lab Review Bot | Updated for commit 5b41c70

@greptile-apps

greptile-apps Bot commented May 21, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR replaces the old test_visualizer_cartpole_integration.py with a new, more comprehensive test_visualizer_integration.py that unskips all visualizer tests, adds retry logic for flaky pause/replay steps, fixes the Kit Viz + Newton MJWarp camera pose, and introduces a debug image capture mode. It also adds new SceneDataBackend and SceneDataProvider abstractions in isaaclab.physics and isaaclab.scene, and adjusts KitVisualizer to allow camera sensor creation in headless mode while skipping only the UI image window.

  • _WRITE_VIS_DEBUG_FRAMES = True is set in the new test file, contradicting the PR description of "disabled by default"; this causes PNG files to be written on every CI run and introduces a hard Pillow dependency.
  • pyglet.options[\"headless\"] = True was present in the deleted file with an explicit comment explaining its necessity for Newton's ViewerGL but was not carried over, risking display-related failures on headless CI.
  • SceneDataProvider._walk_camera_prims assigns world_id=0 to cameras outside /World/envs/ paths, which may silently conflate global cameras with env_0 slot data."

Confidence Score: 3/5

Two test-file issues need fixing before merging: the debug PNG write flag is enabled by default and the pyglet headless guard was dropped, both of which can cause unexpected CI failures.

The debug flag being left as True will write PNG files on every CI run and adds an implicit Pillow requirement; the missing pyglet headless setup is a regression from the deleted file that the original comment explicitly called out as required for headless Newton ViewerGL. Both issues are in the test path, not production code, but can cause CI runs to fail or behave unexpectedly.

source/isaaclab_visualizers/test/test_visualizer_integration.py needs the debug flag corrected and the pyglet headless setup restored; source/isaaclab/isaaclab/scene/scene_data_provider.py warrants a second look at how non-env-path cameras are handled in _walk_camera_prims.

Important Files Changed

Filename Overview
source/isaaclab_visualizers/test/test_visualizer_integration.py New consolidated integration test replacing the deleted file; _WRITE_VIS_DEBUG_FRAMES is True (should be False per PR description) and the pyglet headless setup required for Newton's ViewerGL on headless CI was dropped.
source/isaaclab_visualizers/isaaclab_visualizers/kit/kit_visualizer.py Moves the headless-mode guard for camera setup to allow camera sensor creation in headless mode while skipping only the UI image window; splits _update_camera_image_panel guard so camera pose updates run even without an image provider.
source/isaaclab/isaaclab/scene/scene_data_provider.py New SceneDataProvider with Warp conversion kernels and USD camera/env traversal. Non-env-path cameras silently get world_id=0 in _walk_camera_prims, which can mix global cameras into env_0 slots.
source/isaaclab/isaaclab/physics/scene_data_backend.py New abstract SceneDataBackend interface with four Warp struct formats; clean separation from isaaclab.scene to avoid AppLauncher import ordering issues.
source/isaaclab_visualizers/test/test_visualizer_cartpole_integration.py Deleted file intentionally replaced by test_visualizer_integration.py.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A["KitVisualizer._setup_camera_sensor_view()"] --> B{uses_camera_sensor_view?}
    B -- No --> Z[return]
    B -- Yes --> C{cameras_enabled?}
    C -- No --> ERR[raise RuntimeError]
    C -- Yes --> D[Create / resolve camera sensor]
    D --> E{runtime_headless?}
    E -- No --> F[_setup_camera_image_window]
    E -- Yes --> G[Log: running without image panel]
    H["KitVisualizer._update_camera_image_panel()"] --> I{camera_sensor is None?}
    I -- Yes --> Z2[return]
    I -- No --> J[Update owned camera poses]
    J --> K[camera_sensor.update]
    K --> L{camera_image_provider is None?}
    L -- Yes --> Z3[return]
    L -- No --> M[camera_rgb_batch → upload to panel]
Loading

Reviews (1): Last reviewed commit: "sync" | Re-trigger Greptile

"isaaclab.sim.simulation_context",
)

_WRITE_VIS_DEBUG_FRAMES = True

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Debug flag enabled in test code — contradicts PR description

The PR description states that debug image capture is "disabled by default," but _WRITE_VIS_DEBUG_FRAMES = True means every CI run will write PNG files to logs/viz_integration_captures. This also introduces a hard runtime dependency on PIL (imported inside _save_visualizer_debug_image); if Pillow is not in the test dependencies, every test that reaches frame-capture paths will raise an ImportError.

Suggested change
_WRITE_VIS_DEBUG_FRAMES = True
_WRITE_VIS_DEBUG_FRAMES = False
"""Whether to emit visualizer debug PNGs during integration tests."""

Comment on lines +22 to +29
from __future__ import annotations

import warp as wp

from isaaclab.app import AppLauncher

# launch Kit app
simulation_app = AppLauncher(headless=True, enable_cameras=True).app

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Missing pyglet headless configuration required for Newton ViewerGL

The deleted test_visualizer_cartpole_integration.py contained an explicit comment explaining this requirement: "Pyglet must use HeadlessWindow (EGL) before pyglet.window is imported so Newton ViewerGL can construct without an X11 display." The new file omits import pyglet; pyglet.options["headless"] = True before AppLauncher. On headless CI machines without X11/Wayland, pyglet may fall back to a crashing backend at window-construction time, causing Newton visualizer tests to fail with a display error.

Suggested change
from __future__ import annotations
import warp as wp
from isaaclab.app import AppLauncher
# launch Kit app
simulation_app = AppLauncher(headless=True, enable_cameras=True).app
from __future__ import annotations
import warp as wp
# Pyglet must use HeadlessWindow (EGL) before ``pyglet.window`` is imported so Newton
# ViewerGL can construct without an X11 display (matches ``headless=True`` on NewtonVisualizerCfg).
import pyglet
pyglet.options["headless"] = True
from isaaclab.app import AppLauncher
# launch Kit app
simulation_app = AppLauncher(headless=True, enable_cameras=True).app

Comment on lines +264 to +268
if visualizer_kind == "rerun":
__import__("newton")
from isaaclab_visualizers.rerun import RerunVisualizer, RerunVisualizerCfg

web_port, grpc_port = _allocate_rerun_test_ports(host="127.0.0.1")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Missing __import__("rerun") guard inconsistent with viser handling

For viser (line 255) the code guards with __import__("viser") before the lazy import, allowing a clean skip if the package is absent. The __import__("rerun") guard present in the deleted file was removed, so a missing rerun package now produces an ImportError from the from isaaclab_visualizers.rerun import ... line rather than a clean skip signal.

Comment on lines +390 to +400
Args:
stage: USD stage to traverse, or ``None``.

Returns:
Dictionary with keys ``order`` (template prim paths using ``env_%d``),
``positions``, ``orientations`` (per-camera, per-env, with ``None`` for
absent envs), and ``num_envs``. Returns ``None`` when ``stage`` is ``None``.
"""
if stage is None:
return None

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Non-env-path cameras silently assigned world_id=0

In _walk_camera_prims, world_id is reset to 0 at the top of each iteration. Any camera prim whose path does not match _ENV_PATH_RE (e.g. /World/Camera) gets world_id=0, and the if world_id > num_envs: num_envs = world_id branch updates num_envs from -1 to 0. After num_envs += 1 the result is 1, so that global camera is placed into per_world_pos[0] — indistinguishable from an env_0 camera. Callers iterating the result by environment index will silently receive a global camera pose as if it belongs to env_0.

@isaaclab-review-bot

Copy link
Copy Markdown

🤖 Incremental Review Update (395ee63)

Changes since last review (004a12c):

Summary of New Changes

This update continues the CI/test infrastructure improvements with the following key changes:

1. New exclude-pattern Parameter for Test Filtering (.github/actions/*)

  • Added exclude-pattern input to both run-package-tests and run-tests actions
  • Supports comma-separated substrings for excluding test files
  • Combines with existing filter-pattern (include + exclude logic)
  • Good enhancement for more granular test selection ✅

2. JUnit XML Upload Steps (.github/workflows/*)

  • Added JUnit XML artifact upload to run-tests/action.yml
  • Added JUnit XML artifact upload to install-ci.yml for both uv and conda jobs
  • Improves test result visibility and integration with CI tooling ✅

3. Non-root Verification Consolidation (.github/workflows/build.yaml)

  • Removed standalone verify-base-non-root and verify-curobo-non-root jobs
  • Folded non-root verification steps directly into test-isaaclab-ov and test-curobo jobs
  • This reduces runner usage while maintaining the regression checks ✅
  • Updated triggered_jobs description to reflect the change

4. Rendering Test Exclusion

  • Added exclude-pattern: "test_rendering_" to all three test-isaaclab-tasks-* shards
  • Removed RENDERING_CORRECTNESS_TESTS and RENDERING_CORRECTNESS_KITLESS_TESTS from tools/test_settings.py
  • Removed large integration test section from test_shadow_hand_vision_presets.py (the _RENDER_CORRECTNESS_CASES parametrized tests and test_camera_renders_not_empty)

5. Dockerfile arm64 Dependency

  • Added libgmp-dev for arm64 builds (required by pytetwild's fTetWild source build)

6. Changelog Consolidation

  • Removed changelog.d fragments that were incorporated into main changelogs
  • Updated isaaclab_ov version to 0.4.1 with proper CHANGELOG.rst entry

Review Assessment

The changes are well-structured CI improvements that:

  • Reduce runner time by consolidating non-root verification into existing test jobs
  • Improve test filtering flexibility with the new exclude-pattern
  • Clean up test settings by properly excluding rendering tests via CI config rather than Python skip lists

No concerns identified with this update. The changes appear to be straightforward CI workflow optimizations and test infrastructure cleanup.


Reviewed commits: 004a12c395ee63

@matthewtrepte matthewtrepte changed the title Update Visualizer Integration Tests Update Visualizer Integration Tests + Some Visualizer Fixes May 27, 2026
@matthewtrepte matthewtrepte force-pushed the mtrepte/update_viz_integration_test branch from 8db166f to 742b5aa Compare May 27, 2026 21:17
@matthewtrepte matthewtrepte force-pushed the mtrepte/update_viz_integration_test branch from 26f8d6d to c4af93a Compare May 27, 2026 22:44
@kellyguo11 kellyguo11 merged commit 1efa48a into isaac-sim:develop May 28, 2026
60 of 61 checks passed
kellyguo11 pushed a commit to kellyguo11/IsaacLab-public that referenced this pull request May 28, 2026
…m#5733)

<!--
Thank you for your interest in sending a pull request. Please make sure
to check the contribution guidelines.

Link:
https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html

💡 Please try to keep PRs small and focused. Large PRs are harder to
review and merge.
-->

Improve the visualizer integration test
- Unskip all tests
- Add retry loop for pause/replay step to reduce flakyness
- Separate out visualizer tests into separate files and kit processes
- Fix camera pose of Kit Viz + Newton_MJWarp case
- Add debug image capture mode (disabled by default)
- Add simple tiled image check

Other visualizer fixes
- Pin pyarrow to 22.0.0 to fix Rerun on Windows
- Make tiled camera view instructions in the visualizer docs more
explicit
- Fix Newton Pyglet issue on Windows
- Change tiled camera panel in Newton Viewer from "Image Logger (1)" to
"Tiled Visualizer Camera"

<!-- As a practice, it is recommended to open an issue to have
discussions on the proposed pull request.
This makes it easier for the community to keep track of what is being
developed or added, and if a given feature
is demanded by more than one party. -->

<!-- As you go through the list, delete the ones that are not
applicable. -->

- Bug fix (non-breaking change which fixes an issue)
- New feature (non-breaking change which adds functionality)

Please attach before and after screenshots of the change if applicable.

<!--
Example:

| Before | After |
| ------ | ----- |
| _gif/png before_ | _gif/png after_ |

To upload images to a PR -- simply drag and drop an image while in edit
mode and it should upload the image directly. You can then paste that
source into the above before/after sections.
-->

- [ ] I have read and understood the [contribution
guidelines](https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html)
- [ ] I have run the [`pre-commit` checks](https://pre-commit.com/) with
`./isaaclab.sh --format`
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have updated the changelog and the corresponding version in the
extension's `config/extension.toml` file
- [ ] I have added my name to the `CONTRIBUTORS.md` or my name already
exists there

<!--
As you go through the checklist above, you can mark something as done by
putting an x character in it

For example,
- [x] I have done this task
- [ ] I have not done this task
-->
kellyguo11 added a commit that referenced this pull request May 28, 2026
… teleop/mimic dependency (#5829)

# Description

Cherry pick bug fix PRs from develop:

- #5821
- #5820
- #5733
- #5824

---------

Signed-off-by: peterd-NV <peterd@nvidia.com>
Co-authored-by: peterd-NV <peterd@nvidia.com>
Co-authored-by: ooctipus <zhengyuz@nvidia.com>
Co-authored-by: matthewtrepte <mtrepte@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation infrastructure isaac-lab Related to Isaac Lab team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants