Skip to content

Move rendering-correctness tests into dedicated CI jobs#5649

Merged
pbarejko merged 5 commits into
isaac-sim:developfrom
nvsekkin:esekkin/ci-rendering-correctness-job
May 16, 2026
Merged

Move rendering-correctness tests into dedicated CI jobs#5649
pbarejko merged 5 commits into
isaac-sim:developfrom
nvsekkin:esekkin/ci-rendering-correctness-job

Conversation

@nvsekkin

@nvsekkin nvsekkin commented May 15, 2026

Copy link
Copy Markdown
Collaborator

Description

Factor 7 test_rendering_*.py tests into two new jobs split along the kit-less boundary that tools/conftest.py already special-cases:

  • test-rendering-correctness (4 in-kit files)
  • test-rendering-correctness-kitless (3 kitless files)

Note: This uses the same TESTS_TO_SKIP + include-files pattern that test-curobo, test-skillgen, and test-environments-training already use.

Checklist

  • I have read and understood the contribution guidelines
  • I have run the pre-commit checks with ./isaaclab.sh --format
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • I have updated the changelog and the corresponding version in the extension's config/extension.toml file
  • I have added my name to the CONTRIBUTORS.md or my name already exists there

@github-actions github-actions Bot added isaac-lab Related to Isaac Lab team infrastructure labels May 15, 2026

@isaaclab-review-bot isaaclab-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isaac Lab PR Review: CI Configuration Changes

Summary

This PR improves CI efficiency by moving 7 rendering-correctness tests from the generic isaaclab_tasks [N/3] shards into two dedicated CI jobs, addressing load imbalance issues.

✅ Strengths

Architecture & Design

  • Clean separation following the existing kit/kitless boundary already established in tools/conftest.py
  • Consistent pattern usage with TESTS_TO_SKIP + include-files, matching existing jobs (test-curobo, test-skillgen, test-environments-training)
  • Well-documented constants with clear docstrings explaining the rationale

CI Workflow Correctness

  • Both new jobs (rendering-correctness, rendering-correctness-kitless) correctly depend on [build, config] with appropriate conditional execution
  • Proper continue-on-error: true maintains consistency with other similar test jobs
  • 120-minute timeout is reasonable for rendering tests
  • fetch-depth: 1 with LFS enabled is correct for tests that may need large assets

Key Observation: ovrtx Package Relocation
The extra-pip-packages: "ovrtx" was removed from all 3 isaaclab_tasks [N/3] shards and added only to the test-rendering-correctness-kitless job. This is correct because:

  • Only kitless rendering tests require ovrtx (as noted in the docstring referencing _install_ovrtx_optional_dep)
  • Removing unnecessary dependency installation from the main shards improves their startup time

📋 Observations (Non-blocking)

  1. Test file alignment: The 7 test files match the split:

    • In-kit (4): test_rendering_cartpole.py, test_rendering_dexsuite_kuka.py, test_rendering_registered_tasks.py, test_rendering_shadow_hand.py
    • Kitless (3): test_rendering_cartpole_kitless.py, test_rendering_dexsuite_kuka_kitless.py, test_rendering_shadow_hand_kitless.py
  2. No test_rendering_registered_tasks_kitless.py: The kitless job has 3 files vs 4 in-kit files. This appears intentional based on the existing test structure.

  3. Changelog skip file: Appropriate use of .skip extension in changelog.d/ for CI-only changes that don't require user-facing changelog entries.

🔍 Minor Suggestions

  1. Consider documenting the runtime improvement: The PR description mentions the [3/3] shard going from ~35 min to ~60 min. It might be valuable to add expected runtime for the new jobs in the PR description for future reference.

  2. Job naming consistency: The YAML job IDs use hyphens (test-rendering-correctness) while the display names don't include the "test-" prefix. This matches existing patterns (test-curobo → display name curobo), so this is consistent.

Verdict: LGTM

This is a well-structured CI optimization that follows established patterns in the Isaac Lab codebase. The changes are minimal, targeted, and low-risk. The test isolation will improve shard balance and make rendering test failures easier to identify and debug.


Automated review by Isaac Lab Review Bot

@greptile-apps

greptile-apps Bot commented May 15, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR moves all 7 test_rendering_*.py files out of the round-robin isaaclab_tasks [N/3] shards into two new dedicated CI jobs (test-rendering-correctness for 4 in-kit tests, test-rendering-correctness-kitless for 3 OVRTX kitless tests), following the same TESTS_TO_SKIP + include-files pattern already used by test-curobo, test-skillgen, and test-environments-training.

  • New constants RENDERING_CORRECTNESS_TESTS and RENDERING_CORRECTNESS_KITLESS_TESTS are added to test_settings.py and splat into TESTS_TO_SKIP, so conftest.py's existing override logic correctly re-includes them only in the dedicated jobs.
  • extra-pip-packages: \"ovrtx\" is removed from all three shard jobs (confirmed safe: remaining shard tests that mention ovrtx either use it only as a preset string or skip the relevant cases with pytest.mark.skip).

Confidence Score: 4/5

The change is safe to merge; the only concern is a stale Sphinx cross-reference in a docstring.

The CI restructuring is mechanically straightforward and mirrors an established pattern used by three other dedicated test jobs. The include-files override of TESTS_TO_SKIP in conftest.py is already well-tested. Removing ovrtx from the shard jobs is verified safe: no non-rendering test in isaaclab_tasks actually imports the package at runtime. The one finding is a docstring that points to tools.conftest._install_ovrtx_optional_dep, a function that does not exist anywhere in the repo.

tools/test_settings.py — the RENDERING_CORRECTNESS_KITLESS_TESTS docstring has a stale function cross-reference.

Important Files Changed

Filename Overview
.github/workflows/build.yaml Adds two new CI jobs (test-rendering-correctness and test-rendering-correctness-kitless) following the existing include-files pattern, and removes the now-redundant ovrtx extra-pip-packages from the three isaaclab_tasks shard jobs.
tools/test_settings.py Introduces RENDERING_CORRECTNESS_TESTS and RENDERING_CORRECTNESS_KITLESS_TESTS lists and splats them into TESTS_TO_SKIP so the shard jobs skip them; one docstring cross-reference points to a non-existent function.
source/isaaclab_tasks/changelog.d/esekkin-ci-rendering-correctness-job.skip CI-only changelog skip file; no functional changes.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    B[build] --> S1[isaaclab_tasks shard 1/3\nno ovrtx]
    B --> S2[isaaclab_tasks shard 2/3\nno ovrtx]
    B --> S3[isaaclab_tasks shard 3/3\nno ovrtx]
    B --> RC[test-rendering-correctness\n4 in-kit files\nno extra packages]
    B --> RCK[test-rendering-correctness-kitless\n3 kitless files\nextra-pip: ovrtx]

    subgraph TESTS_TO_SKIP
        direction LR
        T1["test_rendering_cartpole.py\ntest_rendering_dexsuite_kuka.py\ntest_rendering_registered_tasks.py\ntest_rendering_shadow_hand.py"]
        T2["test_rendering_cartpole_kitless.py\ntest_rendering_dexsuite_kuka_kitless.py\ntest_rendering_shadow_hand_kitless.py"]
    end

    S1 -.skips via TESTS_TO_SKIP.-> T1
    S1 -.skips via TESTS_TO_SKIP.-> T2
    RC --include-files overrides skip--> T1
    RCK --include-files overrides skip--> T2
Loading

Reviews (1): Last reviewed commit: "Move rendering-correctness tests into de..." | Re-trigger Greptile

Comment thread tools/test_settings.py
@isaaclab-review-bot

Copy link
Copy Markdown

Incremental Review (e299c70)

Previous feedback addressed - The stale :func: cross-reference to tools.conftest._install_ovrtx_optional_dep has been removed from the RENDERING_CORRECTNESS_KITLESS_TESTS docstring. The simplified docstring is now accurate and avoids Sphinx warnings.

No new issues found in this iteration. LGTM! 🚀

@isaaclab-review-bot

Copy link
Copy Markdown

Bot Review Update (2af4bd9):

✅ Previous concern about the stale docstring reference has been addressed.

New changes in this push look good:

  • Docker non-root user migration is well-structured with proper CI verification jobs
  • Frame stacking observation term (stacked_image) includes comprehensive tests
  • Pink IK test fixes and tolerance adjustments are reasonable
  • Documentation updates for XR performance are helpful

No new issues identified.

@isaaclab-review-bot

Copy link
Copy Markdown

🔄 Incremental Review Update (commit 3fb23dd)

Reviewed changes since previously reviewed commit (2af4bd9).

Previous Concerns

Docstring reference fixed: The stale :func: cross-reference to tools.conftest._install_ovrtx_optional_dep in tools/test_settings.py has been corrected — the new docstring no longer contains the invalid reference.

New Changes Summary

The incremental changes continue the consistent refactoring patterns already present in the PR:

  • Import path updates (from isaaclab.utils import configclassfrom isaaclab.utils.configclass import configclass)
  • Renderer updates to accept ProxyArray instead of torch.Tensor
  • Test assertion updates to use warp dtypes (wp.uint8, wp.float32) matching the new warp-backed storage

Assessment

No new issues found. The incremental changes are consistent with the PR's refactoring goals.

@pbarejko pbarejko self-requested a review May 16, 2026 15:47
@pbarejko pbarejko merged commit c67ed38 into isaac-sim:develop May 16, 2026
38 checks passed
matthewtrepte pushed a commit to matthewtrepte/IsaacLab that referenced this pull request May 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

infrastructure isaac-lab Related to Isaac Lab team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants