Move rendering-correctness tests into dedicated CI jobs#5649
Conversation
There was a problem hiding this comment.
Isaac Lab PR Review: CI Configuration Changes
Summary
This PR improves CI efficiency by moving 7 rendering-correctness tests from the generic isaaclab_tasks [N/3] shards into two dedicated CI jobs, addressing load imbalance issues.
✅ Strengths
Architecture & Design
- Clean separation following the existing kit/kitless boundary already established in
tools/conftest.py - Consistent pattern usage with
TESTS_TO_SKIP+include-files, matching existing jobs (test-curobo,test-skillgen,test-environments-training) - Well-documented constants with clear docstrings explaining the rationale
CI Workflow Correctness
- Both new jobs (
rendering-correctness,rendering-correctness-kitless) correctly depend on[build, config]with appropriate conditional execution - Proper
continue-on-error: truemaintains consistency with other similar test jobs - 120-minute timeout is reasonable for rendering tests
fetch-depth: 1with LFS enabled is correct for tests that may need large assets
Key Observation: ovrtx Package Relocation
The extra-pip-packages: "ovrtx" was removed from all 3 isaaclab_tasks [N/3] shards and added only to the test-rendering-correctness-kitless job. This is correct because:
- Only kitless rendering tests require
ovrtx(as noted in the docstring referencing_install_ovrtx_optional_dep) - Removing unnecessary dependency installation from the main shards improves their startup time
📋 Observations (Non-blocking)
-
Test file alignment: The 7 test files match the split:
- In-kit (4):
test_rendering_cartpole.py,test_rendering_dexsuite_kuka.py,test_rendering_registered_tasks.py,test_rendering_shadow_hand.py - Kitless (3):
test_rendering_cartpole_kitless.py,test_rendering_dexsuite_kuka_kitless.py,test_rendering_shadow_hand_kitless.py
- In-kit (4):
-
No
test_rendering_registered_tasks_kitless.py: The kitless job has 3 files vs 4 in-kit files. This appears intentional based on the existing test structure. -
Changelog skip file: Appropriate use of
.skipextension inchangelog.d/for CI-only changes that don't require user-facing changelog entries.
🔍 Minor Suggestions
-
Consider documenting the runtime improvement: The PR description mentions the
[3/3]shard going from ~35 min to ~60 min. It might be valuable to add expected runtime for the new jobs in the PR description for future reference. -
Job naming consistency: The YAML job IDs use hyphens (
test-rendering-correctness) while the display names don't include the "test-" prefix. This matches existing patterns (test-curobo→ display namecurobo), so this is consistent.
Verdict: LGTM ✅
This is a well-structured CI optimization that follows established patterns in the Isaac Lab codebase. The changes are minimal, targeted, and low-risk. The test isolation will improve shard balance and make rendering test failures easier to identify and debug.
Automated review by Isaac Lab Review Bot
Greptile SummaryThis PR moves all 7
Confidence Score: 4/5The change is safe to merge; the only concern is a stale Sphinx cross-reference in a docstring. The CI restructuring is mechanically straightforward and mirrors an established pattern used by three other dedicated test jobs. The include-files override of TESTS_TO_SKIP in conftest.py is already well-tested. Removing ovrtx from the shard jobs is verified safe: no non-rendering test in isaaclab_tasks actually imports the package at runtime. The one finding is a docstring that points to tools.conftest._install_ovrtx_optional_dep, a function that does not exist anywhere in the repo. tools/test_settings.py — the RENDERING_CORRECTNESS_KITLESS_TESTS docstring has a stale function cross-reference. Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
B[build] --> S1[isaaclab_tasks shard 1/3\nno ovrtx]
B --> S2[isaaclab_tasks shard 2/3\nno ovrtx]
B --> S3[isaaclab_tasks shard 3/3\nno ovrtx]
B --> RC[test-rendering-correctness\n4 in-kit files\nno extra packages]
B --> RCK[test-rendering-correctness-kitless\n3 kitless files\nextra-pip: ovrtx]
subgraph TESTS_TO_SKIP
direction LR
T1["test_rendering_cartpole.py\ntest_rendering_dexsuite_kuka.py\ntest_rendering_registered_tasks.py\ntest_rendering_shadow_hand.py"]
T2["test_rendering_cartpole_kitless.py\ntest_rendering_dexsuite_kuka_kitless.py\ntest_rendering_shadow_hand_kitless.py"]
end
S1 -.skips via TESTS_TO_SKIP.-> T1
S1 -.skips via TESTS_TO_SKIP.-> T2
RC --include-files overrides skip--> T1
RCK --include-files overrides skip--> T2
Reviews (1): Last reviewed commit: "Move rendering-correctness tests into de..." | Re-trigger Greptile |
Incremental Review (e299c70)✅ Previous feedback addressed - The stale No new issues found in this iteration. LGTM! 🚀 |
|
Bot Review Update (2af4bd9): ✅ Previous concern about the stale docstring reference has been addressed. New changes in this push look good:
No new issues identified. |
🔄 Incremental Review Update (commit 3fb23dd)Reviewed changes since previously reviewed commit (2af4bd9). Previous Concerns✅ Docstring reference fixed: The stale New Changes SummaryThe incremental changes continue the consistent refactoring patterns already present in the PR:
AssessmentNo new issues found. The incremental changes are consistent with the PR's refactoring goals. |
Description
Factor 7
test_rendering_*.pytests into two new jobs split along the kit-less boundary thattools/conftest.pyalready special-cases:test-rendering-correctness(4 in-kit files)test-rendering-correctness-kitless(3 kitless files)Note: This uses the same
TESTS_TO_SKIP+include-filespattern thattest-curobo,test-skillgen, andtest-environments-trainingalready use.Checklist
pre-commitchecks with./isaaclab.sh --formatconfig/extension.tomlfileCONTRIBUTORS.mdor my name already exists there