ci(e2e): move parity inventory to report workflow by jyaunches · Pull Request #3800 · NVIDIA/NemoClaw

jyaunches · 2026-05-19T15:46:42Z

Summary

stop normal PR convention lint from enforcing global legacy E2E parity-map/inventory bookkeeping
generate the legacy assertion inventory inside the manual E2E parity workflow and upload it as an artifact
remove the committed generated parity inventory so feature PRs do not absorb broad generated churn

Rationale

The parity inventory is derived from legacy test/e2e/test-*.sh assertions and can churn when unrelated main-branch E2E changes land. Requiring every PR to keep that global migration ledger current creates review noise and opaque CI failures. Parity validation/reporting should happen when producing an E2E parity report, not as a blocker for unrelated PRs.

Validation

npx tsx scripts/e2e/lint-conventions.ts
npx vitest run test/e2e/scenario-framework-tests/e2e-convention-lint.test.ts test/e2e/scenario-framework-tests/e2e-legacy-assertion-inventory.test.ts test/e2e/scenario-framework-tests/e2e-parity-map.test.ts --testTimeout 60000

Summary by CodeRabbit

Chores
- A dedicated manual CI workflow now generates and uploads the parity inventory artifact (.e2e/parity/parity-inventory.generated.json); PRs no longer need regenerated inventory commits.
- Normal linting excludes legacy parity bookkeeping and focuses on suite conventions and retired-wrapper rules.
Tests
- E2E tests updated to accept legacy-style scripts without parity-map entries and to validate coverage using generated parity artifacts (temp dirs where appropriate).
Documentation
- Docs updated with guidance for producing, locating, and locally debugging the generated legacy assertion inventory.

coderabbitai · 2026-05-19T15:47:00Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 08b5a5a7-372f-4139-8f53-555982e78938

📥 Commits

Reviewing files that changed from the base of the PR and between 48330b0 and 3cfdb73.

📒 Files selected for processing (1)

test/e2e/scenario-framework-tests/e2e-coverage-report.test.ts

🚧 Files skipped from review as they are similar to previous changes (1)

test/e2e/scenario-framework-tests/e2e-coverage-report.test.ts

📝 Walkthrough

Walkthrough

Moves legacy parity bookkeeping out of PR lint into the parity-compare CI workflow: the workflow now generates/uploads the legacy assertion inventory and runs strict parity-map validation. PR lint is reduced to suite-step and retired-wrapper checks; docs and tests are updated to reflect the new flow.

Changes

Parity Workflow Migration

Layer / File(s)	Summary
Parity Workflow Generation and Validation Steps `.github/workflows/e2e-parity-compare.yaml`	Workflow dispatch is documented as the source of truth for generated parity reports; new steps generate the legacy assertion inventory JSON and validate the parity map with strict mode enabled.
PR Lint Refactoring to Remove Parity Checks `scripts/e2e/lint-conventions.ts`	Legacy parity inventory/frontier linting and parity-map validation are removed from the PR lint execution path; `main()` now collects findings only from suite-step and retired-wrapper checks; documentation comment and output formatting are updated.
Documentation and Test Updates `test/e2e/docs/README.md`, `test/e2e/scenario-framework-tests/e2e-convention-lint.test.ts`, `test/e2e/scenario-framework-tests/e2e-coverage-report.test.ts`	E2E docs now describe CI-driven parity inventory generation and a local debug extractor; convention-lint tests no longer fail on missing parity-map entries for legacy scripts; coverage-report test uses a temp directory and written parity artifacts for assertions.

Sequence Diagram(s)

sequenceDiagram
  participant Workflow as e2e-parity-compare workflow
  participant Extractor as extract-legacy-assertions.ts
  participant Inventory as .e2e/parity/parity-inventory.generated.json
  participant Validator as check-parity-map.ts
  Workflow->>Extractor: run extractor (--output) to generate inventory
  Extractor->>Inventory: write parity-inventory.generated.json
  Workflow->>Validator: run check-parity-map.ts --strict
  Validator->>Workflow: return validation result

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related issues

Publish E2E parity and coverage report #3589 — related: also requests publishing parity/coverage reports and uploading parity artifacts from CI.

Possibly related PRs

NVIDIA/NemoClaw#3622 — touches parity inventory consumed by parity/coverage reporting.
NVIDIA/NemoClaw#3663 — related changes to parity-map/inventory inputs used by validation and tests.

Suggested labels

CI/CD, E2E

Suggested reviewers

cv

Poem

🐰 Pipelines hum and artifacts gleam,

Inventory born from a CI dream.
Lint sheds old weights and hops more light,
Tests write temp files and sleep at night.
A tiny rabbit cheers the build's new beam.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the primary change: moving parity inventory generation from normal PR linting to a dedicated CI report workflow.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch fix/e2e-parity-report-workflow

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 ESLint

If the error stems from missing dependencies, add them to the package.json file. For unrecoverable errors (e.g., due to private dependencies), disable the tool in the CodeRabbit configuration.

ESLint skipped: no ESLint configuration detected in root package.json. To enable, add eslint to devDependencies.

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

github-actions · 2026-05-19T15:48:16Z

E2E Advisor Recommendation

Required E2E: None
Optional E2E: e2e-parity-compare/compare

Workflow run

Full advisor summary

E2E Recommendation Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E

None. No required E2E is recommended because the changes are limited to E2E CI/parity tooling, generated inventory handling, documentation, and scenario-framework tests. They do not change installer/onboarding, sandbox lifecycle, credentials, security boundaries, network policy, inference routing, deployment logic, or real assistant user flows.

Optional E2E

e2e-parity-compare/compare (low): Optional confidence check for the modified manual parity workflow: dispatch E2E / Parity Compare with empty legacy_script and scenario to exercise inventory generation, strict parity-map validation, empty compare, coverage report rendering, and artifact upload without running a live user scenario.

New E2E recommendations

e2e-ci-parity-harness (medium): The parity workflow is manual-only, so workflow-level regressions in inventory generation or artifact handling may not be caught automatically on PRs that edit the harness.
- Suggested test: Add a PR-safe parity-harness smoke check that runs extract-legacy-assertions, check-parity-map --strict, compare-parity with empty logs, and coverage-report without launching a sandbox.

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/e2e/scenario-framework-tests/e2e-coverage-report.test.ts`:
- Around line 61-114: The test creates a temporary directory in the variable
sourceDir and never removes it; wrap the test body that calls
loadMetadataFromObjects and renderCoverageReport in a try/finally so sourceDir
is removed in the finally block (use fs.rmSync(sourceDir, { recursive: true,
force: true }) or equivalent) to ensure the temp dir is cleaned up even on
assertion failures.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: fb42aeba-a062-4677-846e-ce2b1d4458c2

📥 Commits

Reviewing files that changed from the base of the PR and between 43732ab and f80884b.

📒 Files selected for processing (1)

test/e2e/scenario-framework-tests/e2e-coverage-report.test.ts

ci(e2e): move parity inventory to report workflow

43732ab

test(e2e): decouple coverage parity fixture

f80884b

cv approved these changes May 19, 2026

View reviewed changes

coderabbitai Bot reviewed May 19, 2026

View reviewed changes

Comment thread test/e2e/scenario-framework-tests/e2e-coverage-report.test.ts

jyaunches added 2 commits May 19, 2026 12:03

style(e2e): apply convention lint formatting

48330b0

test(e2e): clean parity fixture temp dir

3cfdb73

jyaunches merged commit 6038e8b into main May 19, 2026
28 checks passed

coderabbitai Bot mentioned this pull request May 19, 2026

fix(openclaw): bump to 2026.4.25 for TUI correlation #3803

Closed

cjagwani mentioned this pull request May 19, 2026

fix: align OpenClaw pairing state root #3737

Merged

wscurran added the chore Build, CI, dependency, or tooling maintenance label Jun 8, 2026

jyaunches deleted the fix/e2e-parity-report-workflow branch June 12, 2026 13:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci(e2e): move parity inventory to report workflow#3800

ci(e2e): move parity inventory to report workflow#3800
jyaunches merged 4 commits into
mainfrom
fix/e2e-parity-report-workflow

jyaunches commented May 19, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented May 19, 2026 •

edited

Loading

Walkthrough

Changes

Sequence Diagram(s)

Estimated code review effort

Possibly related issues

Possibly related PRs

Suggested labels

Suggested reviewers

Poem

❌ Failed checks (1 warning)

Uh oh!

github-actions Bot commented May 19, 2026 •

edited

Loading

E2E Recommendation Advisor

Required E2E

Optional E2E

New E2E recommendations

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

jyaunches commented May 19, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Rationale

Validation

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented May 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

Estimated code review effort

Possibly related issues

Possibly related PRs

Suggested labels

Suggested reviewers

Poem

❌ Failed checks (1 warning)

Uh oh!

github-actions Bot commented May 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

E2E Advisor Recommendation

E2E Recommendation Advisor

Required E2E

Optional E2E

New E2E recommendations

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jyaunches commented May 19, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented May 19, 2026 •

edited

Loading

github-actions Bot commented May 19, 2026 •

edited

Loading