test(e2e): guard scenario workflow drift by cv · Pull Request #4655 · NVIDIA/NemoClaw

cv · 2026-06-02T07:35:05Z

Summary

Adds drift guards so typed scenario IDs stay aligned with the scenario workflow route table. Also updates the scenario coverage report source-path text to the current test/e2e-scenario layout.

Related Issue

Refs #3588.

Changes

Add a framework test that verifies every typed scenario registry ID has an .github/workflows/e2e-scenarios.yaml runner route and that the route table has no unknown IDs.
Add missing workflow routes for current typed scenarios that were not routed.
Update the coverage report source-path description and cover it in e2e-coverage-report.test.ts.

Type of Change

Code change (feature, bug fix, or refactor)
Code change with doc updates
Doc only (prose changes, no code sample modifications)
Doc only (includes code sample changes)

Verification

npx prek run --all-files passes
npm test passes
Tests added or updated for new or changed behavior
No secrets, API keys, or credentials committed
Docs updated for user-facing behavior changes
npm run docs builds without warnings (doc changes only)
Doc pages follow the style guide (doc changes only)
New doc pages include SPDX header and frontmatter (new pages only)

Additional verification run:

npx vitest run --project e2e-scenario-framework test/e2e-scenario/framework-tests/e2e-scenarios-workflow.test.ts test/e2e-scenario/framework-tests/e2e-coverage-report.test.ts --silent=false --reporter=default
npx prek run --files .github/workflows/e2e-scenarios.yaml test/e2e-scenario/framework-tests/e2e-scenarios-workflow.test.ts test/e2e-scenario/framework-tests/e2e-coverage-report.test.ts test/e2e-scenario/runtime/resolver/coverage.ts

Signed-off-by: Carlos Villela cvillela@nvidia.com

Summary by CodeRabbit

Tests
- Introduced E2E scenario routing validation to ensure all defined test scenarios are properly mapped in the CI/CD workflow.
- Enhanced coverage report assertions to verify test data source references are correctly documented.
Chores
- Updated CI/CD workflow configuration to expand scenario routing mappings for improved test distribution.

copy-pr-bot · 2026-06-02T07:35:10Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

coderabbitai · 2026-06-02T07:35:13Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: eb54d4da-ea2a-483f-a2a8-3ee5039eb809

📥 Commits

Reviewing files that changed from the base of the PR and between 4696ff7 and 8f51ab5.

📒 Files selected for processing (4)

.github/workflows/e2e-scenarios.yaml
test/e2e-scenario/framework-tests/e2e-coverage-report.test.ts
test/e2e-scenario/framework-tests/e2e-scenarios-workflow.test.ts
test/e2e-scenario/runtime/resolver/coverage.ts

📝 Walkthrough

Walkthrough

This PR updates e2e scenario configuration and validation by adding four new scenario routes to the GitHub Actions workflow, correcting coverage report path references, and introducing a test that validates workflow routing stays synchronized with the scenario registry.

Changes

E2E Scenario Routes and Coverage Alignment

Layer / File(s)	Summary
Coverage report paths and test assertions `test/e2e-scenario/runtime/resolver/coverage.ts`, `test/e2e-scenario/framework-tests/e2e-coverage-report.test.ts`	Coverage report header paths updated to reference `test/e2e-scenario/nemoclaw_scenarios/` and `test/e2e-scenario/validation_suites/`, and coverage test asserts these paths appear in rendered markdown.
Workflow scenario route configuration `.github/workflows/e2e-scenarios.yaml`	Four new scenario IDs—`ubuntu-gateway-port-conflict-negative`, `ubuntu-invalid-nvidia-key-negative`, `ubuntu-no-docker-preflight-negative`, and `ubuntu-repo-cloud-openclaw-custom-policies`—are routed to `ubuntu-latest` in the ROUTES mapping.
Workflow-registry routing consistency validation `test/e2e-scenario/framework-tests/e2e-scenarios-workflow.test.ts`	Helper `routesFromWorkflow()` parses ROUTES from the workflow, and a new test verifies all typed scenario registry IDs are present in the workflow ROUTES mapping with matching runner assignments.

🎯 2 (Simple) | ⏱️ ~10 minutes

Suggested labels

E2E

Suggested reviewers

jyaunches
prekshivyas
cjagwani

Poem

🐰 Four routes now routed west, where ubuntu runners rest,
Coverage paths aligned with where the scenarios are blessed,
A test to keep them true—no drift 'tween workflow and the crew,
E2E consistency hops along, freshly checked and strong! 🌟

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'test(e2e): guard scenario workflow drift' directly and clearly describes the main objective of the pull request—adding guards to detect and prevent drift between typed scenario IDs and the e2e workflow route table.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch test/e2e-scenario-drift-guards

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

github-actions · 2026-06-02T07:36:38Z

E2E Advisor Recommendation

Required E2E: None
Optional E2E: ubuntu-repo-cloud-openclaw-custom-policies, ubuntu-invalid-nvidia-key-negative, ubuntu-gateway-port-conflict-negative

Dispatch hint: ubuntu-gateway-port-conflict-negative,ubuntu-invalid-nvidia-key-negative,ubuntu-repo-cloud-openclaw-custom-policies

Workflow run

Full advisor summary

E2E Recommendation Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E

None.

Optional E2E

ubuntu-repo-cloud-openclaw-custom-policies (medium): Optional confidence check for one newly added workflow route covering custom policy onboarding and security/policy assertions; validates the scenario can be dispatched through the changed E2E scenario workflow on ubuntu-latest.
ubuntu-invalid-nvidia-key-negative (low): Optional confidence check for the newly added invalid NVIDIA key negative-path route; useful to prove the workflow route table accepts the typed scenario ID and selects the expected ubuntu-latest runner.
ubuntu-gateway-port-conflict-negative (low): Optional confidence check for the newly added gateway port-conflict negative-path route; validates dispatch/routing for an onboarding failure scenario affected by the workflow change.

New E2E recommendations

None.

Dispatch hint

Workflow: .github/workflows/e2e-scenarios.yaml
jobs input: ubuntu-gateway-port-conflict-negative,ubuntu-invalid-nvidia-key-negative,ubuntu-repo-cloud-openclaw-custom-policies

github-actions · 2026-06-02T07:36:39Z

E2E Scenario Advisor Recommendation

Required scenario E2E: e2e-scenarios-all
Optional scenario E2E: None

Dispatch required scenario E2E:

gh workflow run e2e-scenarios-all.yaml --ref <pr-head-ref>

Workflow run

Full scenario advisor summary

E2E Scenario Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required scenario E2E

e2e-scenarios-all: The PR changes the scenario runner workflow ROUTES table and shared scenario runtime/resolver coverage code under test/e2e-scenario/runtime; policy requires the full scenario fan-out for workflow and runtime changes.
- Dispatch: gh workflow run e2e-scenarios-all.yaml --ref <pr-head-ref>

Optional scenario E2E

None.

Relevant changed files

.github/workflows/e2e-scenarios.yaml
test/e2e-scenario/framework-tests/e2e-coverage-report.test.ts
test/e2e-scenario/framework-tests/e2e-scenarios-workflow.test.ts
test/e2e-scenario/runtime/resolver/coverage.ts

github-actions · 2026-06-02T07:37:54Z

PR Review Advisor

Findings: 0 needs attention, 0 worth checking, 0 nice ideas
Since last review: 2 prior items resolved, 0 still apply, 0 new items found

Consider writing more tests for

**Runtime validation** — Validate that dispatching ubuntu-gateway-port-conflict-negative selects ubuntu-latest before running the scenario.. The main behavior change touches a GitHub Actions workflow route table. The added framework test covers the principal drift class, but a targeted runtime validation would add confidence that the workflow resolve step emits the expected runner labels when invoked by Actions.
**Runtime validation** — Validate that dispatching ubuntu-invalid-nvidia-key-negative selects ubuntu-latest before running the scenario.. The main behavior change touches a GitHub Actions workflow route table. The added framework test covers the principal drift class, but a targeted runtime validation would add confidence that the workflow resolve step emits the expected runner labels when invoked by Actions.
**Runtime validation** — Validate that dispatching ubuntu-repo-cloud-openclaw-custom-policies selects ubuntu-latest before running the scenario.. The main behavior change touches a GitHub Actions workflow route table. The added framework test covers the principal drift class, but a targeted runtime validation would add confidence that the workflow resolve step emits the expected runner labels when invoked by Actions.
**Runtime validation** — Validate that a mixed batch containing one Ubuntu route and one non-Ubuntu route still fails with the multi-runner split-dispatch error.. The main behavior change touches a GitHub Actions workflow route table. The added framework test covers the principal drift class, but a targeted runtime validation would add confidence that the workflow resolve step emits the expected runner labels when invoked by Actions.

Workflow run details

This is an automated advisory review. A human maintainer must make the final merge decision.

wscurran · 2026-06-03T18:10:45Z

✨
Related open issues:

#3588 Implement layered E2E scenario model

Signed-off-by: Carlos Villela <cvillela@nvidia.com>

cv · 2026-06-07T14:20:36Z

Resolved the merge conflicts by merging current main into the PR branch. Also addressed the PR Review Advisor feedback by extending the e2e-scenarios workflow drift guard to compare each workflow ROUTES runner label against resolveRunnerForScenario(scenario).runner.

cv added 3 commits June 1, 2026 23:42

test(e2e): add scenario aliases for layered plans

dbc672f

docs(e2e): reconcile scenario migration docs

c66d2a3

test(e2e): guard scenario workflow drift

401c631

cv self-assigned this Jun 2, 2026

wscurran added area: e2e End-to-end tests, nightly failures, or validation infrastructure bug-fix PR fixes a bug or regression labels Jun 3, 2026

Base automatically changed from docs/e2e-scenario-hybrid-reconciliation to main June 4, 2026 19:46

merge(main): resolve PR 4655 conflicts

8f51ab5

Signed-off-by: Carlos Villela <cvillela@nvidia.com>

cv marked this pull request as ready for review June 7, 2026 14:20

cv merged commit a25d5b3 into main Jun 7, 2026
40 of 42 checks passed

cv deleted the test/e2e-scenario-drift-guards branch June 7, 2026 14:39

cv added the v0.0.61 Release target label Jun 7, 2026

coderabbitai Bot mentioned this pull request Jun 8, 2026

test(e2e): execute real shell assertions; delete dry-run, --validate-only, and the bash runner #4380

Merged

Conversation

cv commented Jun 2, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Related Issue

Changes

Type of Change

Verification

Summary by CodeRabbit

Uh oh!

copy-pr-bot Bot commented Jun 2, 2026

Uh oh!

coderabbitai Bot commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Suggested labels

Suggested reviewers

Poem

❌ Failed checks (1 warning)

Uh oh!

github-actions Bot commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

E2E Advisor Recommendation

E2E Recommendation Advisor

Required E2E

Optional E2E

New E2E recommendations

Dispatch hint

Uh oh!

github-actions Bot commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

E2E Scenario Advisor Recommendation

E2E Scenario Advisor

Required scenario E2E

Optional scenario E2E

Relevant changed files

Uh oh!

github-actions Bot commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Review Advisor

Uh oh!

wscurran commented Jun 3, 2026

Uh oh!

cv commented Jun 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

cv commented Jun 2, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jun 2, 2026 •

edited

Loading

github-actions Bot commented Jun 2, 2026 •

edited

Loading

github-actions Bot commented Jun 2, 2026 •

edited

Loading

github-actions Bot commented Jun 2, 2026 •

edited

Loading

cv commented Jun 7, 2026 •

edited

Loading