test(e2e): discover live scenarios from registry by cv · Pull Request #5005 · NVIDIA/NemoClaw

cv · 2026-06-09T01:03:33Z

Summary

Adds registry-driven live Vitest scenario discovery so typed scenarios automatically appear as live tests. Supported scenarios run through environment, onboarding, and state-validation fixtures; unsupported scenarios are registered as skipped tests with explicit wiring reasons.

Related Issue

Refs #4941
Refs #4990

Changes

Adds liveScenarioSupport(...) metadata for currently fixture-supported live scenarios.
Adds test/e2e-scenario/live/registry-scenarios.test.ts to generate one live Vitest test per typed registry scenario.
Records pending runtime suites in live artifacts while runtime-suite migration remains future work.
Adds framework tests for supported, unsupported, and no-Docker runtime support classification.

Type of Change

Code change (feature, bug fix, or refactor)
Code change with doc updates
Doc only (prose changes, no code sample modifications)
Doc only (includes code sample changes)

Verification

npx prek run --all-files passes
npm test passes
Tests added or updated for new or changed behavior
No secrets, API keys, or credentials committed
Docs updated for user-facing behavior changes
npm run docs builds without warnings (doc changes only)
Doc pages follow the style guide (doc changes only)
New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: Carlos Villela cvillela@nvidia.com

Summary by CodeRabbit

Tests
- Added new end-to-end test suite for validating live registry discovery support across scenarios
- Added e2e tests for running supported scenarios with environment validation and artifact collection
- Added runtime support gating for scenario compatibility checks

coderabbitai · 2026-06-09T01:03:40Z

📝 Walkthrough

Walkthrough

This PR introduces infrastructure for live Vitest registry discovery by adding a runtime support gating module, dynamic test registration for supported scenarios, and validation tests that ensure proper support classification across all e2e scenarios.

Changes

Live e2e Discovery Test Infrastructure

Layer / File(s)	Summary
Runtime support classification contract `test/e2e-scenario/scenarios/runtime-support.ts`	`LiveScenarioSupport` interface and `liveScenarioSupport()` function validate scenarios against allowed platform, install, runtime, and onboarding combinations, returning supported flag, reasons list, and pending runtime suites.
Dynamic live scenario test registration and execution `test/e2e-scenario/live/registry-scenarios.test.ts`	Iterates through all registry scenarios, conditionally registers Vitest tests only for supported scenarios, and executes per-scenario tests that validate CLI presence, scenario metadata, environment readiness, instance onboarding, state validation, and artifact output.
Support classification discovery validation `test/e2e-scenario/framework-tests/e2e-live-registry-discovery.test.ts`	Validates all scenarios have proper support classification, asserts concrete reasons for specific unsupported scenarios, and verifies pending runtime suites for supported scenarios.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Suggested labels

area: e2e

Suggested reviewers

jyaunches

Poem

A registry discovery dance, so bright,
Live scenarios waltz through the test night,
Support gates and suites align just right,
Onboarded instances reach their state light,
Hop, hop—the e2e pathways ignite! 🐰✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely summarizes the main change: adding discovery of live scenarios from the registry for e2e testing.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch codex/e2e-4990-stack-04-live-registry-discovery

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

github-actions · 2026-06-09T01:04:08Z

PR Review Advisor

Findings: 0 needs attention, 2 worth checking, 0 nice ideas
Since last review: 1 prior item resolved, 0 still apply, 1 new item found

Review findings

🛠️ Needs attention

None.

🔎 Worth checking

Source-of-truth review needed: test/e2e-scenario/scenarios/runtime-support.ts liveScenarioSupport allowlist: The advisor marked localized patch analysis as needs_followup.
- Recommendation: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
- Evidence: runtime-support.ts contains local SUPPORTED_* sets and lifecycle/expectedState checks; registry-scenarios.test.ts consumes that classification to decide live versus skipped tests.
Document the temporary live-support allowlist removal path (test/e2e-scenario/scenarios/runtime-support.ts:6): The new liveScenarioSupport allowlist is a reasonable bridge while Vitest fixtures only support a subset of typed scenarios, but it is now another place that decides which registry scenarios are executable versus skipped. The tests lock a few classifications, but the helper itself does not state how it relates to the Adopt Vitest fixtures as the E2E scenario execution model #4941 single-runner migration, what exact unsupported state is being bridged, or when this allowlist should be removed or replaced.
- Recommendation: Add a short comment near the support sets, or a focused contract test, that describes this as a temporary Vitest-fixture wiring gate, points to the source registry/fixture boundary, and states the retirement condition: remove or collapse the allowlist once all typed scenario dimensions have first-class live fixture support.
- Evidence: runtime-support.ts introduces SUPPORTED_PLATFORMS/SUPPORTED_INSTALLS/SUPPORTED_RUNTIMES/SUPPORTED_ONBOARDING and returns skip reasons for unsupported dimensions; e2e-live-registry-discovery.test.ts verifies the canonical OpenClaw path plus Hermes and docker-missing skips, but no local removal condition is encoded.

🌱 Nice ideas

None.

Consider writing more tests for

**Acceptance clause:** Refs Adopt Vitest fixtures as the E2E scenario execution model #4941 — add test evidence or identify existing coverage. The PR body references Adopt Vitest fixtures as the E2E scenario execution model #4941, but deterministic context did not include the linked issue body or comments, so no literal issue acceptance clauses were available to verify.
**Acceptance clause:** Refs Adopt phase fixtures + registry-driven test discovery for Vitest E2E scenarios #4990 — add test evidence or identify existing coverage. The PR body references Adopt phase fixtures + registry-driven test discovery for Vitest E2E scenarios #4990, but deterministic context did not include the linked issue body or comments, so no literal issue acceptance clauses were available to verify.
**test/e2e-scenario/scenarios/runtime-support.ts liveScenarioSupport allowlist** — e2e-live-registry-discovery.test.ts verifies that every registry scenario is classified, that ubuntu-repo-cloud-openclaw is supported, and that Hermes and docker-missing paths remain skipped with explicit reasons.. runtime-support.ts contains local SUPPORTED_* sets and lifecycle/expectedState checks; registry-scenarios.test.ts consumes that classification to decide live versus skipped tests.

Since last review details

Current findings:

Source-of-truth review needed: test/e2e-scenario/scenarios/runtime-support.ts liveScenarioSupport allowlist: The advisor marked localized patch analysis as needs_followup.
- Recommendation: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
- Evidence: runtime-support.ts contains local SUPPORTED_* sets and lifecycle/expectedState checks; registry-scenarios.test.ts consumes that classification to decide live versus skipped tests.
Document the temporary live-support allowlist removal path (test/e2e-scenario/scenarios/runtime-support.ts:6): The new liveScenarioSupport allowlist is a reasonable bridge while Vitest fixtures only support a subset of typed scenarios, but it is now another place that decides which registry scenarios are executable versus skipped. The tests lock a few classifications, but the helper itself does not state how it relates to the Adopt Vitest fixtures as the E2E scenario execution model #4941 single-runner migration, what exact unsupported state is being bridged, or when this allowlist should be removed or replaced.
- Recommendation: Add a short comment near the support sets, or a focused contract test, that describes this as a temporary Vitest-fixture wiring gate, points to the source registry/fixture boundary, and states the retirement condition: remove or collapse the allowlist once all typed scenario dimensions have first-class live fixture support.
- Evidence: runtime-support.ts introduces SUPPORTED_PLATFORMS/SUPPORTED_INSTALLS/SUPPORTED_RUNTIMES/SUPPORTED_ONBOARDING and returns skip reasons for unsupported dimensions; e2e-live-registry-discovery.test.ts verifies the canonical OpenClaw path plus Hermes and docker-missing skips, but no local removal condition is encoded.

Workflow run details

This is an automated advisory review. A human maintainer must make the final merge decision.

github-actions · 2026-06-09T01:05:23Z

E2E Advisor Recommendation

Required E2E: None
Optional E2E: e2e-vitest-scenarios/live-scenarios, e2e-scenarios/ubuntu-repo-cloud-openclaw

Dispatch hint: test_filter=test/e2e-scenario/live/registry-scenarios.test.ts

Workflow run

Full advisor summary

E2E Recommendation Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E

None. No merge-blocking E2E is required because the PR is limited to E2E test/framework files and cannot directly affect NemoClaw runtime or user flows. Run the optional E2E jobs only to validate the new live scenario harness itself.

Optional E2E

e2e-vitest-scenarios/live-scenarios (medium): Validates the newly added Vitest live scenario file and runtime-support classification path without making it merge-blocking for a tests-only PR.
e2e-scenarios/ubuntu-repo-cloud-openclaw (high): Optional parity check for the canonical supported scenario that runtime-support.ts marks live-supported and routes through onboarding/state validation.

New E2E recommendations

None.

Dispatch hint

Workflow: .github/workflows/e2e-vitest-scenarios.yaml
jobs input: test_filter=test/e2e-scenario/live/registry-scenarios.test.ts

github-actions · 2026-06-09T01:05:24Z

E2E Scenario Advisor Recommendation

Required scenario E2E: ubuntu-repo-cloud-openclaw
Optional scenario E2E: None

Dispatch required scenario E2E:

gh workflow run e2e-scenarios.yaml --ref <pr-head-ref> --field scenarios=ubuntu-repo-cloud-openclaw

Workflow run

Full scenario advisor summary

E2E Scenario Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required scenario E2E

ubuntu-repo-cloud-openclaw: The new live registry support explicitly wires only the Ubuntu repo-current Docker-running cloud OpenClaw path and adds live registry coverage around that supported scenario; run the canonical scenario that exercises this supported surface.
- Dispatch: gh workflow run e2e-scenarios.yaml --ref <pr-head-ref> --field scenarios=ubuntu-repo-cloud-openclaw

Optional scenario E2E

None.

Relevant changed files

test/e2e-scenario/framework-tests/e2e-live-registry-discovery.test.ts
test/e2e-scenario/live/registry-scenarios.test.ts
test/e2e-scenario/scenarios/runtime-support.ts

Stack PRs 4941/4990 stack-01/02/03 were squash-merged into main as #5003, #5004, #5035; merging main resolves the resulting add/add and content conflicts by taking main's versions for already-merged files. Only the 3 new files from 93fb874 remain: runtime-support.ts, registry-scenarios.test.ts, and e2e-live-registry-discovery.test.ts. Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>

jyaunches · 2026-06-09T14:46:48Z

@coderabbitai review

coderabbitai · 2026-06-09T14:46:53Z

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/e2e-scenario/live/registry-scenarios.test.ts`:
- Around line 46-50: The artifact currently records the resolved validation
state (validation.state.id) as expectedStateId; change the payload passed to
artifacts.writeJson so expectedStateId uses scenario.expectedStateId instead,
and (optionally) add an actualStateId field set to validation.state.id to keep
both expected and resolved states. Update the object passed in the call to
artifacts.writeJson (the block that includes scenario.id, expectedStateId,
probes, pendingRuntimeSuites) to replace validation.state.id with
scenario.expectedStateId and add actualStateId: validation.state.id if you want
to preserve the resolved value.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: b85c34a1-aaf3-460d-b1ad-bfb19fcfa6c4

📥 Commits

Reviewing files that changed from the base of the PR and between 44bc078 and 3234d37.

📒 Files selected for processing (3)

test/e2e-scenario/framework-tests/e2e-live-registry-discovery.test.ts
test/e2e-scenario/live/registry-scenarios.test.ts
test/e2e-scenario/scenarios/runtime-support.ts

coderabbitai · 2026-06-09T14:51:36Z

+    await artifacts.writeJson("scenario-result.json", {
+      id: scenario.id,
+      expectedStateId: validation.state.id,
+      probes: validation.probes.map((probe) => probe.id),
+      pendingRuntimeSuites: support.pendingRuntimeSuites,


⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Persist the expected state from the scenario contract, not the resolved validation state.

scenario-result.json currently records expectedStateId from validation.state.id (actual resolved state). This makes artifact metadata misleading. Store scenario.expectedStateId as expected, and optionally add a separate actualStateId.

Suggested fix

await artifacts.writeJson("scenario-result.json", { id: scenario.id, - expectedStateId: validation.state.id, + expectedStateId: scenario.expectedStateId, + actualStateId: validation.state.id, probes: validation.probes.map((probe) => probe.id), pendingRuntimeSuites: support.pendingRuntimeSuites, });

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

await artifacts.writeJson("scenario-result.json", {

id: scenario.id,

expectedStateId: validation.state.id,

probes: validation.probes.map((probe) => probe.id),

pendingRuntimeSuites: support.pendingRuntimeSuites,

await artifacts.writeJson("scenario-result.json", {

id: scenario.id,

expectedStateId: scenario.expectedStateId,

actualStateId: validation.state.id,

probes: validation.probes.map((probe) => probe.id),

pendingRuntimeSuites: support.pendingRuntimeSuites,

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@test/e2e-scenario/live/registry-scenarios.test.ts` around lines 46 - 50, The artifact currently records the resolved validation state (validation.state.id) as expectedStateId; change the payload passed to artifacts.writeJson so expectedStateId uses scenario.expectedStateId instead, and (optionally) add an actualStateId field set to validation.state.id to keep both expected and resolved states. Update the object passed in the call to artifacts.writeJson (the block that includes scenario.id, expectedStateId, probes, pendingRuntimeSuites) to replace validation.state.id with scenario.expectedStateId and add actualStateId: validation.state.id if you want to preserve the resolved value.

Resolves divergence with parent stack PRs (#5002, #5003, #5004, #5005) already merged into main, plus #5022 and #5035 refinements. Conflict resolution prefers main's canonical phase-fixture and framework-plumbing files (those have already received post-#5005 refactors), and re-applies this PR's unique deltas on top: - liveScenarioTestName() helper in runtime-support.ts - Skip-name realignment + module-load [not wired] log in registry-scenarios.test.ts - New e2e-live-skip-name-contract framework test Net effect of this PR vs main is the matrix fan-out workflow, --emit-live-matrix CLI, workflow boundary updates, and the filter-alignment fix. See `git diff origin/main` for the full delta. Co-authored-by: Julie Yaunches <jyaunches@nvidia.com>

cv added 4 commits June 8, 2026 17:39

test(e2e): add environment phase fixture

32ad96a

test(e2e): add onboarding phase fixture

ada6eee

test(e2e): add state validation phase fixture

82c0543

test(e2e): discover live scenarios from registry

93fb874

cv self-assigned this Jun 9, 2026

cv mentioned this pull request Jun 9, 2026

Adopt Vitest fixtures as the E2E scenario execution model #4941

Open

7 tasks

wscurran added the chore Build, CI, dependency, or tooling maintenance label Jun 9, 2026

Base automatically changed from codex/e2e-4990-stack-03-state-validation-phase to main June 9, 2026 13:35

coderabbitai Bot reviewed Jun 9, 2026

View reviewed changes

jyaunches approved these changes Jun 9, 2026

View reviewed changes

jyaunches merged commit fc9a123 into main Jun 9, 2026
44 checks passed

jyaunches deleted the codex/e2e-4990-stack-04-live-registry-discovery branch June 9, 2026 14:51

This was referenced Jun 9, 2026

ci(e2e): fan out Vitest scenarios from registry #5006

Merged

test(e2e): add post-reboot-recovery regression guard for #4423 #5046

Merged

cv added the v0.0.62 Release target label Jun 9, 2026

coderabbitai Bot mentioned this pull request Jun 10, 2026

test(e2e): migrate cloud inference and routing scenarios #5057

Closed

12 tasks

jyaunches mentioned this pull request Jun 10, 2026

Adopt phase fixtures + registry-driven test discovery for Vitest E2E scenarios #4990

Closed

8 tasks

Conversation

cv commented Jun 9, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Related Issue

Changes

Type of Change

Verification

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Suggested labels

Suggested reviewers

Poem

❌ Failed checks (1 warning)

Uh oh!

github-actions Bot commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Review Advisor

🛠️ Needs attention

🔎 Worth checking

🌱 Nice ideas

Uh oh!

github-actions Bot commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

E2E Advisor Recommendation

E2E Recommendation Advisor

Required E2E

Optional E2E

New E2E recommendations

Dispatch hint

Uh oh!

github-actions Bot commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

E2E Scenario Advisor Recommendation

E2E Scenario Advisor

Required scenario E2E

Optional scenario E2E

Relevant changed files

Uh oh!

jyaunches commented Jun 9, 2026

Uh oh!

coderabbitai Bot commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

cv commented Jun 9, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jun 9, 2026 •

edited

Loading

github-actions Bot commented Jun 9, 2026 •

edited

Loading

github-actions Bot commented Jun 9, 2026 •

edited

Loading

github-actions Bot commented Jun 9, 2026 •

edited

Loading

coderabbitai Bot commented Jun 9, 2026 •

edited

Loading