ci(e2e): add Vitest scenario workflow by cv · Pull Request #4968 · NVIDIA/NemoClaw

cv · 2026-06-08T16:42:23Z

Summary

Adds a manual GitHub Actions entry point for running the opt-in Vitest live scenario project and collecting fixture artifacts. This is stack PR 6/7 for #4941. cc @jyaunches for review.

Related Issue

Refs #4941.

Changes

Adds .github/workflows/e2e-vitest-scenarios.yaml for manually dispatched live scenario runs.
Sets NEMOCLAW_RUN_E2E_SCENARIOS=1 in the workflow and supports an optional test filter.
Builds the CLI before invoking the live Vitest scenario project.
Passes manual dispatch input through environment variables before shell use.
Uploads .e2e/vitest/ artifacts with hidden-file collection enabled.
Adds workflow-boundary coverage for the new Vitest scenario workflow.
Updates the scenario README with the CI entry point.

Type of Change

Code change (feature, bug fix, or refactor)
Code change with doc updates
Doc only (prose changes, no code sample modifications)
Doc only (includes code sample changes)

Verification

Signed-off-by: Carlos Villela cvillela@nvidia.com

Signed-off-by: Carlos Villela <cvillela@nvidia.com>

copy-pr-bot · 2026-06-08T16:42:26Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

coderabbitai · 2026-06-08T16:42:30Z

📝 Walkthrough

Walkthrough

This PR introduces a new opt-in GitHub Actions workflow for running Vitest E2E scenarios on manual dispatch, complete with validation infrastructure, test coverage, and documentation. The workflow runs filtered Vitest tests, summarizes artifacts, and uploads results.

Changes

Vitest E2E Workflow and Validation

Layer / File(s)	Summary
Workflow boundary validation infrastructure `tools/e2e-scenarios/workflow-boundary.mts`	Defines `DEFAULT_VITEST_WORKFLOW_PATH` constant and introduces reusable helper validators (`validateHasWorkflowDispatch`, `rejectAutomaticTriggers`, `requirePinnedFullShaActions`, `rejectDirectDispatchInputInterpolation`) for enforcing workflow contract boundaries.
Refactored existing workflow validator `tools/e2e-scenarios/workflow-boundary.mts`	Updates `validateE2eScenariosWorkflowBoundary` to use the new trigger-validation helpers, removing inline checks for `workflow_dispatch` presence and disallowed automatic triggers while preserving existing job/step validations.
New Vitest workflow validator `tools/e2e-scenarios/workflow-boundary.mts`	Adds exported `validateE2eVitestScenariosWorkflowBoundary` function that validates dispatch inputs, required permissions, job/environment configuration, safe filter propagation, pinned action SHAs, vitest command structure, and strict artifact upload settings including hidden-file inclusion.
Vitest E2E scenarios workflow definition `.github/workflows/e2e-vitest-scenarios.yaml`	Defines new `E2E / Vitest Scenarios` workflow: accepts optional `test_filter` input, sets up Node.js 22 with npm caching, runs `npx vitest run` with conditional filtering, appends artifact summary to step summary, and uploads `.e2e/vitest/` with hidden files included and missing-artifact handling.
Workflow boundary test suite `test/e2e-scenario/framework-tests/e2e-scenarios-workflow.test.ts`	Imports new validator and adds Vitest workflow boundary test that constructs a temporary unsafe workflow, calls `validateE2eVitestScenariosWorkflowBoundary`, and asserts specific contract-violation errors for unpinned actions, unsafe dispatch-input patterns, and missing artifact configuration.
E2E scenario framework documentation `test/e2e-scenario/docs/README.md`	Updates CI entry points section to document the new opt-in Vitest live scenario workflow, noting its upload of `.e2e/vitest/` fixture artifacts.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Suggested labels

area: ci

Suggested reviewers

jyaunches
prekshivyas
ericksoa

Poem

🐰 A workflow to run tests live and bright,
With validation that guards us each night,
Vitest scenarios uploaded with care,
Dispatch-driven checks, boundary-fair,
E2E fixtures now flow through the air! ✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The PR title 'ci(e2e): add Vitest scenario workflow' directly and clearly describes the main change: adding a new Vitest scenario workflow to the CI/CD pipeline.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch codex/e2e-4941-stack-06-workflow

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

github-actions · 2026-06-08T16:44:15Z

E2E Advisor Recommendation

Required E2E: e2e-vitest-scenarios/live-scenarios
Optional E2E: e2e-scenarios/ubuntu-repo-cloud-openclaw

Dispatch hint: test_filter=test/e2e-scenario/live/ubuntu-repo-cli-smoke.test.ts

Workflow run

Full advisor summary

E2E Recommendation Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E

e2e-vitest-scenarios/live-scenarios (low-medium; Ubuntu hosted runner, timeout 20 minutes): Directly validates the newly added E2E / Vitest Scenarios workflow path, including CLI build, opt-in live project gate, test_filter handling, and artifact upload behavior. Use the existing live repo CLI smoke test as the lowest-cost canary.

Optional E2E

e2e-scenarios/ubuntu-repo-cloud-openclaw (medium-high; live install/onboarding/sandbox/inference scenario): Optional confidence check that the established typed scenario runner still works while the workflow-boundary validator is being refactored nearby. Not merge-blocking because the PR does not change product installer/onboarding/sandbox/inference code or the existing scenario runner workflow.

New E2E recommendations

e2e-vitest-workflow-pr-canary (medium): The new Vitest live scenario workflow is manual-only, so future changes to this workflow may not be automatically exercised on PRs. Consider a reusable or maintainer-dispatched canary that runs the repo CLI smoke filter and verifies artifacts are captured safely.
- Suggested test: Add a PR-safe/reusable canary for .github/workflows/e2e-vitest-scenarios.yaml that runs test/e2e-scenario/live/ubuntu-repo-cli-smoke.test.ts and checks the .e2e/vitest artifact contract.

Dispatch hint

Workflow: .github/workflows/e2e-vitest-scenarios.yaml
jobs input: test_filter=test/e2e-scenario/live/ubuntu-repo-cli-smoke.test.ts

github-actions · 2026-06-08T16:44:16Z

E2E Scenario Advisor Recommendation

Required scenario E2E: None
Optional scenario E2E: None

Workflow run

Full scenario advisor summary

E2E Scenario Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required scenario E2E

None. No dispatchable scenario E2E run is required: the PR adds/validates a separate Vitest live scenario workflow and updates scenario docs/framework boundary tests/tooling, but does not change the scenario runner/runtime, scenario catalog metadata, expected-state contracts, suite catalog/scripts, onboarding/install helpers, or the dispatchable e2e-scenarios.yaml/e2e-scenarios-all.yaml workflows.

Optional scenario E2E

None.

Relevant changed files

.github/workflows/e2e-vitest-scenarios.yaml
test/e2e-scenario/docs/README.md
test/e2e-scenario/framework-tests/e2e-scenarios-workflow.test.ts
tools/e2e-scenarios/workflow-boundary.mts

github-actions · 2026-06-08T16:45:51Z

PR Review Advisor

Findings: 0 needs attention, 0 worth checking, 1 nice ideas
Since last review: 1 prior item resolved, 0 still apply, 1 new item found

Review findings

🛠️ Needs attention

None.

🔎 Worth checking

None.

🌱 Nice ideas

Guard Build CLI ordering before Vitest (tools/e2e-scenarios/workflow-boundary.mts:254): The real workflow now runs `Build CLI` before `Run Vitest live E2E scenarios`, which addresses the prior failure path. The boundary validator, however, only checks that both named steps exist and contain expected commands; it would still pass if a future edit moved `Build CLI` after the Vitest run.
- Recommendation: Add an ordering assertion in `validateE2eVitestScenariosWorkflowBoundary()` and a negative fixture where `Run Vitest live E2E scenarios` precedes `Build CLI`, expecting an error that the CLI build must run before the live Vitest scenario step.
- Evidence: .github/workflows/e2e-vitest-scenarios.yaml currently orders `Install root dependencies` -> `Build CLI` -> `Run Vitest live E2E scenarios`. In tools/e2e-scenarios/workflow-boundary.mts, the validator calls `requireStep(errors, steps, "Build CLI")` and then `requireStep(errors, steps, "Run Vitest live E2E scenarios")`, but does not compare their positions.

Consider writing more tests for

**Guard Build CLI ordering before Vitest** — Add an ordering assertion in `validateE2eVitestScenariosWorkflowBoundary()` and a negative fixture where `Run Vitest live E2E scenarios` precedes `Build CLI`, expecting an error that the CLI build must run before the live Vitest scenario step.
**Acceptance clause:** Refs Adopt Vitest fixtures as the E2E scenario execution model #4941. — add test evidence or identify existing coverage. The deterministic GitHub context reported `linkedIssues: []`, so the literal Adopt Vitest fixtures as the E2E scenario execution model #4941 body and comments were not available to verify any issue-level acceptance clauses.

Since last review details

Current findings:

Guard Build CLI ordering before Vitest (tools/e2e-scenarios/workflow-boundary.mts:254): The real workflow now runs `Build CLI` before `Run Vitest live E2E scenarios`, which addresses the prior failure path. The boundary validator, however, only checks that both named steps exist and contain expected commands; it would still pass if a future edit moved `Build CLI` after the Vitest run.
- Recommendation: Add an ordering assertion in `validateE2eVitestScenariosWorkflowBoundary()` and a negative fixture where `Run Vitest live E2E scenarios` precedes `Build CLI`, expecting an error that the CLI build must run before the live Vitest scenario step.
- Evidence: .github/workflows/e2e-vitest-scenarios.yaml currently orders `Install root dependencies` -> `Build CLI` -> `Run Vitest live E2E scenarios`. In tools/e2e-scenarios/workflow-boundary.mts, the validator calls `requireStep(errors, steps, "Build CLI")` and then `requireStep(errors, steps, "Run Vitest live E2E scenarios")`, but does not compare their positions.

Workflow run details

This is an automated advisory review. A human maintainer must make the final merge decision.

…06-workflow

coderabbitai

🧹 Nitpick comments (1)

tools/e2e-scenarios/workflow-boundary.mts (1)
84-96: 💤 Low value

Consider adding bracket notation detection for dispatch input interpolation.

The regex pattern /\$\{\{\s*(inputs|github\.event\.inputs)\./ catches common dot-notation interpolation like ${{ inputs.test_filter }}, but would miss bracket notation like ${{ inputs['test_filter'] }}. While bracket notation is less common in GitHub Actions workflows, adding support would make the validation more comprehensive.
🔍 Enhanced pattern to also catch bracket notation
 function requireNoDispatchInputInterpolation(
   errors: string[],
   steps: readonly WorkflowStep[],
 ): void {
-  const expressionPattern = /\$\{\{\s*(inputs|github\.event\.inputs)\./;
+  const expressionPattern = /\$\{\{\s*(inputs(\[|\.)|github\.event\.inputs(\[|\.))/;
   for (const step of steps) {
     if (expressionPattern.test(stringValue(step.run))) {
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tools/e2e-scenarios/workflow-boundary.mts` around lines 84 - 96, The current
regex in requireNoDispatchInputInterpolation (expressionPattern) only matches
dot-notation like `${{ inputs.foo }}`; update expressionPattern to also detect
bracket-notation such as `${{ inputs['foo'] }}` or `${{
github.event.inputs["foo"] }}` by extending the pattern to allow either `.\w+`
or `['"...']` access after the `inputs` or `github.event.inputs` token (e.g. add
a non-capturing group like `(?:\.\w+|\[['"]\w+['"]\])` to the existing pattern)
so stringValue(step.run) will be flagged for both styles.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@tools/e2e-scenarios/workflow-boundary.mts`:
- Around line 84-96: The current regex in requireNoDispatchInputInterpolation
(expressionPattern) only matches dot-notation like `${{ inputs.foo }}`; update
expressionPattern to also detect bracket-notation such as `${{ inputs['foo'] }}`
or `${{ github.event.inputs["foo"] }}` by extending the pattern to allow either
`.\w+` or `['"...']` access after the `inputs` or `github.event.inputs` token
(e.g. add a non-capturing group like `(?:\.\w+|\[['"]\w+['"]\])` to the existing
pattern) so stringValue(step.run) will be flagged for both styles.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: d5b4445a-563d-40fc-8c51-a20d33eccd1b

📥 Commits

Reviewing files that changed from the base of the PR and between 8f4fa7e and 6bf481d.

📒 Files selected for processing (4)

.github/workflows/e2e-vitest-scenarios.yaml
test/e2e-scenario/docs/README.md
test/e2e-scenario/framework-tests/e2e-scenarios-workflow.test.ts
tools/e2e-scenarios/workflow-boundary.mts

cv · 2026-06-08T23:57:34Z

E2E advisor requested e2e-vitest-scenarios/live-scenarios. I attempted to dispatch the new workflow against this PR branch with:\n\nbash\ngh workflow run e2e-vitest-scenarios.yaml --ref codex/e2e-4941-stack-06-workflow --field test_filter=test/e2e-scenario/live/ubuntu-repo-cli-smoke.test.ts\n\n\nGitHub returned HTTP 404: workflow e2e-vitest-scenarios.yaml not found on the default branch, which is expected for a brand-new workflow_dispatch file before it lands on main. I addressed the advisor's build-contract finding by adding an explicit npm run build:cli step before the live Vitest run and a workflow-boundary assertion that preserves that contract.

cv added 6 commits June 8, 2026 09:34

docs(e2e): choose Vitest fixtures as scenario runner

3e98b63

Signed-off-by: Carlos Villela <cvillela@nvidia.com>

test(e2e): add opt-in live scenario project

9e355f0

Signed-off-by: Carlos Villela <cvillela@nvidia.com>

test(e2e): add scenario fixture context

a11bc2a

Signed-off-by: Carlos Villela <cvillela@nvidia.com>

test(e2e): add fixture-friendly clients

d92715c

Signed-off-by: Carlos Villela <cvillela@nvidia.com>

test(e2e): add first Vitest live smoke scenario

bb10adf

Signed-off-by: Carlos Villela <cvillela@nvidia.com>

ci(e2e): add Vitest scenario workflow

8d0686b

Signed-off-by: Carlos Villela <cvillela@nvidia.com>

cv added the area: e2e End-to-end tests, nightly failures, or validation infrastructure label Jun 8, 2026

cv self-assigned this Jun 8, 2026

cv requested a review from jyaunches June 8, 2026 16:42

cv mentioned this pull request Jun 8, 2026

Adopt Vitest fixtures as the E2E scenario execution model #4941

Open

7 tasks

cv added the v0.0.62 Release target label Jun 8, 2026

jyaunches mentioned this pull request Jun 8, 2026

Adopt phase fixtures + registry-driven test discovery for Vitest E2E scenarios #4990

Closed

8 tasks

wscurran added the chore Build, CI, dependency, or tooling maintenance label Jun 8, 2026

Base automatically changed from codex/e2e-4941-stack-05-live-smoke to main June 8, 2026 23:32

cv added 2 commits June 8, 2026 16:35

Merge remote-tracking branch 'origin/main' into codex/e2e-4941-stack-…

957c49c

…06-workflow

ci(e2e): harden Vitest scenario workflow

6bf481d

cv marked this pull request as ready for review June 8, 2026 23:43

coderabbitai Bot reviewed Jun 8, 2026

View reviewed changes

ci(e2e): build CLI before Vitest scenarios

df09480

jyaunches approved these changes Jun 9, 2026

View reviewed changes

cv merged commit b65a7e0 into main Jun 9, 2026
43 checks passed

cv deleted the codex/e2e-4941-stack-06-workflow branch June 9, 2026 00:07

jyaunches mentioned this pull request Jun 9, 2026

Vitest E2E coverage: assert NemoClaw's high-value behavioral catalog #4999

Open

coderabbitai Bot mentioned this pull request Jun 9, 2026

ci(e2e): fan out Vitest scenarios from registry #5006

Merged

12 tasks

coderabbitai Bot mentioned this pull request Jun 9, 2026

test(e2e): failing-test-first guard for #2701 + recovery framework helpers #5049

Merged

9 tasks

Conversation

cv commented Jun 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Related Issue

Changes

Type of Change

Verification

Uh oh!

copy-pr-bot Bot commented Jun 8, 2026

Uh oh!

coderabbitai Bot commented Jun 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Suggested labels

Suggested reviewers

Poem

❌ Failed checks (1 warning)

Uh oh!

github-actions Bot commented Jun 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

E2E Advisor Recommendation

E2E Recommendation Advisor

Required E2E

Optional E2E

New E2E recommendations

Dispatch hint

Uh oh!

github-actions Bot commented Jun 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

E2E Scenario Advisor Recommendation

E2E Scenario Advisor

Required scenario E2E

Optional scenario E2E

Relevant changed files

Uh oh!

github-actions Bot commented Jun 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Review Advisor

🛠️ Needs attention

🔎 Worth checking

🌱 Nice ideas

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cv commented Jun 8, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

cv commented Jun 8, 2026 •

edited

Loading

coderabbitai Bot commented Jun 8, 2026 •

edited

Loading

github-actions Bot commented Jun 8, 2026 •

edited

Loading

github-actions Bot commented Jun 8, 2026 •

edited

Loading

github-actions Bot commented Jun 8, 2026 •

edited

Loading