Skip to content

test(e2e): add opt-in live scenario project#4964

Merged
cv merged 5 commits into
mainfrom
codex/e2e-4941-stack-02-live-project
Jun 8, 2026
Merged

test(e2e): add opt-in live scenario project#4964
cv merged 5 commits into
mainfrom
codex/e2e-4941-stack-02-live-project

Conversation

@cv

@cv cv commented Jun 8, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds an opt-in Vitest project for live E2E scenarios so #4941 can use Vitest as the single runner without running live scenarios by default. Also keeps existing gated projects explicit so installer integration and branch validation do not silently run or silently empty out in the Brev workflow path.

Related Issue

Refs #4941.

Changes

  • Adds gate helpers for installer integration, live E2E scenarios, and branch validation Vitest projects.
  • Adds the e2e-scenarios-live Vitest project with conditional include patterns.
  • Updates existing gated project comments to name the required opt-in env vars.
  • Sets NEMOCLAW_RUN_BRANCH_VALIDATION_E2E=1 in the reusable Brev branch-validation workflow's Vitest step.
  • Adds framework tests for gated project defaults, env-gate behavior, and the branch-validation workflow sentinel.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Verification

  • npx prek run --all-files passes
  • npm test passes
  • Tests added or updated for new or changed behavior
  • No secrets, API keys, or credentials committed
  • Docs updated for user-facing behavior changes
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: Carlos Villela cvillela@nvidia.com

cv added 2 commits June 8, 2026 09:34
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
@cv cv added the area: e2e End-to-end tests, nightly failures, or validation infrastructure label Jun 8, 2026
@cv cv self-assigned this Jun 8, 2026
@cv cv requested a review from jyaunches June 8, 2026 16:42
@copy-pr-bot

copy-pr-bot Bot commented Jun 8, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@coderabbitai

coderabbitai Bot commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: ce518170-4af3-4aed-994f-76b8b7ca6887

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/e2e-4941-stack-02-live-project

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions

github-actions Bot commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

E2E Advisor Recommendation

Required E2E: e2e-branch-validation:full
Optional E2E: e2e-scenarios:ubuntu-repo-cloud-openclaw

Workflow run

Full advisor summary

E2E Recommendation Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E

  • e2e-branch-validation:full (Brev CPU instance credits; workflow header estimates about $0.10/run, typically around 10-30 minutes for full suite provisioning plus execution): Required because the PR changes the exact workflow and Vitest gating that enable the Brev branch-validation E2E. Running the existing full branch-validation suite verifies the sentinel is present, the e2e-branch-validation Vitest project includes test/e2e/brev-e2e.test.ts, and the clean Brev source install/onboard/live-inference path still executes instead of silently matching no tests.

Optional E2E

  • e2e-scenarios:ubuntu-repo-cloud-openclaw (Moderate; GitHub-hosted Ubuntu runner with live scenario setup and NVIDIA API secret): Optional adjacent confidence for the live scenario framework path because the PR introduces an e2e-scenarios-live Vitest project gate. The existing typed scenario workflow remains the authoritative live scenario runner, and this representative OpenClaw source-install scenario can confirm scenario dispatch still works, but the PR does not change scenario runtime code so it should not block merge.

New E2E recommendations

  • e2e-scenarios-live-vitest-project (medium): The PR adds a gated e2e-scenarios-live Vitest project pointing at test/e2e-scenario/live/**/*.test.ts, but this checkout does not currently contain that live test directory and no workflow appears to invoke NEMOCLAW_RUN_E2E_SCENARIOS=1 npx vitest run --project e2e-scenarios-live. Consider adding a future workflow-level smoke or contract once live Vitest scenario tests exist so this project cannot remain configured but unexercised.
    • Suggested test: Add a workflow-dispatched e2e-scenarios-live smoke that sets NEMOCLAW_RUN_E2E_SCENARIOS=1 and verifies at least one live scenario test is collected/executed.

@github-actions

github-actions Bot commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

E2E Scenario Advisor Recommendation

Required scenario E2E: None
Optional scenario E2E: None

Workflow run

Full scenario advisor summary

E2E Scenario Advisor

Base: origin/main
Head: HEAD
Confidence: medium

Required scenario E2E

  • None. No scenario E2E dispatch is recommended. The scenario-relevant changes add Vitest gating/config tests for live scenario project selection, but they do not change the scenario runner/runtime used by .github/workflows/e2e-scenarios.yaml, scenario catalog metadata, expected-state contracts, suite catalog metadata, or suite scripts. The branch-validation workflow change belongs to the non-scenario/general E2E surface and is intentionally not considered here.

Optional scenario E2E

  • None.

Relevant changed files

  • test/e2e-scenario/framework-tests/e2e-live-project-config.test.ts
  • test/e2e-scenario/framework/live-project-gate.ts
  • vitest.config.ts

@github-actions

github-actions Bot commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor

Findings: 0 needs attention, 0 worth checking, 0 nice ideas
Since last review: 2 prior items resolved, 0 still apply, 0 new items found

Consider writing more tests for
  • **Runtime validation** — Import `vitest.config.ts` with no CI or opt-in env and verify `installer-integration`, `e2e-scenarios-live`, and `e2e-branch-validation` include arrays are all empty.. The helper-level and workflow-contract coverage is adequate for the PR, and the prior CI-dependent test failure is resolved. Because `vitest.config.ts` computes include arrays at import time from `process.env`, controlled config-import permutations would improve confidence for this workflow/test-runner infrastructure.
  • **Runtime validation** — Import `vitest.config.ts` with `CI=true` and verify `installer-integration` includes `test/install-preflight.test.ts` and `test/install-openshell-version-check.test.ts` while live and branch-validation projects remain empty absent their gates.. The helper-level and workflow-contract coverage is adequate for the PR, and the prior CI-dependent test failure is resolved. Because `vitest.config.ts` computes include arrays at import time from `process.env`, controlled config-import permutations would improve confidence for this workflow/test-runner infrastructure.
  • **Runtime validation** — Import `vitest.config.ts` with `NEMOCLAW_RUN_INSTALLER_TESTS=1` and no CI env and verify installer integration includes both installer tests.. The helper-level and workflow-contract coverage is adequate for the PR, and the prior CI-dependent test failure is resolved. Because `vitest.config.ts` computes include arrays at import time from `process.env`, controlled config-import permutations would improve confidence for this workflow/test-runner infrastructure.
  • **Runtime validation** — Import `vitest.config.ts` with `NEMOCLAW_RUN_E2E_SCENARIOS=1` and verify `e2e-scenarios-live` includes `test/e2e-scenario/live/**/*.test.ts`.. The helper-level and workflow-contract coverage is adequate for the PR, and the prior CI-dependent test failure is resolved. Because `vitest.config.ts` computes include arrays at import time from `process.env`, controlled config-import permutations would improve confidence for this workflow/test-runner infrastructure.
  • **Runtime validation** — Import `vitest.config.ts` with `NEMOCLAW_RUN_BRANCH_VALIDATION_E2E=1` and no Brev auth env and verify `e2e-branch-validation` includes `test/e2e/brev-e2e.test.ts`.. The helper-level and workflow-contract coverage is adequate for the PR, and the prior CI-dependent test failure is resolved. Because `vitest.config.ts` computes include arrays at import time from `process.env`, controlled config-import permutations would improve confidence for this workflow/test-runner infrastructure.
  • **Acceptance clause:** Refs Adopt Vitest fixtures as the E2E scenario execution model #4941. — add test evidence or identify existing coverage. The deterministic context did not include linked issue Adopt Vitest fixtures as the E2E scenario execution model #4941 body or comments (`linkedIssues: []`), so issue-specific clauses could not be mapped. Repository docs confirm Adopt Vitest fixtures as the E2E scenario execution model #4941 is the Vitest fixture/single-runner execution-model decision context.

Workflow run details

This is an automated advisory review. A human maintainer must make the final merge decision.

Base automatically changed from codex/e2e-4941-stack-01-direction to main June 8, 2026 18:07
@cv cv marked this pull request as ready for review June 8, 2026 18:17
@cv

cv commented Jun 8, 2026

Copy link
Copy Markdown
Collaborator Author

Addressed the PR review advisor's branch-validation gate feedback in 018ea1d59: the branch-validation project now accepts an explicit workflow sentinel, the reusable Brev workflow sets NEMOCLAW_RUN_BRANCH_VALIDATION_E2E=1 for the Vitest step, and the config tests cover installer/live/branch gates plus the workflow contract.

Local verification passed:

  • npx prek run --all-files
  • npm test

Signed-off-by: Carlos Villela <cvillela@nvidia.com>
@github-actions

github-actions Bot commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

Brev E2E (full): PASSED on branch codex/e2e-4941-stack-02-live-projectSee logs

@cv cv merged commit 015a1b4 into main Jun 8, 2026
34 checks passed
@cv cv deleted the codex/e2e-4941-stack-02-live-project branch June 8, 2026 18:57
@cv cv added the v0.0.61 Release target label Jun 8, 2026
@wscurran wscurran added the chore Build, CI, dependency, or tooling maintenance label Jun 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: e2e End-to-end tests, nightly failures, or validation infrastructure chore Build, CI, dependency, or tooling maintenance v0.0.61 Release target

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants