Skip to content

test(e2e): migrate test-runtime-overrides.sh to vitest#5229

Merged
jyaunches merged 14 commits into
mainfrom
e2e-migrate/test-runtime-overrides-simple
Jun 11, 2026
Merged

test(e2e): migrate test-runtime-overrides.sh to vitest#5229
jyaunches merged 14 commits into
mainfrom
e2e-migrate/test-runtime-overrides-simple

Conversation

@jyaunches

@jyaunches jyaunches commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

Summary

Migrate test/e2e/test-runtime-overrides.sh with focused live Vitest coverage.

Related Issues

Refs #5098
Refs #759
Refs #719
Refs #2653

Contract mapping

  • Legacy assertion: baseline sandbox image config is valid and .config-hash matches.
    • Replacement: test/e2e-scenario/live/runtime-overrides.test.ts captures /sandbox/.openclaw/openclaw.json and runs sha256sum -c .config-hash inside a short-lived container.
    • Boundary preserved: real Docker image build and real nemoclaw-start ENTRYPOINT.
  • Legacy assertion: runtime model, inference API, context-window, max-token, and reasoning env overrides patch OpenClaw config.
    • Replacement: Vitest starts containers with the same NEMOCLAW_* env vars and asserts the patched config fields, including valid NEMOCLAW_INFERENCE_API_OVERRIDE=anthropic-messages.
    • Boundary preserved: real container entrypoint patching /sandbox/.openclaw/openclaw.json.
  • Legacy assertion: NEMOCLAW_CORS_ORIGIN extends gateway.controlUi.allowedOrigins.
    • Replacement: Vitest container run asserts the added origin and increased origin count.
    • Boundary preserved: real entrypoint CORS override path.
  • Legacy assertion: invalid override values are rejected and rejected config remains unchanged.
    • Replacement: Vitest asserts stderr validation messages and that rejected context-window override leaves baseline model/context intact.
    • Boundary preserved: real entrypoint validation path.

Simplicity check

  • Test shape: simple live Vitest test.
  • New shared helpers: none; Docker/process helpers stay local to the test.
  • New framework/registry/ledger: none.
  • Workflow changes: adds a free-standing runtime-overrides-vitest job to .github/workflows/e2e-vitest-scenarios.yaml so the new Vitest migration runs in CI. Legacy shell deletion and shell workflow retirement are deferred to Epic: Migrate legacy bash E2E into the Vitest E2E system #5098 Phase 11.
  • Docker Hub auth: none in the new job; it avoids secret-backed Docker login before running repo code.

Verification

  • NEMOCLAW_RUN_E2E_SCENARIOS=1 npx vitest run --project e2e-scenarios-live test/e2e-scenario/live/runtime-overrides.test.ts --silent=false --reporter=default (local Docker daemon unavailable, test records Docker-required skip)
  • npx vitest run --project e2e-vitest-support test/e2e-scenario/support-tests/e2e-scenarios-workflow.test.ts --silent=false --reporter=default
  • npm run build:cli
  • npm run typecheck:cli
  • git diff --check

Summary by CodeRabbit

  • Tests
    • Added an end-to-end scenario validating runtime configuration overrides (model, inference endpoint, context window, max tokens, reasoning, CORS), including invalid-input handling, integrity/hash checks, artifact/log collection, and conditional execution when e2e scenarios are enabled.
  • Chores
    • CI now includes a standalone E2E job for runtime-override scenarios, uploads artifacts with retention, and reports its result on PRs; workflow validation updated to recognize and enforce this job.

@coderabbitai

coderabbitai Bot commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Adds a Vitest E2E scenario that builds a NemoClaw runtime sandbox image, runs containers with environment-driven runtime config overrides, validates emitted openclaw.json and .config-hash (including invalid/rejected cases), and adds a GitHub Actions job plus workflow-boundary validator to run and report artifacts.

Changes

Runtime Config Overrides E2E Test

Layer / File(s) Summary
Test setup and types
test/e2e-scenario/live/runtime-overrides.test.ts
Repository root resolution, timeouts/buffer constants, Docker-required skip logic, and TypeScript types for Docker command results and the OpenClaw config subset used in assertions.
Command execution and logging utilities
test/e2e-scenario/live/runtime-overrides.test.ts
spawnSync result normalization, status/stdout/stderr/error formatting, and labeled Docker operation logging helpers.
Docker and config utilities
test/e2e-scenario/live/runtime-overrides.test.ts
Config accessors (first provider/model, primary model, allowed origins), docker run helper, captureConfig with up-to-3 retry attempts and JSON parsing, runConfigHashCheck using sha256sum, dockerAvailable check, and buildImage with conditional inspect/build steps.
End-to-end test workflow
test/e2e-scenario/live/runtime-overrides.test.ts
Full scenario orchestration: scenario metadata, Docker availability handling, image build, baseline capture and hash validation, multiple override variants and combined overrides, validation-failure stderr assertions, rejection checks, artifact writes, redacted docker.log, and optional image cleanup.
GitHub Actions workflow updates
.github/workflows/e2e-vitest-scenarios.yaml
Adds runtime-overrides-vitest free-standing job and allowlist entry, expands generate-matrix identifiers, conditional job execution based on inputs.jobs/matrix, runs the live test, uploads e2e-artifacts/vitest/runtime-overrides/ with ignore-if-empty and 14-day retention, and adds job to report-to-pr needs.
Workflow-boundary validation
tools/e2e-scenarios/workflow-boundary.mts
Adds validateRuntimeOverridesVitestJob, includes the job id in allowed_jobs, validates runner/if/env/steps/commands/artifact upload shape, and wires the validator into the main workflow-boundary checks.

Sequence Diagram(s)

sequenceDiagram
  participant Dispatcher as WorkflowDispatcher
  participant GitHubActions as ActionsRunner
  participant Validator as WorkflowBoundary
  participant VitestJob as runtime-overrides-vitest
  participant DockerBuild as DockerImageBuilder
  participant DockerContainer as RuntimeContainer
  Dispatcher->>GitHubActions: workflow_dispatch (inputs.jobs)
  GitHubActions->>Validator: validate workflow boundary & allowed_jobs
  GitHubActions->>VitestJob: schedule runtime-overrides-vitest job
  VitestJob->>DockerBuild: build runtime sandbox image
  DockerBuild->>DockerContainer: run ENTRYPOINT with env overrides
  DockerContainer->>VitestJob: emit openclaw.json and .config-hash
  VitestJob->>GitHubActions: upload artifacts (e2e-artifacts/vitest/runtime-overrides/)
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

  • NVIDIA/NemoClaw#5243: The main PR’s addition of the new runtime-overrides-vitest free-standing job and selector/validation plumbing is directly built on the selector/validation framework introduced here.
  • NVIDIA/NemoClaw#5150: Adds a similar free-standing live Vitest job and workflow-boundary validator pattern for another scenario.

Suggested labels

area: e2e, area: ci, area: inference

Suggested reviewers

  • cv
  • prekshivyas

"I dug a sandbox, snug and light,
Env flags hopping in the night.
Hashes checked, small carrots logged,
Overrides proved, no errors fogged.
The rabbit cheers — tests passed bright."

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title accurately and concisely describes the main change: migrating a shell-based e2e test to Vitest.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch e2e-migrate/test-runtime-overrides-simple

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions

Copy link
Copy Markdown
Contributor

This repository limits contributors to 10 open pull requests. Please close or merge existing PRs before opening new ones.

@github-actions github-actions Bot closed this Jun 11, 2026
@github-actions

github-actions Bot commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

E2E Advisor Recommendation

Required E2E: runtime-overrides-vitest
Optional E2E: network-policy-vitest

Dispatch hint: runtime-overrides-vitest

Workflow run

Full advisor summary

E2E Recommendation Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E

  • runtime-overrides-vitest (medium): This PR adds and wires the runtime-overrides live E2E job. It should be dispatched to prove the new workflow selector, job definition, Docker image build path, ENTRYPOINT override boundary, config-hash validation, and artifact upload path work in CI.

Optional E2E

  • network-policy-vitest (high): Optional adjacent confidence for another Docker/OpenShell live free-standing workflow job with security-sensitive CI wiring. Not merge-blocking because the PR only adds runtime-overrides to selector/reporting paths and does not modify network policy runtime code.

New E2E recommendations

  • None.

Dispatch hint

  • Workflow: .github/workflows/e2e-vitest-scenarios.yaml
  • jobs input: runtime-overrides-vitest

@github-actions

github-actions Bot commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

Vitest E2E Scenario Recommendation

Required Vitest E2E scenarios: runtime-overrides-vitest
Optional Vitest E2E scenarios: None

Dispatch required Vitest E2E scenarios:

  • gh workflow run e2e-vitest-scenarios.yaml --ref <pr-head-ref> --field jobs=runtime-overrides-vitest

Workflow run

Full Vitest E2E advisor summary

Vitest E2E Scenario Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required Vitest E2E scenarios

  • runtime-overrides-vitest: Focused free-standing Vitest job wired for changed live test test/e2e-scenario/live/runtime-overrides.test.ts.
    • Dispatch: gh workflow run e2e-vitest-scenarios.yaml --ref <pr-head-ref> --field jobs=runtime-overrides-vitest

Optional Vitest E2E scenarios

  • None.

Relevant changed files

  • .github/workflows/e2e-vitest-scenarios.yaml
  • test/e2e-scenario/live/runtime-overrides.test.ts
  • test/e2e-scenario/support-tests/e2e-scenarios-workflow.test.ts
  • tools/e2e-scenarios/workflow-boundary.mts

@github-actions

github-actions Bot commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor

Findings: 0 needs attention, 2 worth checking, 0 nice ideas
Since last review: 2 prior items resolved, 2 still apply, 0 new items found

Review findings

🛠️ Needs attention

  • None.

🔎 Worth checking

  • Source-of-truth review needed: test/e2e-scenario/live/runtime-overrides.test.ts captureConfig retry: The advisor marked localized patch analysis as needs_followup.
    • Recommendation: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
    • Evidence: captureConfig retries up to three attempts and stores the last parse error before throwing.
  • Clarify or regression-lock the Docker ENTRYPOINT config-capture retry (test/e2e-scenario/live/runtime-overrides.test.ts:157): The migrated Vitest test preserves the legacy three-attempt retry around short-lived Docker/ENTRYPOINT config capture. The comment now identifies the likely tee/fd3 race and a removal condition, but the retry remains a localized workaround that can hide invalid first-attempt output from the startup capture path. The PR does not include focused source-level coverage for that ENTRYPOINT stdout/fd3 behavior or an explicit explanation of why fixing that source path is out of scope for this migration.
    • Recommendation: Either add focused regression coverage around the ENTRYPOINT stdout/fd3 capture behavior, or expand the comment to state why changing the startup capture path is out of scope here and what concrete source change should remove this retry.
    • Evidence: captureConfig loops for attempts 1 through 3, catches parse errors, and retries before failing; the comment says to keep the retry until the startup capture path no longer uses tee for container stdout/stderr fanout.

🌱 Nice ideas

  • None.
Consider writing more tests for
  • **Runtime validation** — Docker ENTRYPOINT stdout/fd3 config capture emits valid JSON without relying on captureConfig retry, or fails only for the documented tee race. The PR changes a security-sensitive workflow dispatch boundary and a Docker/ENTRYPOINT sandbox runtime test. The added direct live test and workflow support tests are strong, but a small amount of additional targeted runtime/source-boundary coverage would reduce risk around the remaining retry workaround and invalid override rejection semantics.
  • **Runtime validation** — Each invalid runtime override leaves openclaw.json and .config-hash unchanged, not only emits the expected stderr message. The PR changes a security-sensitive workflow dispatch boundary and a Docker/ENTRYPOINT sandbox runtime test. The added direct live test and workflow support tests are strong, but a small amount of additional targeted runtime/source-boundary coverage would reduce risk around the remaining retry workaround and invalid override rejection semantics.
  • **Runtime validation** — validateE2eVitestScenariosWorkflowBoundary rejects runtime-overrides-vitest when needs omits validate-jobs or generate-matrix. The PR changes a security-sensitive workflow dispatch boundary and a Docker/ENTRYPOINT sandbox runtime test. The added direct live test and workflow support tests are strong, but a small amount of additional targeted runtime/source-boundary coverage would reduce risk around the remaining retry workaround and invalid override rejection semantics.
  • **Clarify or regression-lock the Docker ENTRYPOINT config-capture retry** — Either add focused regression coverage around the ENTRYPOINT stdout/fd3 capture behavior, or expand the comment to state why changing the startup capture path is out of scope here and what concrete source change should remove this retry.
  • **Acceptance clause:** No trusted linked issue acceptance clauses were provided in deterministic context. — add test evidence or identify existing coverage. Deterministic context reports linkedIssues: []. PR-body references to Epic: Migrate legacy bash E2E into the Vitest E2E system #5098, Sandbox config file (openclaw.json) is root-owned and unwritable — no supported path to change primary model #759, openclaw.json created as root with read-only permissions, making all config writes impossible from sandbox #719, and bug(e2e): runtime-overrides-e2e crashes immediately at baseline config capture #2653 were treated as untrusted, non-authoritative evidence.
  • **test/e2e-scenario/live/runtime-overrides.test.ts captureConfig retry** — No focused source-level regression test was added for the ENTRYPOINT stdout/fd3 capture behavior; the live test relies on the retry.. captureConfig retries up to three attempts and stores the last parse error before throwing.
Since last review details

Current findings:

  • Source-of-truth review needed: test/e2e-scenario/live/runtime-overrides.test.ts captureConfig retry: The advisor marked localized patch analysis as needs_followup.
    • Recommendation: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
    • Evidence: captureConfig retries up to three attempts and stores the last parse error before throwing.
  • Clarify or regression-lock the Docker ENTRYPOINT config-capture retry (test/e2e-scenario/live/runtime-overrides.test.ts:157): The migrated Vitest test preserves the legacy three-attempt retry around short-lived Docker/ENTRYPOINT config capture. The comment now identifies the likely tee/fd3 race and a removal condition, but the retry remains a localized workaround that can hide invalid first-attempt output from the startup capture path. The PR does not include focused source-level coverage for that ENTRYPOINT stdout/fd3 behavior or an explicit explanation of why fixing that source path is out of scope for this migration.
    • Recommendation: Either add focused regression coverage around the ENTRYPOINT stdout/fd3 capture behavior, or expand the comment to state why changing the startup capture path is out of scope here and what concrete source change should remove this retry.
    • Evidence: captureConfig loops for attempts 1 through 3, catches parse errors, and retries before failing; the comment says to keep the retry until the startup capture path no longer uses tee for container stdout/stderr fanout.

Workflow run details

This is an automated advisory review. A human maintainer must make the final merge decision.

@jyaunches jyaunches reopened this Jun 11, 2026
@jyaunches

Copy link
Copy Markdown
Contributor Author

Addressed PR advisor findings in d813a53:\n- added positive live coverage for valid NEMOCLAW_INFERENCE_API_OVERRIDE=anthropic-messages and hash validation;\n- redacted docker.log through the E2E SecretStore before writing artifacts;\n- wired the new free-standing runtime-overrides Vitest job into e2e-vitest-scenarios.yaml.\n\nValidation:\n- NEMOCLAW_RUN_E2E_SCENARIOS=1 npx vitest run --project e2e-scenarios-live test/e2e-scenario/live/runtime-overrides.test.ts --silent=false --reporter=default (Docker unavailable locally, records skip)\n- npx vitest run --project e2e-vitest-support test/e2e-scenario/support-tests/e2e-scenarios-workflow.test.ts --silent=false --reporter=default\n- npm run typecheck:cli\n- git diff --check

@github-actions

Copy link
Copy Markdown
Contributor

This repository limits contributors to 10 open pull requests. Please close or merge existing PRs before opening new ones.

@github-actions github-actions Bot closed this Jun 11, 2026
@jyaunches jyaunches reopened this Jun 11, 2026
@github-actions

Copy link
Copy Markdown
Contributor

This repository limits contributors to 10 open pull requests. Please close or merge existing PRs before opening new ones.

@github-actions github-actions Bot closed this Jun 11, 2026
@jyaunches

Copy link
Copy Markdown
Contributor Author

Follow-up c621d1f resolves the remaining PR advisor workflow item by removing Docker Hub login from the new runtime-overrides-vitest job. I also updated the PR body to accurately list the Vitest workflow job wiring and note that legacy shell deletion/workflow retirement are still deferred to #5098 Phase 11.

@jyaunches jyaunches reopened this Jun 11, 2026
@jyaunches

Copy link
Copy Markdown
Contributor Author

Follow-up ac48069 documents the local captureConfig retry source condition/removal point. The Docker Hub auth finding referenced line numbers from the separate openclaw-tui-chat-correlation job; runtime-overrides-vitest has no Docker Hub auth step after c621d1f.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
test/e2e-scenario/live/runtime-overrides.test.ts (1)

347-380: ⚡ Quick win

Collapse repeated invalid-override assertions into a table-driven loop.

Line 347 through Line 380 repeats the same assertion shape six times, which unnecessarily increases function complexity. Converting this block into case-table iteration keeps the scenario easier to extend and aligns with the repo’s “low complexity” guidance.

♻️ Suggested refactor
-      expect(
-        runOverrideStderr(dockerLog, image, "invalid model override", {
-          NEMOCLAW_MODEL_OVERRIDE: "bad\u0001model",
-        }),
-      ).toContain("control characters");
-      expect(
-        runOverrideStderr(dockerLog, image, "invalid context window", {
-          NEMOCLAW_MODEL_OVERRIDE: "test",
-          NEMOCLAW_CONTEXT_WINDOW: "notanumber",
-        }),
-      ).toContain("must be a positive integer");
-      expect(
-        runOverrideStderr(dockerLog, image, "invalid max tokens", {
-          NEMOCLAW_MODEL_OVERRIDE: "test",
-          NEMOCLAW_MAX_TOKENS: "abc",
-        }),
-      ).toContain("must be a positive integer");
-      expect(
-        runOverrideStderr(dockerLog, image, "invalid reasoning", {
-          NEMOCLAW_MODEL_OVERRIDE: "test",
-          NEMOCLAW_REASONING: "maybe",
-        }),
-      ).toContain('must be "true" or "false"');
-      expect(
-        runOverrideStderr(dockerLog, image, "invalid CORS origin", {
-          NEMOCLAW_CORS_ORIGIN: "ftp://evil.com",
-        }),
-      ).toContain("must start with http");
-      expect(
-        runOverrideStderr(dockerLog, image, "invalid inference API", {
-          NEMOCLAW_MODEL_OVERRIDE: "test",
-          NEMOCLAW_INFERENCE_API_OVERRIDE: "graphql",
-        }),
-      ).toContain("openai-completions");
+      const invalidCases: Array<{
+        label: string;
+        env: Record<string, string>;
+        expected: string;
+      }> = [
+        {
+          label: "invalid model override",
+          env: { NEMOCLAW_MODEL_OVERRIDE: "bad\u0001model" },
+          expected: "control characters",
+        },
+        {
+          label: "invalid context window",
+          env: { NEMOCLAW_MODEL_OVERRIDE: "test", NEMOCLAW_CONTEXT_WINDOW: "notanumber" },
+          expected: "must be a positive integer",
+        },
+        {
+          label: "invalid max tokens",
+          env: { NEMOCLAW_MODEL_OVERRIDE: "test", NEMOCLAW_MAX_TOKENS: "abc" },
+          expected: "must be a positive integer",
+        },
+        {
+          label: "invalid reasoning",
+          env: { NEMOCLAW_MODEL_OVERRIDE: "test", NEMOCLAW_REASONING: "maybe" },
+          expected: 'must be "true" or "false"',
+        },
+        {
+          label: "invalid CORS origin",
+          env: { NEMOCLAW_CORS_ORIGIN: "ftp://evil.com" },
+          expected: "must start with http",
+        },
+        {
+          label: "invalid inference API",
+          env: {
+            NEMOCLAW_MODEL_OVERRIDE: "test",
+            NEMOCLAW_INFERENCE_API_OVERRIDE: "graphql",
+          },
+          expected: "openai-completions",
+        },
+      ];
+
+      for (const c of invalidCases) {
+        expect(runOverrideStderr(dockerLog, image, c.label, c.env)).toContain(c.expected);
+      }

As per coding guidelines, “Keep function complexity low in JavaScript and TypeScript code.”

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/e2e-scenario/live/runtime-overrides.test.ts` around lines 347 - 380,
Replace the six repeated expect(...) assertions with a table-driven loop: create
an array of test cases (each containing a label/description, env object like
{NEMOCLAW_MODEL_OVERRIDE: "...", ...}, and expected substring) and iterate
(e.g., forEach or for..of) calling runOverrideStderr(dockerLog, image,
description, env) and asserting the returned stderr contains the expected
substring; update the block around the existing runOverrideStderr, dockerLog,
and image usages so each case uses the same call site and assertion pattern to
reduce duplication.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@test/e2e-scenario/live/runtime-overrides.test.ts`:
- Around line 347-380: Replace the six repeated expect(...) assertions with a
table-driven loop: create an array of test cases (each containing a
label/description, env object like {NEMOCLAW_MODEL_OVERRIDE: "...", ...}, and
expected substring) and iterate (e.g., forEach or for..of) calling
runOverrideStderr(dockerLog, image, description, env) and asserting the returned
stderr contains the expected substring; update the block around the existing
runOverrideStderr, dockerLog, and image usages so each case uses the same call
site and assertion pattern to reduce duplication.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 67e23d94-e66e-4cf3-87ef-57c5da56a549

📥 Commits

Reviewing files that changed from the base of the PR and between be0ed7b and ac48069.

📒 Files selected for processing (2)
  • .github/workflows/e2e-vitest-scenarios.yaml
  • test/e2e-scenario/live/runtime-overrides.test.ts

…ime-overrides-simple

# Conflicts:
#	.github/workflows/e2e-vitest-scenarios.yaml
@jyaunches jyaunches changed the title test(e2e): migrate runtime overrides to Vitest test(e2e): P2 independent 7 migrate test-runtime-overrides.sh to vitest Jun 11, 2026
@jyaunches jyaunches changed the title test(e2e): P2 independent 7 migrate test-runtime-overrides.sh to vitest test(e2e): migrate runtime overrides to Vitest Jun 11, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/e2e-vitest-scenarios.yaml:
- Around line 505-506: Replace direct template interpolation of inputs into JS
by setting them as environment variables in the workflow and reading them from
process.env inside the script: stop using the expressions that assign to
prNumberInput and requestedScenarios via ${{ toJSON(inputs.pr_number) }} and ${{
toJSON(inputs.scenarios) }}, instead expose inputs.pr_number and
inputs.scenarios into env (e.g., JOB_PR_NUMBER, JOB_SCENARIOS) and change the
script to read process.env.JOB_PR_NUMBER and process.env.JOB_SCENARIOS into
prNumberInput and requestedScenarios (with defaults and explicit
sanitization/parsing as needed) so no untrusted input is directly interpolated
into code execution context.
- Around line 482-493: The report-to-pr job's needs array (job: report-to-pr) is
missing the newly added runtime-overrides-vitest dependency; update the needs
list for the report-to-pr job to include runtime-overrides-vitest so the report
waits for and aggregates that job's results (add runtime-overrides-vitest
alongside the existing entries in the needs array).
- Around line 393-474: The job "gateway-guard-recovery" lacks the selective
job-run filter used by other Vitest jobs, so add the same `if` condition that
checks github.event.inputs.jobs (e.g., using contains(...,
'gateway-guard-recovery') or contains(..., 'all')) to the gateway-guard-recovery
job header to match the pattern used by jobs at lines near other Vitest jobs;
update the job declaration named gateway-guard-recovery to include that `if`
guard so it only runs when selected via inputs.jobs.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: dbfc6753-dbcb-4006-aadf-bec59fdb63cc

📥 Commits

Reviewing files that changed from the base of the PR and between ac48069 and 9eb8b3c.

📒 Files selected for processing (2)
  • .github/workflows/e2e-vitest-scenarios.yaml
  • tools/e2e-scenarios/workflow-boundary.mts

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Inline review comments failed to post. This is likely due to GitHub's internal server error or limits when posting large numbers of comments. If you are seeing this consistently it is likely a permissions issue. Please check "Moderation" -> "Code review limits" under your organization settings.

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/e2e-vitest-scenarios.yaml:
- Around line 505-506: Replace direct template interpolation of inputs into JS
by setting them as environment variables in the workflow and reading them from
process.env inside the script: stop using the expressions that assign to
prNumberInput and requestedScenarios via ${{ toJSON(inputs.pr_number) }} and ${{
toJSON(inputs.scenarios) }}, instead expose inputs.pr_number and
inputs.scenarios into env (e.g., JOB_PR_NUMBER, JOB_SCENARIOS) and change the
script to read process.env.JOB_PR_NUMBER and process.env.JOB_SCENARIOS into
prNumberInput and requestedScenarios (with defaults and explicit
sanitization/parsing as needed) so no untrusted input is directly interpolated
into code execution context.
- Around line 482-493: The report-to-pr job's needs array (job: report-to-pr) is
missing the newly added runtime-overrides-vitest dependency; update the needs
list for the report-to-pr job to include runtime-overrides-vitest so the report
waits for and aggregates that job's results (add runtime-overrides-vitest
alongside the existing entries in the needs array).
- Around line 393-474: The job "gateway-guard-recovery" lacks the selective
job-run filter used by other Vitest jobs, so add the same `if` condition that
checks github.event.inputs.jobs (e.g., using contains(...,
'gateway-guard-recovery') or contains(..., 'all')) to the gateway-guard-recovery
job header to match the pattern used by jobs at lines near other Vitest jobs;
update the job declaration named gateway-guard-recovery to include that `if`
guard so it only runs when selected via inputs.jobs.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: dbfc6753-dbcb-4006-aadf-bec59fdb63cc

📥 Commits

Reviewing files that changed from the base of the PR and between ac48069 and 9eb8b3c.

📒 Files selected for processing (2)
  • .github/workflows/e2e-vitest-scenarios.yaml
  • tools/e2e-scenarios/workflow-boundary.mts
🛑 Comments failed to post (3)
.github/workflows/e2e-vitest-scenarios.yaml (3)

393-474: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

gateway-guard-recovery job missing required job filter condition.

All free-standing Vitest jobs in this workflow have a job filter condition (e.g., line 188, 228, 272, 315) to support selective execution via inputs.jobs. The new gateway-guard-recovery job does not have an if condition, so it will always run even when inputs.jobs selects other specific jobs.

🔧 Proposed fix to add job filter
   gateway-guard-recovery:
+    if: ${{ inputs.jobs == '' || contains(format(',{0},', inputs.jobs), ',gateway-guard-recovery,') }}
     runs-on: ubuntu-latest

Potential scope creep: gateway-guard-recovery not mentioned in PR objectives.

The PR objectives describe migrating test-runtime-overrides.sh to Vitest and adding the runtime-overrides-vitest job. The gateway-guard-recovery job (lines 393-474) and associated recovery scenario appear unrelated to runtime config overrides. Consider splitting unrelated features into separate PRs for clearer review and history.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/e2e-vitest-scenarios.yaml around lines 393 - 474, The job
"gateway-guard-recovery" lacks the selective job-run filter used by other Vitest
jobs, so add the same `if` condition that checks github.event.inputs.jobs (e.g.,
using contains(..., 'gateway-guard-recovery') or contains(..., 'all')) to the
gateway-guard-recovery job header to match the pattern used by jobs at lines
near other Vitest jobs; update the job declaration named gateway-guard-recovery
to include that `if` guard so it only runs when selected via inputs.jobs.

482-493: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

report-to-pr job missing runtime-overrides-vitest in needs array.

The report-to-pr job aggregates results from scenario jobs but does not list runtime-overrides-vitest in its needs array (lines 484-492), even though that job was added in this PR. The report will not wait for or include the runtime-overrides job results.

📋 Proposed fix to add runtime-overrides-vitest to needs
     needs:
       [
         generate-matrix,
         live-scenarios,
         openshell-version-pin-vitest,
         onboard-negative-paths-vitest,
+        runtime-overrides-vitest,
         openclaw-tui-chat-correlation-vitest,
         gateway-guard-recovery,
       ]
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

  report-to-pr:
    runs-on: ubuntu-latest
    needs:
      [
        generate-matrix,
        live-scenarios,
        openshell-version-pin-vitest,
        onboard-negative-paths-vitest,
        runtime-overrides-vitest,
        openclaw-tui-chat-correlation-vitest,
        gateway-guard-recovery,
      ]
    if: ${{ always() && github.event_name == 'workflow_dispatch' }}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/e2e-vitest-scenarios.yaml around lines 482 - 493, The
report-to-pr job's needs array (job: report-to-pr) is missing the newly added
runtime-overrides-vitest dependency; update the needs list for the report-to-pr
job to include runtime-overrides-vitest so the report waits for and aggregates
that job's results (add runtime-overrides-vitest alongside the existing entries
in the needs array).

505-506: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Template injection vulnerability flagged by static analysis.

Lines 505 and 506 directly interpolate inputs.pr_number and inputs.scenarios into JavaScript code via ${{ toJSON(...) }} without sanitization. While toJSON provides some safety, untrusted workflow_dispatch inputs expanded into code contexts can enable code injection attacks.

The current pattern on line 505:

const prNumberInput = ${{ toJSON(inputs.pr_number) }} || '';

Should assign inputs to environment variables first, then read them from process.env in the script:

🛡️ Recommended mitigation
       - name: Post Vitest scenario results to PR
         uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
+        env:
+          PR_NUMBER_INPUT: ${{ inputs.pr_number }}
+          REQUESTED_SCENARIOS: ${{ inputs.scenarios }}
         with:
           script: |
             const needs = ${{ toJSON(needs) }};
             const runUrl = `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`;
             const workflowBranch = context.ref.replace('refs/heads/', '');
-            const prNumberInput = ${{ toJSON(inputs.pr_number) }} || '';
-            const requestedScenarios = ${{ toJSON(inputs.scenarios) }} || '';
+            const prNumberInput = process.env.PR_NUMBER_INPUT || '';
+            const requestedScenarios = process.env.REQUESTED_SCENARIOS || '';
🧰 Tools
🪛 zizmor (1.25.2)

[error] 505-505: code injection via template expansion (template-injection): may expand into attacker-controllable code

(template-injection)


[error] 506-506: code injection via template expansion (template-injection): may expand into attacker-controllable code

(template-injection)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/e2e-vitest-scenarios.yaml around lines 505 - 506, Replace
direct template interpolation of inputs into JS by setting them as environment
variables in the workflow and reading them from process.env inside the script:
stop using the expressions that assign to prNumberInput and requestedScenarios
via ${{ toJSON(inputs.pr_number) }} and ${{ toJSON(inputs.scenarios) }}, instead
expose inputs.pr_number and inputs.scenarios into env (e.g., JOB_PR_NUMBER,
JOB_SCENARIOS) and change the script to read process.env.JOB_PR_NUMBER and
process.env.JOB_SCENARIOS into prNumberInput and requestedScenarios (with
defaults and explicit sanitization/parsing as needed) so no untrusted input is
directly interpolated into code execution context.

Source: Linters/SAST tools

@jyaunches jyaunches changed the title test(e2e): migrate runtime overrides to Vitest test(e2e): migrate runtime overrides to vitest Jun 11, 2026
@jyaunches jyaunches changed the title test(e2e): migrate runtime overrides to vitest test(e2e): migrate runtime overrides to vitest [IND-7] Jun 11, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Vitest E2E Scenario Results — ✅ All jobs passed

Run: 27358255067
Workflow ref: e2e-migrate/test-runtime-overrides-simple
Requested scenarios: (default — all supported)
Summary: 2 passed, 0 failed, 5 skipped

Job Result
gateway-guard-recovery ⏭️ skipped
generate-matrix ✅ success
live-scenarios ⏭️ skipped
onboard-negative-paths-vitest ⏭️ skipped
openclaw-tui-chat-correlation-vitest ⏭️ skipped
openshell-version-pin-vitest ⏭️ skipped
runtime-overrides-vitest ✅ success

…ime-overrides-simple

# Conflicts:
#	.github/workflows/e2e-vitest-scenarios.yaml
#	tools/e2e-scenarios/workflow-boundary.mts
@github-actions

Copy link
Copy Markdown
Contributor

Vitest E2E Scenario Results — ✅ All jobs passed

Run: 27360616676
Workflow ref: e2e-migrate/test-runtime-overrides-simple
Requested scenarios: (default — all supported)
Requested jobs: runtime-overrides-vitest
Summary: 3 passed, 0 failed, 5 skipped

Job Result
gateway-guard-recovery ⏭️ skipped
generate-matrix ✅ success
live-scenarios ⏭️ skipped
onboard-negative-paths-vitest ⏭️ skipped
openclaw-tui-chat-correlation-vitest ⏭️ skipped
openshell-version-pin-vitest ⏭️ skipped
runtime-overrides-vitest ✅ success
validate-jobs ✅ success

@cv cv added the v0.0.64 Release target label Jun 11, 2026
…ime-overrides-simple

# Conflicts:
#	.github/workflows/e2e-vitest-scenarios.yaml

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
.github/workflows/e2e-vitest-scenarios.yaml (1)

66-70: ⚡ Quick win

Skip matrix generation for jobs-only runs.

When inputs.jobs is set, live-scenarios is skipped, but generate-matrix still performs setup and matrix computation. Adding an equivalent gate (or dependency short-circuit via validate-jobs) will cut unnecessary CI time and reduce noisy runs.

Suggested change
   generate-matrix:
+    if: ${{ inputs.jobs == '' }}
     runs-on: ubuntu-latest

Also applies to: 118-121

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/e2e-vitest-scenarios.yaml around lines 66 - 70, The
generate-matrix job currently always runs even when a jobs-only run is
requested; add a gate so matrix computation is skipped when workflow_dispatch
input "jobs" is provided (e.g., add a job-level conditional using the workflow
input like if: ${{ !github.event.inputs.jobs }} or make generate-matrix depend
on validate-jobs and short-circuit via an output/flag from validate-jobs),
update the generate-matrix job declaration (job name generate-matrix) and apply
the same conditional to the other affected job block mentioned (the second
matrix-generation block around lines 118-121) so matrix setup is bypassed for
jobs-only runs.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In @.github/workflows/e2e-vitest-scenarios.yaml:
- Around line 66-70: The generate-matrix job currently always runs even when a
jobs-only run is requested; add a gate so matrix computation is skipped when
workflow_dispatch input "jobs" is provided (e.g., add a job-level conditional
using the workflow input like if: ${{ !github.event.inputs.jobs }} or make
generate-matrix depend on validate-jobs and short-circuit via an output/flag
from validate-jobs), update the generate-matrix job declaration (job name
generate-matrix) and apply the same conditional to the other affected job block
mentioned (the second matrix-generation block around lines 118-121) so matrix
setup is bypassed for jobs-only runs.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: d6744b85-f148-4789-a5a3-cca687981a2c

📥 Commits

Reviewing files that changed from the base of the PR and between 9eb8b3c and 57fca09.

📒 Files selected for processing (2)
  • .github/workflows/e2e-vitest-scenarios.yaml
  • tools/e2e-scenarios/workflow-boundary.mts

@github-actions

Copy link
Copy Markdown
Contributor

Vitest E2E Scenario Results — ✅ All jobs passed

Run: 27375571851
Workflow ref: e2e-migrate/test-runtime-overrides-simple
Requested scenarios: (default — all supported)
Requested jobs: runtime-overrides-vitest
Summary: 3 passed, 0 failed, 6 skipped

Job Result
gateway-guard-recovery ⏭️ skipped
generate-matrix ✅ success
live-scenarios ⏭️ skipped
onboard-negative-paths-vitest ⏭️ skipped
openclaw-tui-chat-correlation-vitest ⏭️ skipped
openshell-version-pin-vitest ⏭️ skipped
runtime-overrides-vitest ✅ success
token-rotation-vitest ⏭️ skipped
validate-jobs ✅ success

…ime-overrides-simple

# Conflicts:
#	.github/workflows/e2e-vitest-scenarios.yaml
#	tools/e2e-scenarios/workflow-boundary.mts
@github-actions

Copy link
Copy Markdown
Contributor

Vitest E2E Scenario Results — ✅ All jobs passed

Run: 27376207423
Workflow ref: e2e-migrate/test-runtime-overrides-simple
Requested scenarios: (default — all supported)
Requested jobs: runtime-overrides-vitest
Summary: 3 passed, 0 failed, 7 skipped

Job Result
gateway-guard-recovery ⏭️ skipped
generate-matrix ✅ success
live-scenarios ⏭️ skipped
network-policy-vitest ⏭️ skipped
onboard-negative-paths-vitest ⏭️ skipped
openclaw-tui-chat-correlation-vitest ⏭️ skipped
openshell-version-pin-vitest ⏭️ skipped
runtime-overrides-vitest ✅ success
token-rotation-vitest ⏭️ skipped
validate-jobs ✅ success

…ime-overrides-simple

# Conflicts:
#	.github/workflows/e2e-vitest-scenarios.yaml
#	tools/e2e-scenarios/workflow-boundary.mts
@github-actions

Copy link
Copy Markdown
Contributor

Vitest E2E Scenario Results — ✅ All jobs passed

Run: 27378767806
Workflow ref: e2e-migrate/test-runtime-overrides-simple
Requested scenarios: (default — all supported)
Requested jobs: runtime-overrides-vitest
Summary: 3 passed, 0 failed, 8 skipped

Job Result
gateway-guard-recovery ⏭️ skipped
generate-matrix ✅ success
hermes-e2e-vitest ⏭️ skipped
live-scenarios ⏭️ skipped
network-policy-vitest ⏭️ skipped
onboard-negative-paths-vitest ⏭️ skipped
openclaw-tui-chat-correlation-vitest ⏭️ skipped
openshell-version-pin-vitest ⏭️ skipped
runtime-overrides-vitest ✅ success
token-rotation-vitest ⏭️ skipped
validate-jobs ✅ success

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tools/e2e-scenarios/workflow-boundary.mts`:
- Around line 732-740: The validation currently blocks DOCKERHUB_* env keys but
does not prevent docker login via inline secrets in step.run; update the runtime
override checks to also scan each step.run for docker login usage and for
secrets interpolation patterns. Add or call a helper like
requireNoDockerHubAuthInRun(errors, stepName, step.run) (or extend
requireNoDispatchInputInterpolation) to detect patterns such as
/docker\s+login/i and interpolation tokens like /secrets\.[A-Za-z0-9_]+/ or
/\${{\s*secrets\.[^}]+}}/ and push an error when found, alongside the existing
requireEnvDoesNotExposeSecret and asSteps/asRecord usage. Ensure this runs
inside the same loop over steps (runtime-overrides-vitest step '<name>') so
inline runs cannot bypass the DOCKERHUB auth ban.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 9cebcd96-b228-4f67-84d8-85c33622c766

📥 Commits

Reviewing files that changed from the base of the PR and between f196b52 and 18a4071.

📒 Files selected for processing (2)
  • .github/workflows/e2e-vitest-scenarios.yaml
  • tools/e2e-scenarios/workflow-boundary.mts
🚧 Files skipped from review as they are similar to previous changes (1)
  • .github/workflows/e2e-vitest-scenarios.yaml

Comment thread tools/e2e-scenarios/workflow-boundary.mts
…ime-overrides-simple

# Conflicts:
#	.github/workflows/e2e-vitest-scenarios.yaml
#	tools/e2e-scenarios/workflow-boundary.mts
@jyaunches jyaunches changed the title test(e2e): migrate runtime overrides to vitest [IND-7] test(e2e): migrate test-runtime-overrides.sh to vitest Jun 11, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Vitest E2E Scenario Results — ✅ All jobs passed

Run: 27379912291
Workflow ref: e2e-migrate/test-runtime-overrides-simple
Requested scenarios: (default — all supported)
Requested jobs: runtime-overrides-vitest
Summary: 3 passed, 0 failed, 9 skipped

Job Result
credential-migration-vitest ⏭️ skipped
gateway-guard-recovery ⏭️ skipped
generate-matrix ✅ success
hermes-e2e-vitest ⏭️ skipped
live-scenarios ⏭️ skipped
network-policy-vitest ⏭️ skipped
onboard-negative-paths-vitest ⏭️ skipped
openclaw-tui-chat-correlation-vitest ⏭️ skipped
openshell-version-pin-vitest ⏭️ skipped
runtime-overrides-vitest ✅ success
token-rotation-vitest ⏭️ skipped
validate-jobs ✅ success

@github-actions

Copy link
Copy Markdown
Contributor

Vitest E2E Scenario Results — ✅ All jobs passed

Run: 27379993075
Workflow ref: e2e-migrate/test-runtime-overrides-simple
Requested scenarios: (default — all supported)
Requested jobs: runtime-overrides-vitest
Summary: 3 passed, 0 failed, 9 skipped

Job Result
credential-migration-vitest ⏭️ skipped
gateway-guard-recovery ⏭️ skipped
generate-matrix ✅ success
hermes-e2e-vitest ⏭️ skipped
live-scenarios ⏭️ skipped
network-policy-vitest ⏭️ skipped
onboard-negative-paths-vitest ⏭️ skipped
openclaw-tui-chat-correlation-vitest ⏭️ skipped
openshell-version-pin-vitest ⏭️ skipped
runtime-overrides-vitest ✅ success
token-rotation-vitest ⏭️ skipped
validate-jobs ✅ success

@jyaunches jyaunches merged commit bdb8b86 into main Jun 11, 2026
68 checks passed
@jyaunches jyaunches deleted the e2e-migrate/test-runtime-overrides-simple branch June 11, 2026 22:06
@wscurran wscurran added area: e2e End-to-end tests, nightly failures, or validation infrastructure refactor PR restructures code without intended behavior change labels Jun 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: e2e End-to-end tests, nightly failures, or validation infrastructure refactor PR restructures code without intended behavior change v0.0.64 Release target

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants