Skip to content

test(e2e): migrate test-gateway-drift-preflight.sh to vitest#5350

Merged
cv merged 7 commits into
mainfrom
e2e-migrate/test-gateway-drift-preflight
Jun 13, 2026
Merged

test(e2e): migrate test-gateway-drift-preflight.sh to vitest#5350
cv merged 7 commits into
mainfrom
e2e-migrate/test-gateway-drift-preflight

Conversation

@jyaunches

@jyaunches jyaunches commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

Summary

Migrate test/e2e/test-gateway-drift-preflight.sh with the simplest equivalent Vitest coverage.

Related Issues

Refs #5098
Refs #3399
Refs #3423

Contract mapping

  • Legacy assertion: protobuf invalid-wire sandbox-list failures fail closed and do not classify a registered sandbox as stopped.
    • Replacement: test/gateway-drift-preflight.test.ts drives the real repo CLI through fake openshell/docker binaries and asserts fail-closed diagnostics/no backup success.
    • Boundary preserved: real node bin/nemoclaw.js backup-all, PATH-resolved openshell sandbox list, fake process exit/output boundary, and filesystem registry state.
  • Legacy assertion: stale patched cluster gateway image drift fails before openshell sandbox list.
    • Replacement: test/gateway-drift-preflight.test.ts fakes active Docker cluster inspect output and asserts image drift output plus no sandbox-list call.
    • Boundary preserved: real CLI preflight, PATH-resolved Docker inspect shim, OpenShell version shim, and process exit behavior.
  • Legacy assertion: host-process gateway binary drift blocks both backup-all and upgrade-sandboxes --check before sandbox-list.
    • Replacement: test/gateway-drift-preflight.test.ts seeds runtime markers/fallback binaries and invokes both real CLI commands.
    • Boundary preserved: real host-process runtime marker JSON, live fake gateway PID for marker trust, /proc/process identity probe, and fallback binary resolution.
  • Legacy assertion: a stale dead-PID runtime marker does not fabricate host-process drift when the live fallback gateway binary matches the installed CLI.
    • Replacement: test/gateway-drift-preflight.test.ts seeds a stale marker pointing at an old binary and asserts the CLI proceeds to openshell sandbox list.
    • Boundary preserved: real marker file semantics, fallback PATH sibling resolution, and subprocess call logging.

Simplicity check

  • Test shape: focused process Vitest test in test/gateway-drift-preflight.test.ts.
  • Original runner/lane: regression-e2e.yaml job gateway-drift-preflight-e2e, runs-on: ubuntu-latest, Node 22, fake Docker/OpenShell shims, no secrets.
  • Replacement runner: same runner class via e2e-vitest-scenarios.yaml job gateway-drift-preflight-vitest, runs-on: ubuntu-latest, Node 22, no secrets.
  • New shared helpers: none; one-off fake CLI/Docker/OpenShell helpers stay local to the test.
  • New framework/registry/ledger: none
  • Workflow changes: add selective free-standing Vitest job and selector inventory mapping; legacy shell script and regression lane deletion deferred to Epic: Migrate legacy bash E2E into the Vitest E2E system #5098 Phase 11.
  • Selective dispatch: e2e-vitest-scenarios.yaml with jobs=gateway-drift-preflight-vitest on this PR branch.

Verification

Note: local full pre-push CLI suite was attempted by hooks but is not part of the focused migration validation and failed on unrelated local environment/pre-existing issues (nemoclaw/node_modules/json5 missing, nemoclaw/dist/blueprint/private-networks.js missing, macOS timeout/platform expectations). Focused migration validations above passed.

Summary by CodeRabbit

  • New Features

    • Added gateway drift preflight checks to prevent unsafe backup and upgrade operations by detecting schema mismatches, gateway image drift, and host-process incompatibilities.
  • Tests

    • Expanded end-to-end Vitest scenarios covering gateway drift detection, fail-closed behavior, and various drift permutations.
  • Chores

    • CI updated with a new automated job for gateway-drift-preflight and integrated so its results appear in PR reports.

Reserve Phase 4 E2E migration work for test-gateway-drift-preflight.sh.

Refs #5098
@jyaunches jyaunches added area: e2e End-to-end tests, nightly failures, or validation infrastructure chore Build, CI, dependency, or tooling maintenance labels Jun 12, 2026
@copy-pr-bot

copy-pr-bot Bot commented Jun 12, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@jyaunches jyaunches added the v0.0.65 Release target label Jun 12, 2026
@coderabbitai

coderabbitai Bot commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 6b76bc88-3cc9-4840-b440-833bb060184d

📥 Commits

Reviewing files that changed from the base of the PR and between b12a2df and 6a7de6d.

📒 Files selected for processing (3)
  • .github/workflows/e2e-vitest-scenarios.yaml
  • test/e2e-scenario/support-tests/e2e-scenarios-workflow.test.ts
  • tools/e2e-scenarios/workflow-boundary.mts
🚧 Files skipped from review as they are similar to previous changes (3)
  • test/e2e-scenario/support-tests/e2e-scenarios-workflow.test.ts
  • .github/workflows/e2e-vitest-scenarios.yaml
  • tools/e2e-scenarios/workflow-boundary.mts

📝 Walkthrough

Walkthrough

This PR adds a gateway drift preflight Vitest E2E test that exercises fail-closed behavior when gateway schema or binary drift is detected, wires it as a free-standing CI job in e2e-vitest-scenarios.yaml, and validates the scenario-to-job mapping through workflow-boundary tests.

Changes

Gateway Drift Preflight E2E Test and CI Integration

Layer / File(s) Summary
E2E Test Implementation
test/gateway-drift-preflight.test.ts
Implements a Vitest E2E with filesystem lifecycle management, fake openshell/docker/openshell-gateway binaries/markers, and a single test case that runs the CLI across multiple drift scenarios (schema mismatch, image drift, host-process drift, stale markers), asserting fail-closed behavior, version reporting, and sandbox-list invocation control.
CI Workflow Job and Reporting
.github/workflows/e2e-vitest-scenarios.yaml
Adds the gateway-drift-preflight-vitest free-standing job with conditional execution, environment setup, Vitest invocation for the preflight test, and artifact upload; registers it in report-to-pr dependencies for PR results.
Workflow Boundary Validation
test/e2e-scenario/support-tests/e2e-scenarios-workflow.test.ts, tools/e2e-scenarios/workflow-boundary.mts
Extends dispatch-selector tests and workflow-boundary validation to assert the gateway-drift-preflight scenario maps to gateway-drift-preflight-vitest job via both scenarios and jobs dispatch paths.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

  • NVIDIA/NemoClaw#5370: The main PR's added gateway-drift-preflight-vitest selector wiring/coverage directly builds on this PR's refactor to derive/validate free-standing Vitest selectors from workflow metadata.
  • NVIDIA/NemoClaw#5233: Both PRs wire new free-standing Vitest jobs into e2e-vitest-scenarios.yaml and extend tools/e2e-scenarios/workflow-boundary.mts to validate the shared free-standing job selector.
  • NVIDIA/NemoClaw#5336: Both PRs extend the same Vitest e2e scenario/job wiring and validation paths by adding a new free-standing job (gateway-drift-preflight-vitest vs credential-sanitization-vitest).

Suggested reviewers

  • cv
  • prekshivyas

Poem

🐰 A rabbit wrote scripts in a sandbox so green,

fakes for the gateway and markers unseen.
Preflight said "halt" when the versions don't match,
CI recorded the fail and saved every patch.
Hops of approval — tests pass, scene clean.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately describes the main change: migrating a legacy E2E shell script test to Vitest format, which is the primary objective of this PR.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch e2e-migrate/test-gateway-drift-preflight

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions

github-actions Bot commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

E2E Advisor Recommendation

Required E2E: None
Optional E2E: None

Workflow run

Full advisor summary

E2E Recommendation Advisor

Failed: Could not parse JSON from advisor output; see /home/runner/work/NemoClaw/NemoClaw/artifacts/e2e-advisor/e2e-advisor-raw-output.txt

@github-actions

github-actions Bot commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

Vitest E2E Scenario Recommendation

Required Vitest E2E scenarios: None
Optional Vitest E2E scenarios: None

Workflow run

Full Vitest E2E advisor summary

Vitest E2E Scenario Advisor

Failed: Could not parse JSON from advisor output; see /home/runner/work/NemoClaw/NemoClaw/artifacts/e2e-advisor/e2e-scenario-advisor-raw-output.txt

@github-actions

github-actions Bot commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor

Findings: 0 needs attention, 1 worth checking, 0 nice ideas
Top item: PR review advisor unavailable

Review findings

🛠️ Needs attention

  • None.

🔎 Worth checking

  • PR review advisor unavailable: The automated advisor could not complete: Could not parse JSON from PR review advisor output; see /home/runner/work/NemoClaw/NemoClaw/artifacts/pr-review-advisor/pr-review-advisor-raw-output.txt
    • Recommendation: Re-run the PR Review Advisor or perform a manual review.
    • Evidence: Could not parse JSON from PR review advisor output; see /home/runner/work/NemoClaw/NemoClaw/artifacts/pr-review-advisor/pr-review-advisor-raw-output.txt

🌱 Nice ideas

  • None.
Consider writing more tests for
  • **Runtime validation** — Add or identify targeted runtime/integration validation for the changed behavior; do not report external E2E job pass/fail here.. Runtime/sandbox/infrastructure paths need behavioral runtime validation: .github/workflows/e2e-vitest-scenarios.yaml, tools/e2e-scenarios/workflow-boundary.mts.

Workflow run details

This is an automated advisory review. A human maintainer must make the final merge decision.

Add a focused Vitest replacement for the gateway drift preflight legacy bash guard and wire it into e2e-vitest-scenarios for selective same-runner dispatch.

Refs #5098

Refs #3399

Refs #3423
@jyaunches jyaunches marked this pull request as ready for review June 12, 2026 19:33
@github-actions

Copy link
Copy Markdown
Contributor

Vitest E2E Scenario Results — ✅ All jobs passed

Run: 27438428046
Workflow ref: e2e-migrate/test-gateway-drift-preflight
Requested scenarios: (default — all supported)
Requested jobs: gateway-drift-preflight-vitest
Summary: 2 passed, 0 failed, 22 skipped

Job Result
credential-migration-vitest ⏭️ skipped
credential-sanitization-vitest ⏭️ skipped
double-onboard-vitest ⏭️ skipped
gateway-drift-preflight-vitest ✅ success
gateway-guard-recovery ⏭️ skipped
generate-matrix ✅ success
hermes-e2e-vitest ⏭️ skipped
hermes-root-entrypoint-smoke-vitest ⏭️ skipped
inference-routing-vitest ⏭️ skipped
issue-4434-tui-unreachable-inference-vitest ⏭️ skipped
launchable-smoke-vitest ⏭️ skipped
live-scenarios ⏭️ skipped
model-router-provider-routed-inference-vitest ⏭️ skipped
network-policy-vitest ⏭️ skipped
onboard-negative-paths-vitest ⏭️ skipped
openclaw-tui-chat-correlation-vitest ⏭️ skipped
openshell-version-pin-vitest ⏭️ skipped
rebuild-openclaw-vitest ⏭️ skipped
runtime-overrides-vitest ⏭️ skipped
sandbox-rebuild-vitest ⏭️ skipped
sandbox-survival-vitest ⏭️ skipped
shields-config-vitest ⏭️ skipped
skill-agent-vitest ⏭️ skipped
token-rotation-vitest ⏭️ skipped

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/gateway-drift-preflight.test.ts`:
- Around line 306-313: The current expectSandboxListCalled uses a literal
includes("sandbox list") which will miss invocations that add flags; update the
detection in expectSandboxListCalled to parse the openshell-calls.log lines and
test each line with a regex or startsWith check for the command name (e.g.,
match a pattern like word-boundary "sandbox" followed by whitespace and "list"
so trailing flags are allowed) instead of exact substring equality; change the
called assignment in expectSandboxListCalled to set true when any log line
matches that regex and keep the existing failure message and assertions.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: e7c60aae-139d-4d6d-9c42-6cfcbd6ab16a

📥 Commits

Reviewing files that changed from the base of the PR and between 62f7243 and 3a5f65d.

📒 Files selected for processing (5)
  • .github/workflows/e2e-vitest-scenarios.yaml
  • test/e2e-scenario/support-tests/e2e-scenarios-workflow.test.ts
  • test/gateway-drift-preflight.test.ts
  • tools/e2e-scenarios/free-standing-jobs.env
  • tools/e2e-scenarios/workflow-boundary.mts

Comment thread test/gateway-drift-preflight.test.ts
@github-actions

Copy link
Copy Markdown
Contributor

Vitest E2E Scenario Results — ❌ Some jobs failed

Run: 27445160405
Workflow ref: e2e-migrate/test-gateway-drift-preflight
Requested scenarios: (default — all supported)
Requested jobs: (default — all free-standing when no scenarios are requested)
Summary: 19 passed, 5 failed, 0 skipped

Job Result
credential-migration-vitest ✅ success
credential-sanitization-vitest ❌ failure
double-onboard-vitest ✅ success
gateway-drift-preflight-vitest ✅ success
gateway-guard-recovery ✅ success
generate-matrix ✅ success
hermes-e2e-vitest ✅ success
hermes-root-entrypoint-smoke-vitest ✅ success
inference-routing-vitest ✅ success
issue-4434-tui-unreachable-inference-vitest ❌ failure
launchable-smoke-vitest ✅ success
live-scenarios ❌ failure
model-router-provider-routed-inference-vitest ✅ success
network-policy-vitest ✅ success
onboard-negative-paths-vitest ✅ success
openclaw-tui-chat-correlation-vitest ❌ failure
openshell-version-pin-vitest ✅ success
rebuild-openclaw-vitest ✅ success
runtime-overrides-vitest ✅ success
sandbox-rebuild-vitest ✅ success
sandbox-survival-vitest ✅ success
shields-config-vitest ❌ failure
skill-agent-vitest ✅ success
token-rotation-vitest ✅ success

Failed jobs: credential-sanitization-vitest, issue-4434-tui-unreachable-inference-vitest, live-scenarios, openclaw-tui-chat-correlation-vitest, shields-config-vitest. Check run artifacts for logs.

@github-actions

Copy link
Copy Markdown
Contributor

Vitest E2E Scenario Results — ✅ All jobs passed

Run: 27448614112
Workflow ref: e2e-migrate/test-gateway-drift-preflight
Requested scenarios: (default — all supported)
Requested jobs: gateway-drift-preflight-vitest
Summary: 2 passed, 0 failed, 23 skipped

Job Result
credential-migration-vitest ⏭️ skipped
credential-sanitization-vitest ⏭️ skipped
double-onboard-vitest ⏭️ skipped
gateway-drift-preflight-vitest ✅ success
gateway-guard-recovery ⏭️ skipped
generate-matrix ✅ success
hermes-e2e-vitest ⏭️ skipped
hermes-root-entrypoint-smoke-vitest ⏭️ skipped
inference-routing-vitest ⏭️ skipped
issue-4434-tui-unreachable-inference-vitest ⏭️ skipped
launchable-smoke-vitest ⏭️ skipped
live-scenarios ⏭️ skipped
messaging-providers-vitest ⏭️ skipped
model-router-provider-routed-inference-vitest ⏭️ skipped
network-policy-vitest ⏭️ skipped
onboard-negative-paths-vitest ⏭️ skipped
openclaw-tui-chat-correlation-vitest ⏭️ skipped
openshell-version-pin-vitest ⏭️ skipped
rebuild-openclaw-vitest ⏭️ skipped
runtime-overrides-vitest ⏭️ skipped
sandbox-rebuild-vitest ⏭️ skipped
sandbox-survival-vitest ⏭️ skipped
shields-config-vitest ⏭️ skipped
skill-agent-vitest ⏭️ skipped
token-rotation-vitest ⏭️ skipped

@github-actions

Copy link
Copy Markdown
Contributor

Vitest E2E Scenario Results — ✅ All jobs passed

Run: 27452322362
Workflow ref: e2e-migrate/test-gateway-drift-preflight
Requested scenarios: (default — all supported)
Requested jobs: gateway-drift-preflight-vitest
Summary: 2 passed, 0 failed, 23 skipped

Job Result
credential-migration-vitest ⏭️ skipped
credential-sanitization-vitest ⏭️ skipped
double-onboard-vitest ⏭️ skipped
gateway-drift-preflight-vitest ✅ success
gateway-guard-recovery ⏭️ skipped
generate-matrix ✅ success
hermes-e2e-vitest ⏭️ skipped
hermes-root-entrypoint-smoke-vitest ⏭️ skipped
inference-routing-vitest ⏭️ skipped
issue-4434-tui-unreachable-inference-vitest ⏭️ skipped
launchable-smoke-vitest ⏭️ skipped
live-scenarios ⏭️ skipped
messaging-providers-vitest ⏭️ skipped
model-router-provider-routed-inference-vitest ⏭️ skipped
network-policy-vitest ⏭️ skipped
onboard-negative-paths-vitest ⏭️ skipped
openclaw-tui-chat-correlation-vitest ⏭️ skipped
openshell-version-pin-vitest ⏭️ skipped
rebuild-openclaw-vitest ⏭️ skipped
runtime-overrides-vitest ⏭️ skipped
sandbox-rebuild-vitest ⏭️ skipped
sandbox-survival-vitest ⏭️ skipped
shields-config-vitest ⏭️ skipped
skill-agent-vitest ⏭️ skipped
token-rotation-vitest ⏭️ skipped

@cv cv merged commit 85a72aa into main Jun 13, 2026
42 checks passed
@cv cv deleted the e2e-migrate/test-gateway-drift-preflight branch June 13, 2026 02:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: e2e End-to-end tests, nightly failures, or validation infrastructure chore Build, CI, dependency, or tooling maintenance v0.0.65 Release target

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants