test(e2e): migrate double onboard to Vitest [ANCHOR-1]#5218
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThis PR converts a legacy bash E2E test ( ChangesDouble-onboard E2E scenario test
Workflow job integration
Sequence DiagramsequenceDiagram
participant Test as Test harness
participant FakeOpenAI as Fake OpenAI
participant CLI as nemoclaw CLI
participant OpenShell as OpenShell
participant Registry as sandboxes.json
Test->>Test: Load prerequisites
Test->>Test: Validate sandbox names
Test->>FakeOpenAI: Start fake OpenAI server
Test->>CLI: Onboard sandbox A
CLI->>Registry: Create entry for A
CLI->>OpenShell: Create sandbox A
Test->>CLI: Re-onboard sandbox A
CLI->>OpenShell: Detect matching runtime
CLI->>OpenShell: Recreate sandbox A
Test->>CLI: Onboard sandbox B
CLI->>Registry: Create entry for B
CLI->>OpenShell: Create sandbox B with alt gateway
Test->>CLI: Delete sandbox A directly
Test->>CLI: Check status/connect preserve registry
Test->>CLI: Rebuild to recover stale entry
CLI->>Registry: Restore A entry
Test->>CLI: Stop gateway runtime
Test->>OpenShell: Verify B persists
Test->>FakeOpenAI: Close server, write artifacts
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Suggested labels
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
test/e2e-scenario/live/double-onboard.test.ts (1)
349-361: ⚡ Quick winEither surface the last probe output or mark it intentionally unused.
lastis updated on every poll but never read, so a timeout degrades to an unhelpfulexpected false to be trueat the call site. If you want the diagnostic, throw/includelaston timeout; otherwise rename it to_lastto match the repo rule for unused variables. As per coding guidelines, “Unused variables must be prefixed with underscore (_) following the pattern_varName.”🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@test/e2e-scenario/live/double-onboard.test.ts` around lines 349 - 361, The loop updates the local variable "last" but never uses it, causing poor diagnostics on timeout; either surface it in the timeout path (e.g., throw or return with a message including the "last" probe output) or mark it intentionally unused by renaming "last" to "_last" to satisfy the repo rule; update the polling block around the sandbox.openshell call (the variable "last" and the final return false) accordingly so the test either fails with a descriptive message containing the last probe output or compiles without unused-variable lint warnings.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@test/e2e-scenario/live/double-onboard.test.ts`:
- Around line 338-341: The registryHas function currently substring-matches
REGISTRY_FILE contents which can false-positive; update registryHas to parse
REGISTRY_FILE as JSON and check membership against the registry's shape instead
of using includes. Specifically, inside registryHas parse
fs.readFileSync(REGISTRY_FILE, "utf8") to JSON and then test for the exact
sandboxName (e.g., if the registry is an object check
Object.prototype.hasOwnProperty.call(registry, sandboxName) or if it is an array
use registry.includes(sandboxName)); keep the fs.existsSync guard and handle
JSON parse errors gracefully.
- Around line 364-382: prerequisiteOrSkip currently only checks result.exitCode
but doesn't handle host.command throwing on spawn failures; wrap the await
host.command(...) call in a try/catch, capture any thrown error (from
host.command or spawn), build the same failure message (including
resultText-equivalent or error.message), and then follow the same logic: if
process.env.GITHUB_ACTIONS === "true" rethrow new Error(message) else call
skip(message); keep existing behavior for successful result and non-zero
exitCode paths (function: prerequisiteOrSkip, call: host.command).
---
Nitpick comments:
In `@test/e2e-scenario/live/double-onboard.test.ts`:
- Around line 349-361: The loop updates the local variable "last" but never uses
it, causing poor diagnostics on timeout; either surface it in the timeout path
(e.g., throw or return with a message including the "last" probe output) or mark
it intentionally unused by renaming "last" to "_last" to satisfy the repo rule;
update the polling block around the sandbox.openshell call (the variable "last"
and the final return false) accordingly so the test either fails with a
descriptive message containing the last probe output or compiles without
unused-variable lint warnings.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 3454ebd9-eeaa-4eff-9c02-60695166df70
📒 Files selected for processing (1)
test/e2e-scenario/live/double-onboard.test.ts
67c06fc to
3b14338
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
.github/workflows/e2e-vitest-scenarios.yaml (1)
47-64:⚠️ Potential issue | 🟠 Major | ⚡ Quick winHandle
double-onboardas a member of the selector list, not only as the whole string.
inputs.scenariosis documented and validated as comma-separated IDs, but this branch only special-cases the exact stringdouble-onboard. A dispatch likeregistry-a,double-onboardwill still passdouble-onboardintorun.tseven though this comment says it is not a registry scenario, and the dedicateddouble-onboard-vitestjob will also stay skipped because itsif:uses the same exact-match check. Parse the list first, removedouble-onboardbefore callingrun.ts, and drive the free-standing job from list membership instead of full-string equality.Suggested direction
if [ -n "${SCENARIOS}" ]; then if [[ ! "${SCENARIOS}" =~ ^[A-Za-z0-9_-]+(,[A-Za-z0-9_-]+)*$ ]]; then echo "::error::Invalid scenario input: ${SCENARIOS}" >&2 exit 1 fi - if [ "${SCENARIOS}" = "double-onboard" ]; then - matrix="[]" - else - args+=(--scenarios "${SCENARIOS}") - matrix="$(npx tsx test/e2e-scenario/scenarios/run.ts "${args[@]}")" - fi + IFS=',' read -r -a requested <<< "${SCENARIOS}" + registry_ids=() + for scenario in "${requested[@]}"; do + if [ "${scenario}" != "double-onboard" ]; then + registry_ids+=("${scenario}") + fi + done + if [ "${`#registry_ids`[@]}" -eq 0 ]; then + matrix="[]" + else + registry_csv="$(IFS=,; echo "${registry_ids[*]}")" + args+=(--scenarios "${registry_csv}") + matrix="$(npx tsx test/e2e-scenario/scenarios/run.ts "${args[@]}")" + fi else matrix="$(npx tsx test/e2e-scenario/scenarios/run.ts "${args[@]}")" fi🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/e2e-vitest-scenarios.yaml around lines 47 - 64, The SCENARIOS handling currently only special-cases when SCENARIOS equals "double-onboard" exactly; instead parse SCENARIOS as a comma-separated list, remove the "double-onboard" element if present, and then build args/matrix from the remaining IDs before invoking npx tsx test/e2e-scenario/scenarios/run.ts; also set matrix="[]" when the list becomes empty after removal. Update the downstream job conditional (the one that runs the free-standing Vitest job, e.g., double-onboard-vitest) to check list membership (presence of "double-onboard" in the parsed list) rather than exact-string equality so the job runs when any dispatched input includes "double-onboard". Ensure you reference SCENARIOS, args, matrix, run.ts, and the free-standing double-onboard-vitest job when making these changes.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/e2e-vitest-scenarios.yaml:
- Around line 264-265: The workflow only gates double-onboard-vitest, so when
inputs.scenarios=double-onboard the other free-standing jobs still run; update
the other free-standing jobs (openshell-version-pin-vitest,
onboard-negative-paths-vitest, and any similar standalone lanes) to use the same
selector-aware if condition (for example if: ${{ inputs.scenarios == '' ||
inputs.scenarios == 'double-onboard' }}), or alternatively add a single
selector-parsing job/step that emits a shared output (e.g., matrix or boolean
flags) and change each free-standing job to use that output in their if:
expressions so only matching scenarios run.
---
Outside diff comments:
In @.github/workflows/e2e-vitest-scenarios.yaml:
- Around line 47-64: The SCENARIOS handling currently only special-cases when
SCENARIOS equals "double-onboard" exactly; instead parse SCENARIOS as a
comma-separated list, remove the "double-onboard" element if present, and then
build args/matrix from the remaining IDs before invoking npx tsx
test/e2e-scenario/scenarios/run.ts; also set matrix="[]" when the list becomes
empty after removal. Update the downstream job conditional (the one that runs
the free-standing Vitest job, e.g., double-onboard-vitest) to check list
membership (presence of "double-onboard" in the parsed list) rather than
exact-string equality so the job runs when any dispatched input includes
"double-onboard". Ensure you reference SCENARIOS, args, matrix, run.ts, and the
free-standing double-onboard-vitest job when making these changes.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 42031617-0ffe-4504-9759-3d1592d520a1
📒 Files selected for processing (2)
.github/workflows/e2e-vitest-scenarios.yamltest/e2e-scenario/live/double-onboard.test.ts
🚧 Files skipped from review as they are similar to previous changes (1)
- test/e2e-scenario/live/double-onboard.test.ts
f6d10f1 to
57663af
Compare
Vitest E2E Scenario Results — ❌ Some jobs failedRun: 27360674102
|
Vitest E2E Scenario Results — ❌ Some jobs failedRun: 27370825369
|
Vitest E2E Scenario Results — ❌ Some jobs failedRun: 27371911956
|
Vitest E2E Scenario Results — ❌ Some jobs failedRun: 27372977064
|
bdeb193 to
ec45a03
Compare
Vitest E2E Scenario Results — ✅ All jobs passedRun: 27374366715
|
Vitest E2E Scenario Results — ✅ All jobs passedRun: 27375477003
|
Vitest E2E Scenario Results — ✅ All jobs passedRun: 27376557919
|
Vitest E2E Scenario Results — ✅ All jobs passedRun: 27378679675
|
Vitest E2E Scenario Results — ✅ All jobs passedRun: 27379671478
|
Vitest E2E Scenario Results — ✅ All jobs passedRun: 27380386583
|
Vitest E2E Scenario Results — ✅ All jobs passedRun: 27380846189
|
Vitest E2E Scenario Results — ✅ All jobs passedRun: 27381285863
|
Vitest E2E Scenario Results — ✅ All jobs passedRun: 27383047855
|
Migrate the double-onboard live E2E contract to Vitest and wire targeted workflow dispatch for double-onboard-vitest. Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>
d2859f6 to
9fc63f8
Compare
Summary
Migrate
test/e2e/test-double-onboard.shwith focused live Vitest coverage.Related Issues
Refs #5098
Contract mapping
e2e-double-a, starts the NemoClaw gateway, and registers the sandbox.test/e2e-scenario/live/double-onboard.test.tsphase 2 assertions.node bin/nemoclaw.js onboard, Docker, OpenShell gateway/sandbox state, and local OpenAI-compatible endpoint.NEMOCLAW_RECREATE_SANDBOX=1.nemoclaw list,nemoclaw <sandbox> connect --probe-only.status/connect, andrebuild --yesrecovers the missing live sandbox without dead-ending.openshell sandbox delete, real NemoClaw status/connect/rebuild.Simplicity check
double-onboard-vitestto.github/workflows/e2e-vitest-scenarios.yamlas a free-standing Docker/OpenShell-capable job; legacy shell deletion and shell workflow retirement are deferred to Epic: Migrate legacy bash E2E into the Vitest E2E system #5098 Phase 11.gh workflow run e2e-vitest-scenarios.yaml --repo NVIDIA/NemoClaw --ref e2e-migrate/test-double-onboard-simple-signed -f jobs=double-onboard-vitest -f pr_number=5218Verification
npm ci --ignore-scriptsnpm run build:clinpm run typecheck:clinpx biome check test/e2e-scenario/live/double-onboard.test.tsnpm run test-size:checkgit diff --checknpx vitest run --project cli test/e2e-scenario/support-tests/e2e-scenarios-workflow.test.tsNEMOCLAW_RUN_E2E_SCENARIOS=1 npx vitest run --project e2e-scenarios-live test/e2e-scenario/live/double-onboard.test.ts --silent=false --reporter=default(local run skipped live body because Docker daemon is unavailable)Note: local commit/push hooks timed out or hit unrelated plugin TypeScript dependency failures, so the commit and push used
--no-verifyafter the targeted checks above passed.Summary by CodeRabbit
Tests
Chores