test(e2e): cover dashboard port auto-alloc in nightly by cjagwani · Pull Request #2455 · NVIDIA/NemoClaw

cjagwani · 2026-04-24T21:00:33Z

Summary

Adds three #2174 regression assertions to the existing test-double-onboard.sh and wires it into nightly-e2e.yaml as a new double-onboard-e2e job. Without these, a regression of the auto-alloc path (crashing second onboards or silently stealing the first sandbox's forward) would ship to main with only unit tests as a guard.

Changes

test/e2e/test-double-onboard.sh — three new assertions added to the existing Phase 4 (second-sandbox onboard):
- Second onboard's stdout must include allocated port (confirms auto-alloc fired)
- nemoclaw list must show two distinct dashboard: http://... lines for the two sandboxes
- Existing "first sandbox still alive" + "port 18789 conflict not detected" assertions preserved
.github/workflows/nightly-e2e.yaml — new double-onboard-e2e job that runs the script. Uses the script's built-in fake OpenAI-compatible endpoint so no NVIDIA_API_KEY is needed. Wired into notify-on-failure needs: list so a regression auto-opens a GitHub issue.

Type of Change

Code change (feature, bug fix, or refactor)

Verification

npx prek run --all-files passes (YAML validates, shellcheck clean)
bash -n test/e2e/test-double-onboard.sh — syntax check passes
Script runs locally against a live sandbox pair once fix(onboard): auto-allocate dashboard port on multi-sandbox conflict #2454 lands
Tests added or updated for new or changed behavior
No secrets, API keys, or credentials committed

Notes for reviewer

This is the first multi-sandbox E2E in nightly. Pattern is copy-ready for future dual-sandbox scenarios ([Sandbox][Recovery] Pod restart leaves OpenClaw gateway and dashboard port-forward dead; recovery is a side-effect of nemoclaw connect #2042, [Brev Launchable] Openclaw UI link is unhealthy #1178).
The new job's if: gate matches existing jobs (github.repository == 'NVIDIA/NemoClaw') so forks don't pay the runner cost.
Split from fix(onboard): auto-allocate dashboard port on multi-sandbox conflict #2454 to keep that PR focused on the minimum [All Platform][Onboard][Regression] Second onboard crashes with unhandled exception and leaks ghost sandbox when dashboard port 18789 is already forwarded #2174 close.

AI Disclosure

AI-assisted — tool: Claude Code

Signed-off-by: Charan Jagwani cjagwani@nvidia.com

Summary by CodeRabbit

Tests
- Added validation for concurrent sandbox onboarding with dashboard port collision detection.
Chores
- Enhanced nightly automated testing with improved failure tracking and artifact collection.

Extends test/e2e/test-double-onboard.sh with three #2174 regression checks: - Second-sandbox onboard output must log "allocated port" (auto-alloc fired) - nemoclaw list must show two distinct dashboard ports for A and B The existing script already exercises the two-sandbox flow (Phase 4) but didn't assert anything port-specific — a silent regression of the auto-alloc path would have passed. Also wires the script into nightly-e2e.yaml as a new double-onboard-e2e job. Uses the script's fake OpenAI endpoint so no NVIDIA_API_KEY is required. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>

coderabbitai · 2026-04-24T21:00:47Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 7fca9582-2bbc-4ccc-b7d0-5361487290c1

📥 Commits

Reviewing files that changed from the base of the PR and between 260b237 and 65b9b36.

📒 Files selected for processing (2)

.github/workflows/nightly-e2e.yaml
test/e2e/test-double-onboard.sh

📝 Walkthrough

Walkthrough

A new nightly E2E test job is added to validate concurrent sandbox onboarding without dashboard port collisions, integrated into the existing CI failure notification mechanism using a dedicated test script and local fake OpenAI-compatible endpoint.

Changes

Cohort / File(s)	Summary
CI Workflow Enhancement `.github/workflows/nightly-e2e.yaml`	New `double-onboard-e2e` GitHub Actions job that executes concurrent onboarding test, produces artifact logs on failure, and connects to existing failure notification pipeline via `needs` dependency.
Test Implementation `test/e2e/test-double-onboard.sh`	New bash test script that validates two sandboxes onboard simultaneously with distinct dashboard ports, asserting allocation messages and deduplicating port extraction from `nemoclaw list` output.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Poem

🐰 Two sandboxes hop and play,
Each claiming ports their own way,
Concurrent dance, no collisions here,
The ports stay unique, crystal clear! 🎉

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'test(e2e): cover dashboard port auto-alloc in nightly' accurately describes the main change: adding E2E test coverage for dashboard port auto-allocation in the nightly workflow.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch test/e2e-double-onboard-port-alloc

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

jyaunches · 2026-04-29T22:12:35Z

Superseded by #2709.

This PR had two issues that made reworking it more effort than a fresh branch:

Assertion string mismatch — "allocated port" doesn't appear anywhere in the codebase's runtime output. The actual message from ensureDashboardForward is "is taken. Using port".
Workflow changes superseded — The double-onboard-e2e job already exists on main (wired in via other PRs, fixed by fix(e2e): fix double-onboard assertion strings, CI timeout, and per-phase diagnostics #2658/fix(e2e): increase double-onboard script-level timeout from 900s to 4800s #2676), making these changes a merge conflict with no value.

#2709 carries forward the two test assertions (with the corrected grep string) on a fresh branch from current main. Thanks for the original work identifying the coverage gap, @cjagwani!

) ## Summary Adds two regression assertions to Phase 4 of `test-double-onboard.sh` that verify the #2174 auto-allocation fix works end-to-end: 1. **Log check** — Second-sandbox onboard output contains `"is taken. Using port"` (confirms `ensureDashboardForward` auto-allocated) 2. **List check** — `nemoclaw list` shows two distinct `dashboard: http://127.0.0.1:<port>` entries for both sandboxes ## Related - Guards against regression of #2174 (fix landed via #2411 + #2627) - Supersedes #2455 (which had a stale assertion string `"allocated port"` that doesn't match any runtime output, and conflicting workflow changes already on main) ## Changes - `test/e2e/test-double-onboard.sh` — 21 lines added between the existing #849 regression check and Phase 5 ## Why this supersedes #2455 | Issue | #2455 | This PR | |-------|-------|---------| | Assertion string | `"allocated port"` (doesn't exist in codebase) | `"is taken. Using port"` (matches `ensureDashboardForward` line ~6600) | | Workflow changes | Adds a `double-onboard-e2e` job that already exists on main → merge conflict | None needed — job already runs this script | | Branch freshness | 95 commits behind main | Based on current main | ## Type of Change - [x] Code change (feature, bug fix, or refactor) ## Verification - [x] `bash -n test/e2e/test-double-onboard.sh` — syntax check passes - [x] shellcheck clean - [x] No workflow changes needed — `double-onboard-e2e` nightly job already runs this script - [x] No secrets, API keys, or credentials committed ## AI Disclosure - [x] AI-assisted — tool: Claude (pi agent) --- Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>  ## Summary by CodeRabbit * **Tests** * Added end-to-end test assertions for multi-sandbox onboarding scenarios * Validates dashboard port auto-allocation prevents conflicts and maintains isolation across concurrent sandbox environments  Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>

…rd (NVIDIA#2709) ## Summary Adds two regression assertions to Phase 4 of `test-double-onboard.sh` that verify the NVIDIA#2174 auto-allocation fix works end-to-end: 1. **Log check** — Second-sandbox onboard output contains `"is taken. Using port"` (confirms `ensureDashboardForward` auto-allocated) 2. **List check** — `nemoclaw list` shows two distinct `dashboard: http://127.0.0.1:<port>` entries for both sandboxes ## Related - Guards against regression of NVIDIA#2174 (fix landed via NVIDIA#2411 + NVIDIA#2627) - Supersedes NVIDIA#2455 (which had a stale assertion string `"allocated port"` that doesn't match any runtime output, and conflicting workflow changes already on main) ## Changes - `test/e2e/test-double-onboard.sh` — 21 lines added between the existing NVIDIA#849 regression check and Phase 5 ## Why this supersedes NVIDIA#2455 | Issue | NVIDIA#2455 | This PR | |-------|-------|---------| | Assertion string | `"allocated port"` (doesn't exist in codebase) | `"is taken. Using port"` (matches `ensureDashboardForward` line ~6600) | | Workflow changes | Adds a `double-onboard-e2e` job that already exists on main → merge conflict | None needed — job already runs this script | | Branch freshness | 95 commits behind main | Based on current main | ## Type of Change - [x] Code change (feature, bug fix, or refactor) ## Verification - [x] `bash -n test/e2e/test-double-onboard.sh` — syntax check passes - [x] shellcheck clean - [x] No workflow changes needed — `double-onboard-e2e` nightly job already runs this script - [x] No secrets, API keys, or credentials committed ## AI Disclosure - [x] AI-assisted — tool: Claude (pi agent) --- Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>  ## Summary by CodeRabbit * **Tests** * Added end-to-end test assertions for multi-sandbox onboarding scenarios * Validates dashboard port auto-allocation prevents conflicts and maintains isolation across concurrent sandbox environments  Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>

jyaunches mentioned this pull request Apr 25, 2026

feat(onboard): auto-allocate dashboard port for multi-sandbox #2411

Merged

6 tasks

wscurran added CI/CD dependencies Pull requests that update a dependency file labels Apr 27, 2026

jyaunches self-assigned this Apr 28, 2026

This was referenced Apr 28, 2026

test(e2e): add GPU double-onboard E2E test for Ollama proxy token consistency #2606

Closed

test(e2e): add #2174 port auto-alloc assertions to double-onboard #2709

Merged

jyaunches closed this Apr 29, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test(e2e): cover dashboard port auto-alloc in nightly#2455

test(e2e): cover dashboard port auto-alloc in nightly#2455
cjagwani wants to merge 1 commit into
mainfrom
test/e2e-double-onboard-port-alloc

cjagwani commented Apr 24, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Apr 24, 2026 •

edited

Loading

Walkthrough

Changes

Estimated code review effort

Poem

Uh oh!

jyaunches commented Apr 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

cjagwani commented Apr 24, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Related

Changes

Type of Change

Verification

Notes for reviewer

AI Disclosure

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Apr 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Poem

Uh oh!

jyaunches commented Apr 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

cjagwani commented Apr 24, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Apr 24, 2026 •

edited

Loading