test(e2e): add #2174 port auto-alloc assertions to double-onboard by jyaunches · Pull Request #2709 · NVIDIA/NemoClaw

jyaunches · 2026-04-29T22:12:22Z

Summary

Adds two regression assertions to Phase 4 of test-double-onboard.sh that verify the #2174 auto-allocation fix works end-to-end:

Log check — Second-sandbox onboard output contains "is taken. Using port" (confirms ensureDashboardForward auto-allocated)
List check — nemoclaw list shows two distinct dashboard: http://127.0.0.1:<port> entries for both sandboxes

Guards against regression of [All Platform][Onboard][Regression] Second onboard crashes with unhandled exception and leaks ghost sandbox when dashboard port 18789 is already forwarded #2174 (fix landed via feat(onboard): auto-allocate dashboard port for multi-sandbox #2411 + fix(onboard): roll back orphaned sandbox when dashboard alloc fails (#2174 followup) #2627)
Supersedes test(e2e): cover dashboard port auto-alloc in nightly #2455 (which had a stale assertion string "allocated port" that doesn't match any runtime output, and conflicting workflow changes already on main)

Changes

test/e2e/test-double-onboard.sh — 21 lines added between the existing Cannot connect to the first sandbox after creating the second sandbox #849 regression check and Phase 5

Why this supersedes #2455

Issue	#2455	This PR
Assertion string	`"allocated port"` (doesn't exist in codebase)	`"is taken. Using port"` (matches `ensureDashboardForward` line ~6600)
Workflow changes	Adds a `double-onboard-e2e` job that already exists on main → merge conflict	None needed — job already runs this script
Branch freshness	95 commits behind main	Based on current main

Type of Change

Code change (feature, bug fix, or refactor)

Verification

bash -n test/e2e/test-double-onboard.sh — syntax check passes
shellcheck clean
No workflow changes needed — double-onboard-e2e nightly job already runs this script
No secrets, API keys, or credentials committed

AI Disclosure

AI-assisted — tool: Claude (pi agent)

Signed-off-by: Julie Yaunches jyaunches@nvidia.com

Summary by CodeRabbit

Tests
- Added end-to-end test assertions for multi-sandbox onboarding scenarios
- Validates dashboard port auto-allocation prevents conflicts and maintains isolation across concurrent sandbox environments

Adds two assertions to Phase 4 of test-double-onboard.sh that verify: - Second-sandbox onboard logs port auto-allocation ("is taken. Using port") - nemoclaw list shows two distinct dashboard ports for both sandboxes These guard against silent regression of the auto-alloc path introduced in #2411. The double-onboard-e2e nightly job already runs this script — no workflow changes needed. Supersedes #2455 (which had a stale assertion string and conflicting workflow changes). Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>

coderabbitai · 2026-04-29T22:12:35Z

📝 Walkthrough

Walkthrough

Test enhancements add e2e assertions for regression #2174, validating dashboard port auto-allocation behavior when creating a third onboard sandbox. The test verifies that exactly two distinct dashboard ports are registered, confirming port collision prevention between separate sandbox instances.

Changes

Cohort / File(s)	Summary
E2E Test Assertions `test/e2e/test-double-onboard.sh`	Adds regression test validations for dashboard port auto-allocation, including log message verification and port uniqueness checks across multiple sandboxes.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~18 minutes

Poem

🐰 Three sandboxes leap through the testing ground,
Each dashboard port dancing without a sound,
No collision, no crash—just ports so divine,
The regression is caught at line twenty-nine! 🌟

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title directly and specifically describes the main change: adding regression test assertions for issue `#2174` (port auto-allocation) to the double-onboard e2e test script.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch test/e2e-2174-port-alloc-assertions

_{Review rate limit: 9/10 reviews remaining, refill in 6 minutes.}

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@test/e2e/test-double-onboard.sh`:
- Around line 418-429: The assertion counts all global dashboard entries;
restrict it to sandboxes created in this test by filtering the list output for
the sandbox identifiers you spawned earlier (e.g., use the variables that hold
the sandbox names/IDs instead of the whole run_nemoclaw list output). After
run_nemoclaw list >"$LIST_LOG" capture list_output as you do, then narrow it
with grep -E "(${SANDBOX_NAME1}|${SANDBOX_NAME2})" (or the variables that hold
the test sandbox IDs) before computing dashboard_ports_in_list and
distinct_count so dashboard_ports_in_list only contains ports for the sandboxes
under test (keep using the existing variables LIST_LOG, list_output,
dashboard_ports_in_list, distinct_count). Ensure the failure message still
prints the filtered dashboard_ports_in_list for debugging.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: bdaa3c63-c0c0-420b-b527-26da412b5db8

📥 Commits

Reviewing files that changed from the base of the PR and between 02c55e7 and 9720b9d.

📒 Files selected for processing (1)

test/e2e/test-double-onboard.sh

coderabbitai · 2026-04-29T22:15:00Z

+LIST_LOG="$(mktemp)"
+run_nemoclaw list >"$LIST_LOG" 2>&1 || true
+list_output="$(cat "$LIST_LOG")"
+rm -f "$LIST_LOG"
+
+dashboard_ports_in_list="$(grep -oE 'dashboard: http://127\.0\.0\.1:[0-9]+' <<<"$list_output" | awk -F: '{print $NF}' | sort -u)"
+distinct_count="$(wc -l <<<"$dashboard_ports_in_list" | tr -d ' ')"
+if [ "$distinct_count" = "2" ]; then
+  pass "nemoclaw list shows two distinct dashboard ports (#2174)"
+else
+  fail "nemoclaw list did not show two distinct dashboard ports (got $distinct_count: $(tr '\n' ' ' <<<"$dashboard_ports_in_list"))"
+fi


⚠️ Potential issue | 🟠 Major

Scope the list assertion to the sandboxes under test.

run_nemoclaw list is global, so counting every dashboard: entry can fail or pass because of unrelated sandboxes or stale registry state. That weakens the #2174 regression check.

Suggested cleanup

LIST_LOG="$(mktemp)" -run_nemoclaw list >"$LIST_LOG" 2>&1 || true +if ! run_nemoclaw list >"$LIST_LOG" 2>&1; then + fail "nemoclaw list failed" +fi list_output="$(cat "$LIST_LOG")" rm -f "$LIST_LOG"

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

LIST_LOG="$(mktemp)"

run_nemoclaw list >"$LIST_LOG" 2>&1 || true

list_output="$(cat "$LIST_LOG")"

rm -f "$LIST_LOG"

dashboard_ports_in_list="$(grep -oE 'dashboard: http://127\.0\.0\.1:[0-9]+' <<<"$list_output" | awk -F: '{print $NF}' | sort -u)"

distinct_count="$(wc -l <<<"$dashboard_ports_in_list" | tr -d ' ')"

if [ "$distinct_count" = "2" ]; then

pass "nemoclaw list shows two distinct dashboard ports (#2174)"

else

fail "nemoclaw list did not show two distinct dashboard ports (got $distinct_count: $(tr '\n' ' ' <<<"$dashboard_ports_in_list"))"

fi

LIST_LOG="$(mktemp)"

if ! run_nemoclaw list >"$LIST_LOG" 2>&1; then

fail "nemoclaw list failed"

fi

list_output="$(cat "$LIST_LOG")"

rm -f "$LIST_LOG"

dashboard_ports_in_list="$(grep -oE 'dashboard: http://127\.0\.0\.1:[0-9]+' <<<"$list_output" | awk -F: '{print $NF}' | sort -u)"

distinct_count="$(wc -l <<<"$dashboard_ports_in_list" | tr -d ' ')"

if [ "$distinct_count" = "2" ]; then

pass "nemoclaw list shows two distinct dashboard ports (`#2174`)"

else

fail "nemoclaw list did not show two distinct dashboard ports (got $distinct_count: $(tr '\n' ' ' <<<"$dashboard_ports_in_list"))"

fi

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed. In `@test/e2e/test-double-onboard.sh` around lines 418 - 429, The assertion counts all global dashboard entries; restrict it to sandboxes created in this test by filtering the list output for the sandbox identifiers you spawned earlier (e.g., use the variables that hold the sandbox names/IDs instead of the whole run_nemoclaw list output). After run_nemoclaw list >"$LIST_LOG" capture list_output as you do, then narrow it with grep -E "(${SANDBOX_NAME1}|${SANDBOX_NAME2})" (or the variables that hold the test sandbox IDs) before computing dashboard_ports_in_list and distinct_count so dashboard_ports_in_list only contains ports for the sandboxes under test (keep using the existing variables LIST_LOG, list_output, dashboard_ports_in_list, distinct_count). Ensure the failure message still prints the filtered dashboard_ports_in_list for debugging.

…rd (NVIDIA#2709) ## Summary Adds two regression assertions to Phase 4 of `test-double-onboard.sh` that verify the NVIDIA#2174 auto-allocation fix works end-to-end: 1. **Log check** — Second-sandbox onboard output contains `"is taken. Using port"` (confirms `ensureDashboardForward` auto-allocated) 2. **List check** — `nemoclaw list` shows two distinct `dashboard: http://127.0.0.1:<port>` entries for both sandboxes ## Related - Guards against regression of NVIDIA#2174 (fix landed via NVIDIA#2411 + NVIDIA#2627) - Supersedes NVIDIA#2455 (which had a stale assertion string `"allocated port"` that doesn't match any runtime output, and conflicting workflow changes already on main) ## Changes - `test/e2e/test-double-onboard.sh` — 21 lines added between the existing NVIDIA#849 regression check and Phase 5 ## Why this supersedes NVIDIA#2455 | Issue | NVIDIA#2455 | This PR | |-------|-------|---------| | Assertion string | `"allocated port"` (doesn't exist in codebase) | `"is taken. Using port"` (matches `ensureDashboardForward` line ~6600) | | Workflow changes | Adds a `double-onboard-e2e` job that already exists on main → merge conflict | None needed — job already runs this script | | Branch freshness | 95 commits behind main | Based on current main | ## Type of Change - [x] Code change (feature, bug fix, or refactor) ## Verification - [x] `bash -n test/e2e/test-double-onboard.sh` — syntax check passes - [x] shellcheck clean - [x] No workflow changes needed — `double-onboard-e2e` nightly job already runs this script - [x] No secrets, API keys, or credentials committed ## AI Disclosure - [x] AI-assisted — tool: Claude (pi agent) --- Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>  ## Summary by CodeRabbit * **Tests** * Added end-to-end test assertions for multi-sandbox onboarding scenarios * Validates dashboard port auto-allocation prevents conflicts and maintains isolation across concurrent sandbox environments  Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>

jyaunches mentioned this pull request Apr 29, 2026

test(e2e): cover dashboard port auto-alloc in nightly #2455

Closed

7 tasks

coderabbitai Bot reviewed Apr 29, 2026

View reviewed changes

cv approved these changes Apr 29, 2026

View reviewed changes

cv merged commit 643a984 into main Apr 29, 2026
19 checks passed

This was referenced Apr 29, 2026

refactor(sandbox): default to mutable config, make shields opt-in #2227

Merged

feat(channels): replace Slack token carve-out with Bolt-shape rewriter #2630

Merged

wscurran added the chore Build, CI, dependency, or tooling maintenance label Jun 8, 2026

jyaunches deleted the test/e2e-2174-port-alloc-assertions branch June 12, 2026 13:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test(e2e): add #2174 port auto-alloc assertions to double-onboard#2709

test(e2e): add #2174 port auto-alloc assertions to double-onboard#2709
cv merged 1 commit into
mainfrom
test/e2e-2174-port-alloc-assertions

jyaunches commented Apr 29, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Apr 29, 2026 •

edited

Loading

Walkthrough

Changes

Estimated code review effort

Poem

Uh oh!

coderabbitai Bot left a comment

Uh oh!

coderabbitai Bot Apr 29, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

jyaunches commented Apr 29, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Related

Changes

Why this supersedes #2455

Type of Change

Verification

AI Disclosure

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Apr 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Poem

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot Apr 29, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jyaunches commented Apr 29, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Apr 29, 2026 •

edited

Loading