Skip to content

test(e2e): migrate dashboard remote bind#5186

Merged
cv merged 2 commits into
mainfrom
e2e-migrate/dashboard-remote-bind-simple
Jun 11, 2026
Merged

test(e2e): migrate dashboard remote bind#5186
cv merged 2 commits into
mainfrom
e2e-migrate/dashboard-remote-bind-simple

Conversation

@jyaunches

@jyaunches jyaunches commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

Summary

Migrate the active dashboard-remote-bind Brev suite path to a small live Vitest test while keeping test/e2e/test-dashboard-remote-bind.sh in place per maintainer request.

This uses Carlos's PR #5133 as the seed, but keeps only the direct dashboard remote-bind migration and drops the stale #5126-era framework/ledger/workflow noise.

Related Issues

Refs #5098
Refs #5133

Contract mapping

  • Legacy assertion: required CLIs are available before the remote-bind check runs.
    • Replacement: test/e2e-scenario/live/dashboard-remote-bind.test.ts runs command -v nemoclaw && command -v openshell.
    • Boundary preserved: real remote host shell/PATH.
  • Legacy assertion: dashboard forward is restarted before checking bind behavior.
    • Replacement: the Vitest test runs openshell forward stop <dashboardPort> then nemoclaw <sandbox> connect.
    • Boundary preserved: real OpenShell forward + real NemoClaw connect path.
  • Legacy assertion: NEMOCLAW_DASHBOARD_BIND=0.0.0.0 is honored.
    • Replacement: host.nemoclaw([...], { env: { NEMOCLAW_DASHBOARD_BIND: "0.0.0.0" } }).
    • Boundary preserved: real environment-driven CLI behavior.
  • Legacy assertion: loopback-only binding is rejected/avoided and all-interface bind is proven.
    • Replacement: parse openshell forward list, fail on 127.0.0.1/localhost, require 0.0.0.0/* for the dashboard port.
    • Boundary preserved: real OpenShell forward-list output.

Simplicity check

  • Test shape: simple live Vitest test with local helper functions.
  • New shared helpers: none.
  • New framework/registry/ledger: none.
  • Workflow changes: no workflow YAML change. The existing dashboard-remote-bind suite now invokes the live Vitest test from test/e2e/brev-e2e.test.ts.
  • Legacy shell status: test/e2e/test-dashboard-remote-bind.sh remains in place; this PR no longer deletes any test/e2e shell script.

Verification

  • NEMOCLAW_RUN_E2E_SCENARIOS=1 npx vitest run --project e2e-scenarios-live test/e2e-scenario/live/dashboard-remote-bind.test.ts --silent=false --reporter=default (local opt-in live test is skipped unless NEMOCLAW_E2E_DASHBOARD_REMOTE_BIND=1 is set by branch validation)
  • npx vitest run --project cli test/e2e-script-workflow.test.ts --silent=false --reporter=default
  • git diff --check origin/main...HEAD
  • npm run build:cli && npm run typecheck:cli

@coderabbitai

coderabbitai Bot commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

This PR migrates the dashboard remote-bind E2E test from a legacy shell script to a TypeScript Vitest scenario test. It introduces a new runRemoteCommand helper to standardize remote execution, replaces the shell-script-based test invocation with direct Vitest scenario execution, and removes the legacy shell script from the E2E allowlist.

Changes

Dashboard Remote-Bind Test Migration

Layer / File(s) Summary
Remote command execution refactoring
test/e2e/brev-e2e.test.ts
Introduces runRemoteCommand(command, timeoutMs?) helper that centralizes remote shell pipeline setup (pipefail, nvm initialization, directory change, PATH/npm prefix adjustment), output streaming via tee to /tmp/test-output.log, and diagnostic error handling. runRemoteTest(scriptPath) is refactored to delegate to the new helper.
Dashboard-remote-bind scenario test
test/e2e-scenario/live/dashboard-remote-bind.test.ts
New Vitest scenario test that validates remote dashboard binding. Includes helpers to parse dashboard forward lines, detect all-interface binds (0.0.0.0), and identify loopback-only configurations. Computes remote host from environment or network interfaces, writes scenario.json metadata, verifies CLIs, stops existing forwarding, runs nemoclaw connect with NEMOCLAW_DASHBOARD_BIND=0.0.0.0, and asserts the forward binds to all interfaces.
Test orchestration and legacy cleanup
test/e2e/brev-e2e.test.ts, test/e2e-script-workflow.test.ts
Updates dashboard-remote-bind test execution to invoke the new Vitest scenario directly via runRemoteCommand instead of the legacy shell script. Removes test/e2e/test-dashboard-remote-bind.sh from the legacy allowlist and imports readdirSync for script enumeration. The shell script itself (test/e2e/test-dashboard-remote-bind.sh) is deleted.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

  • NVIDIA/NemoClaw#5155: Both PRs update the legacy E2E shell script allowlist in test/e2e-script-workflow.test.ts by removing retired shell-based E2E tests (dashboard-remote-bind in this PR vs onboard-inference-smoke in #5155).

Suggested labels

area: e2e

Suggested reviewers

  • cv
  • prekshivyas

Poem

🐰 A dashboard test hops from bash to Node,
No shell scripts left on the remote road,
Vitest now runs where /bin/sh once tread,
Remote binds dance on 0.0.0.0 instead,
Cleaner pipelines, assertions all blessed! 🎉

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 12.50% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title 'test(e2e): migrate dashboard remote bind' clearly and concisely describes the main change: migrating an E2E dashboard remote bind test from legacy shell to modern Vitest, which is the core objective of the PR.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch e2e-migrate/dashboard-remote-bind-simple

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions

github-actions Bot commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

E2E Advisor Recommendation

Required E2E: None
Optional E2E: dashboard-remote-bind-e2e

Workflow run

Full advisor summary

E2E Recommendation Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E

  • None. Only E2E test files changed; no installer, onboarding, sandbox lifecycle, credentials, policy, inference routing, deployment, or product runtime/user-flow code was modified, so no merge-blocking E2E is required.

Optional E2E

  • dashboard-remote-bind-e2e (Brev CPU instance; moderate CI cost): Useful to validate the changed Brev E2E harness path and the newly migrated Vitest live scenario for dashboard remote-bind behavior. This is not merge-blocking because no product runtime code changed.

New E2E recommendations

  • None.

@github-actions

github-actions Bot commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

Vitest E2E Scenario Recommendation

Required Vitest E2E scenarios: e2e-scenarios-all
Optional Vitest E2E scenarios: None

Dispatch required Vitest E2E scenarios:

  • gh workflow run e2e-vitest-scenarios.yaml --ref <pr-head-ref>

Workflow run

Full Vitest E2E advisor summary

Vitest E2E Scenario Advisor

Base: origin/main
Head: HEAD
Confidence: medium

Required Vitest E2E scenarios

  • e2e-scenarios-all: This PR adds a new Vitest live dashboard remote-bind test and wires it from the Brev branch-validation harness. The new dashboard-remote-bind entry is not a live-supported typed registry scenario ID on the trusted checkout, so avoid inventing a targeted ID and run the canonical Vitest scenario fan-out instead.
    • Dispatch: gh workflow run e2e-vitest-scenarios.yaml --ref <pr-head-ref>

Optional Vitest E2E scenarios

  • None.

Relevant changed files

  • test/e2e-scenario/live/dashboard-remote-bind.test.ts
  • test/e2e/brev-e2e.test.ts

@github-actions

github-actions Bot commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor

Findings: 1 needs attention, 1 worth checking, 0 nice ideas
Since last review: 1 prior item resolved, 0 still apply, 2 new items found

Review findings

🛠️ Needs attention

  • Resolve the legacy shell deletion contract drift: The PR body states that this migration deletes the legacy shell entry point, but the patch intentionally keeps `test/e2e/test-dashboard-remote-bind.sh` and it remains allowlisted. That contradicts the stated acceptance/contract and can confuse migration governance about whether the legacy script is retired or intentionally frozen.
    • Recommendation: Either update the PR contract/description to say the legacy script is intentionally retained, or delete `test/e2e/test-dashboard-remote-bind.sh` and update the legacy allowlists/wiring in the same change if retirement is intended.
    • Evidence: Acceptance clause: "Migrate `test/e2e/test-dashboard-remote-bind.sh` to a small live Vitest test and delete the legacy shell entry point." Repository evidence: `test/e2e/test-dashboard-remote-bind.sh` still exists and `test/e2e-script-workflow.test.ts` still includes it in `LEGACY_E2E_SHELL_ALLOWLIST`.

🔎 Worth checking

  • Constrain the raw remote shell command helper (test/e2e/brev-e2e.test.ts:399): `runRemoteCommand(command: string)` executes a raw shell string through `sshEnv`, which exports `NVIDIA_API_KEY`, `GITHUB_TOKEN`, and optional messaging tokens into the remote environment. The current call sites use trusted literals, so this is not a confirmed injection bug in this patch, but the generalized helper is a future command-injection and secret-exfiltration footgun if caller-controlled values are interpolated later.
    • Recommendation: Constrain this helper to trusted/branded literals, keep it local to the dashboard suite, or switch to argv-style construction for any future dynamic inputs. Add a short comment documenting that `command` must never include PR/user/env-derived interpolation unless safely escaped.
    • Evidence: `runRemoteCommand(command: string, ...)` builds `${command} 2>&1 | tee /tmp/test-output.log` and passes it to `sshEnv`; `sshEnv` prepends exports for secret-bearing environment variables before executing the command over SSH.

🌱 Nice ideas

  • None.
Consider writing more tests for
  • **Acceptance clause:** and delete the legacy shell entry point. — add test evidence or identify existing coverage. `test/e2e/test-dashboard-remote-bind.sh` still exists and remains in the legacy shell allowlist.
  • **Acceptance clause:** New shared helpers: none. — add test evidence or identify existing coverage. No cross-file shared helper or framework was added, but `test/e2e/brev-e2e.test.ts` gained a new local `runRemoteCommand(command: string)` helper.
  • **Acceptance clause:** Workflow changes: no workflow YAML change; branch validation still uses the existing `dashboard-remote-bind` suite name, but now invokes the live Vitest test instead of the deleted shell script. — add test evidence or identify existing coverage. No workflow YAML changed and `TEST_SUITE === "dashboard-remote-bind"` now invokes `npx vitest run --project e2e-scenarios-live test/e2e-scenario/live/dashboard-remote-bind.test.ts`; however, the referenced shell script was not deleted.
Since last review details

Current findings:

  • Resolve the legacy shell deletion contract drift: The PR body states that this migration deletes the legacy shell entry point, but the patch intentionally keeps `test/e2e/test-dashboard-remote-bind.sh` and it remains allowlisted. That contradicts the stated acceptance/contract and can confuse migration governance about whether the legacy script is retired or intentionally frozen.
    • Recommendation: Either update the PR contract/description to say the legacy script is intentionally retained, or delete `test/e2e/test-dashboard-remote-bind.sh` and update the legacy allowlists/wiring in the same change if retirement is intended.
    • Evidence: Acceptance clause: "Migrate `test/e2e/test-dashboard-remote-bind.sh` to a small live Vitest test and delete the legacy shell entry point." Repository evidence: `test/e2e/test-dashboard-remote-bind.sh` still exists and `test/e2e-script-workflow.test.ts` still includes it in `LEGACY_E2E_SHELL_ALLOWLIST`.
  • Constrain the raw remote shell command helper (test/e2e/brev-e2e.test.ts:399): `runRemoteCommand(command: string)` executes a raw shell string through `sshEnv`, which exports `NVIDIA_API_KEY`, `GITHUB_TOKEN`, and optional messaging tokens into the remote environment. The current call sites use trusted literals, so this is not a confirmed injection bug in this patch, but the generalized helper is a future command-injection and secret-exfiltration footgun if caller-controlled values are interpolated later.
    • Recommendation: Constrain this helper to trusted/branded literals, keep it local to the dashboard suite, or switch to argv-style construction for any future dynamic inputs. Add a short comment documenting that `command` must never include PR/user/env-derived interpolation unless safely escaped.
    • Evidence: `runRemoteCommand(command: string, ...)` builds `${command} 2>&1 | tee /tmp/test-output.log` and passes it to `sshEnv`; `sshEnv` prepends exports for secret-bearing environment variables before executing the command over SSH.

Workflow run details

This is an automated advisory review. A human maintainer must make the final merge decision.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/e2e-scenario/live/dashboard-remote-bind.test.ts`:
- Around line 25-37: The RegExp constructors in bindsAllInterfaces and
bindsLoopback interpolate the env-derived dashboardPort directly, which can
allow regex metacharacters to change matching; fix by escaping dashboardPort
before constructing the RegExp (add a small helper like escapeRegExp that
replaces /[.*+?^${}()|[\]\\]/g with '\\$&' and use the escaped value in new
RegExp calls), and update both functions (bindsAllInterfaces and bindsLoopback)
to use the escaped port value when building the regex.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 8f03a7ed-0b51-454a-9bb9-f5ffe7521006

📥 Commits

Reviewing files that changed from the base of the PR and between 0f99ee5 and c32697f.

📒 Files selected for processing (4)
  • test/e2e-scenario/live/dashboard-remote-bind.test.ts
  • test/e2e-script-workflow.test.ts
  • test/e2e/brev-e2e.test.ts
  • test/e2e/test-dashboard-remote-bind.sh
💤 Files with no reviewable changes (1)
  • test/e2e/test-dashboard-remote-bind.sh

Comment on lines +25 to +37
function bindsAllInterfaces(line: string, dashboardPort: string): boolean {
return (
line.includes(`0.0.0.0:${dashboardPort}`) ||
line.includes(`*:${dashboardPort}`) ||
new RegExp(`\\b0\\.0\\.0\\.0\\s+${dashboardPort}\\b`).test(line)
);
}

function bindsLoopback(line: string, dashboardPort: string): boolean {
return (
line.includes(`127.0.0.1:${dashboardPort}`) ||
line.includes(`localhost:${dashboardPort}`) ||
new RegExp(`\\b127\\.0\\.0\\.1\\s+${dashboardPort}\\b`).test(line)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Escape dashboardPort before injecting into RegExp patterns.

dashboardPort is env-derived and interpolated directly into regex sources, so metacharacters can change match behavior and make this scenario pass/fail incorrectly.

Suggested fix
+function escapeRegExp(value: string): string {
+  return value.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
+}
+
 function bindsAllInterfaces(line: string, dashboardPort: string): boolean {
+  const port = escapeRegExp(dashboardPort);
   return (
     line.includes(`0.0.0.0:${dashboardPort}`) ||
     line.includes(`*:${dashboardPort}`) ||
-    new RegExp(`\\b0\\.0\\.0\\.0\\s+${dashboardPort}\\b`).test(line)
+    new RegExp(`\\b0\\.0\\.0\\.0\\s+${port}\\b`).test(line)
   );
 }
 
 function bindsLoopback(line: string, dashboardPort: string): boolean {
+  const port = escapeRegExp(dashboardPort);
   return (
     line.includes(`127.0.0.1:${dashboardPort}`) ||
     line.includes(`localhost:${dashboardPort}`) ||
-    new RegExp(`\\b127\\.0\\.0\\.1\\s+${dashboardPort}\\b`).test(line)
+    new RegExp(`\\b127\\.0\\.0\\.1\\s+${port}\\b`).test(line)
   );
 }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
function bindsAllInterfaces(line: string, dashboardPort: string): boolean {
return (
line.includes(`0.0.0.0:${dashboardPort}`) ||
line.includes(`*:${dashboardPort}`) ||
new RegExp(`\\b0\\.0\\.0\\.0\\s+${dashboardPort}\\b`).test(line)
);
}
function bindsLoopback(line: string, dashboardPort: string): boolean {
return (
line.includes(`127.0.0.1:${dashboardPort}`) ||
line.includes(`localhost:${dashboardPort}`) ||
new RegExp(`\\b127\\.0\\.0\\.1\\s+${dashboardPort}\\b`).test(line)
function escapeRegExp(value: string): string {
return value.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
}
function bindsAllInterfaces(line: string, dashboardPort: string): boolean {
const port = escapeRegExp(dashboardPort);
return (
line.includes(`0.0.0.0:${dashboardPort}`) ||
line.includes(`*:${dashboardPort}`) ||
new RegExp(`\\b0\\.0\\.0\\.0\\s+${port}\\b`).test(line)
);
}
function bindsLoopback(line: string, dashboardPort: string): boolean {
const port = escapeRegExp(dashboardPort);
return (
line.includes(`127.0.0.1:${dashboardPort}`) ||
line.includes(`localhost:${dashboardPort}`) ||
new RegExp(`\\b127\\.0\\.0\\.1\\s+${port}\\b`).test(line)
);
}
🧰 Tools
🪛 ast-grep (0.43.0)

[warning] 28-28: Regular expression constructed from variable input detected. This can lead to Regular Expression Denial of Service (ReDoS) attacks if the variable contains malicious patterns. Use libraries like 'recheck' to validate regex safety or use static patterns.
Context: new RegExp(\\b0\\.0\\.0\\.0\\s+${dashboardPort}\\b)
Note: [CWE-1333] Inefficient Regular Expression Complexity

(regexp-from-variable)


[warning] 36-36: Regular expression constructed from variable input detected. This can lead to Regular Expression Denial of Service (ReDoS) attacks if the variable contains malicious patterns. Use libraries like 'recheck' to validate regex safety or use static patterns.
Context: new RegExp(\\b127\\.0\\.0\\.1\\s+${dashboardPort}\\b)
Note: [CWE-1333] Inefficient Regular Expression Complexity

(regexp-from-variable)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/e2e-scenario/live/dashboard-remote-bind.test.ts` around lines 25 - 37,
The RegExp constructors in bindsAllInterfaces and bindsLoopback interpolate the
env-derived dashboardPort directly, which can allow regex metacharacters to
change matching; fix by escaping dashboardPort before constructing the RegExp
(add a small helper like escapeRegExp that replaces /[.*+?^${}()|[\]\\]/g with
'\\$&' and use the escaped value in new RegExp calls), and update both functions
(bindsAllInterfaces and bindsLoopback) to use the escaped port value when
building the regex.

Source: Linters/SAST tools

@jyaunches jyaunches added the v0.0.64 Release target label Jun 11, 2026
@jyaunches jyaunches requested a review from cv June 11, 2026 00:55
@cv cv merged commit 0fdb2ce into main Jun 11, 2026
42 checks passed
@cv cv deleted the e2e-migrate/dashboard-remote-bind-simple branch June 11, 2026 05:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

v0.0.64 Release target

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants