test: add full live E2E test (install → onboard → inference) by ericksoa · Pull Request #226 · NVIDIA/NemoClaw

ericksoa · 2026-03-17T17:06:26Z

Summary

Adds a live E2E test that proves the complete user journey with real infrastructure — no mocks. Sends prompts through the sandbox and verifies model responses come back from the NVIDIA Cloud API.

Intended for use in the CD pipeline (#71).

Test phases

Prerequisites — Docker running, openshell installed, NVIDIA_API_KEY valid, network reachable
Install — npm install + npm link from repo
Onboard — creates sandbox e2e-full with NVIDIA Cloud API inference (non-interactive)
Sandbox verification — nemoclaw list, nemoclaw status, registry file checks
Live inference — direct API test + inference through the sandbox (user → sandbox → gateway → NVIDIA API → response)
Cleanup — destroys sandbox, verifies removal

Running

# Requires: Docker, openshell, NVIDIA_API_KEY, network access
bash test/e2e/test-full-e2e.sh

Results

16/16 pass — real inference verified through the full sandbox stack

Notes

Handles nemotron-3-super's reasoning model format (reasoning_content field)
Works around openshell inference set --no-verify incompatibility by configuring inference directly when onboard's attempt fails silently
Uses file redirect instead of $(…) to avoid hanging on openshell's background port-forward
Portable timeout detection (Linux timeout, macOS gtimeout, or fallback)
No API keys or credentials hardcoded

Adds a test that proves the complete user journey with real infrastructure — no mocks. Sends prompts through the sandbox and verifies responses from the NVIDIA Cloud API. Phases: prerequisites → install → onboard → sandbox verification → live inference (direct API + through sandbox) → cleanup Requires: Docker, openshell, NVIDIA_API_KEY, network access. Ref: #205

Fixes #205. Root cause: when the installer runs via `curl | bash`, nvm sources into the subshell and updates PATH. When the subshell exits, all PATH changes are lost — the user's interactive shell still has the old PATH, so `nemoclaw` and `node` are both "command not found". Additionally, `onboard.js` called the full `scripts/install.sh` just to install openshell, which unnecessarily reinstalled Node, Docker, and nemoclaw in a nested subprocess — compounding the PATH issue. Changes: scripts/install.sh: - Add ensure_nvm_loaded() and refresh_path() to update PATH after npm installs nemoclaw - Add nvm fallback detection for $HOME/.nvm when NVM_DIR is unset - Soften verification: warn with instructions instead of hard-failing when the binary exists but isn't on PATH (exit 0, not exit 1) - Add post-install message telling nvm/fnm users to `source ~/.bashrc` - Use sudo for npm install -g when NODE_MGR=nodesource install.sh (root): - Soften verify_nemoclaw() to return 0 when binary exists but PATH is stale (was hard error via error()) - Add post_install_message() for shell reload instructions bin/lib/onboard.js: - Switch installOpenshell() to use dedicated install-openshell.sh instead of the full scripts/install.sh scripts/install-openshell.sh (new): - Dedicated openshell-only installer with macOS + Linux support and conditional sudo Tested: 58/58 unit tests pass, 24/24 E2E tests pass (see PR #226).

Copilot

Pull request overview

Adds a “full” end-to-end (no-mocks) verification flow that installs NemoClaw, onboards an OpenShell sandbox/gateway, and validates live inference against NVIDIA’s hosted API—both directly and through the sandbox.

Changes:

Added a host-oriented full E2E Bash script that runs install → onboard → sandbox verification → live inference → cleanup.
Added a companion Dockerfile intended to run the full E2E script in a containerized environment.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 5 comments.

File	Description
`test/e2e/test-full-e2e.sh`	New full E2E runner that exercises the real OpenShell + NemoClaw onboarding path and validates live inference (direct + via sandbox).
`test/e2e/Dockerfile.full-e2e`	New container image intended to execute the full E2E runner as an ENTRYPOINT.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+# Install Docker CLI only (NOT Docker daemon — will use host socket at runtime)
+RUN curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg && \
+    echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
+    https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" \
+    > /etc/apt/sources.list.d/docker.list && \
+    apt-get update && apt-get install -y --no-install-recommends docker-ce-cli && \
+    rm -rf /var/lib/apt/lists/*


+ENV DEBIAN_FRONTEND=noninteractive
+
+RUN apt-get update && apt-get install -y --no-install-recommends \
+        curl git ca-certificates bash python3 sudo jq \


+section "Phase 1: Prerequisites"
+
+if docker info > /dev/null 2>&1; then
+  pass "Docker is running"
+else
+  fail "Docker is not running — cannot continue"
+  exit 1
+fi
+
+if command -v openshell > /dev/null 2>&1; then
+  pass "openshell CLI installed ($(openshell --version 2>&1 || echo unknown))"
+else
+  fail "openshell CLI not found — cannot continue"
+  exit 1
+fi
+


+#!/bin/bash
+# Full E2E: install → onboard → verify inference (REAL services, no mocks)
+#
+# Proves the COMPLETE user journey including real inference against
+# the NVIDIA Cloud API. Sends prompts through the sandbox and verifies
+# that responses come back from the model.


+# Source bashrc in case nvm/asdf modified it
+if [ -f "$HOME/.bashrc" ]; then
+  source "$HOME/.bashrc" 2>/dev/null || true


Fixes #205. Root cause: when the installer runs via `curl | bash`, nvm sources into the subshell and updates PATH. When the subshell exits, all PATH changes are lost — the user's interactive shell still has the old PATH, so `nemoclaw` and `node` are both "command not found". Additionally, `onboard.js` called the full `scripts/install.sh` just to install openshell, which unnecessarily reinstalled Node, Docker, and nemoclaw in a nested subprocess — compounding the PATH issue. Changes: scripts/install.sh: - Add ensure_nvm_loaded() and refresh_path() to update PATH after npm installs nemoclaw - Add nvm fallback detection for $HOME/.nvm when NVM_DIR is unset - Soften verification: warn with instructions instead of hard-failing when the binary exists but isn't on PATH (exit 0, not exit 1) - Add post-install message telling nvm/fnm users to `source ~/.bashrc` - Use sudo for npm install -g when NODE_MGR=nodesource install.sh (root): - Soften verify_nemoclaw() to return 0 when binary exists but PATH is stale (was hard error via error()) - Add post_install_message() for shell reload instructions bin/lib/onboard.js: - Switch installOpenshell() to use dedicated install-openshell.sh instead of the full scripts/install.sh scripts/install-openshell.sh (new): - Dedicated openshell-only installer with macOS + Linux support and conditional sudo Tested: 58/58 unit tests pass, 24/24 E2E tests pass (see PR #226).

dnandakumar-nv · 2026-03-17T18:49:56Z

bash -c doesn't inherit PS1 even when exported. Bash strips PS1 on non-interactive invocations. So the fix needs to be
inside the script itself. The PR's source "$HOME/.bashrc" 2>/dev/null || true doesn't protect against set -u errors (they exit the
whole script before || true can catch anything). The fix is set +u around the source:

 # Source bashrc in case nvm/asdf modified it
# Temporarily disable -u: .bashrc references $PS1 which is unset in non-interactive shells                           
if [ -f "$HOME/.bashrc" ]; then
  set +u                                                                                                             
  source "$HOME/.bashrc" 2>/dev/null || true
  set -u                                                                                                             
fi
if command -v nemoclaw > /dev/null 2>&1; then

Fixes NVIDIA#205. Root cause: when the installer runs via `curl | bash`, nvm sources into the subshell and updates PATH. When the subshell exits, all PATH changes are lost — the user's interactive shell still has the old PATH, so `nemoclaw` and `node` are both "command not found". Additionally, `onboard.js` called the full `scripts/install.sh` just to install openshell, which unnecessarily reinstalled Node, Docker, and nemoclaw in a nested subprocess — compounding the PATH issue. Changes: scripts/install.sh: - Add ensure_nvm_loaded() and refresh_path() to update PATH after npm installs nemoclaw - Add nvm fallback detection for $HOME/.nvm when NVM_DIR is unset - Soften verification: warn with instructions instead of hard-failing when the binary exists but isn't on PATH (exit 0, not exit 1) - Add post-install message telling nvm/fnm users to `source ~/.bashrc` - Use sudo for npm install -g when NODE_MGR=nodesource install.sh (root): - Soften verify_nemoclaw() to return 0 when binary exists but PATH is stale (was hard error via error()) - Add post_install_message() for shell reload instructions bin/lib/onboard.js: - Switch installOpenshell() to use dedicated install-openshell.sh instead of the full scripts/install.sh scripts/install-openshell.sh (new): - Dedicated openshell-only installer with macOS + Linux support and conditional sudo Tested: 58/58 unit tests pass, 24/24 E2E tests pass (see PR NVIDIA#226).

…A#226) Adds a test that proves the complete user journey with real infrastructure — no mocks. Sends prompts through the sandbox and verifies responses from the NVIDIA Cloud API. Phases: prerequisites → install → onboard → sandbox verification → live inference (direct API + through sandbox) → cleanup Requires: Docker, openshell, NVIDIA_API_KEY, network access. Ref: NVIDIA#205

Fixes NVIDIA#205. Root cause: when the installer runs via `curl | bash`, nvm sources into the subshell and updates PATH. When the subshell exits, all PATH changes are lost — the user's interactive shell still has the old PATH, so `nemoclaw` and `node` are both "command not found". Additionally, `onboard.js` called the full `scripts/install.sh` just to install openshell, which unnecessarily reinstalled Node, Docker, and nemoclaw in a nested subprocess — compounding the PATH issue. Changes: scripts/install.sh: - Add ensure_nvm_loaded() and refresh_path() to update PATH after npm installs nemoclaw - Add nvm fallback detection for $HOME/.nvm when NVM_DIR is unset - Soften verification: warn with instructions instead of hard-failing when the binary exists but isn't on PATH (exit 0, not exit 1) - Add post-install message telling nvm/fnm users to `source ~/.bashrc` - Use sudo for npm install -g when NODE_MGR=nodesource install.sh (root): - Soften verify_nemoclaw() to return 0 when binary exists but PATH is stale (was hard error via error()) - Add post_install_message() for shell reload instructions bin/lib/onboard.js: - Switch installOpenshell() to use dedicated install-openshell.sh instead of the full scripts/install.sh scripts/install-openshell.sh (new): - Dedicated openshell-only installer with macOS + Linux support and conditional sudo Tested: 58/58 unit tests pass, 24/24 E2E tests pass (see PR NVIDIA#226).

…A#226) Adds a test that proves the complete user journey with real infrastructure — no mocks. Sends prompts through the sandbox and verifies responses from the NVIDIA Cloud API. Phases: prerequisites → install → onboard → sandbox verification → live inference (direct API + through sandbox) → cleanup Requires: Docker, openshell, NVIDIA_API_KEY, network access. Ref: NVIDIA#205

ericksoa force-pushed the test/issue-205-install-path-e2e branch from e4a5bc1 to 71bf1b7 Compare March 17, 2026 17:07

ericksoa changed the title ~~test: add E2E test suites for issue #205 (install PATH handling)~~ test: add full live E2E test (install → onboard → inference) Mar 17, 2026

ericksoa force-pushed the test/issue-205-install-path-e2e branch from 71bf1b7 to 0c04ba7 Compare March 17, 2026 17:10

ericksoa requested a review from Copilot March 17, 2026 17:22

Copilot started reviewing on behalf of ericksoa March 17, 2026 17:22 View session

ericksoa mentioned this pull request Mar 17, 2026

fix: resolve PATH issues after nvm installs Node.js #228

Merged

6 tasks

Copilot AI reviewed Mar 17, 2026

View reviewed changes

ericksoa mentioned this pull request Mar 17, 2026

test: add e2e inference smoke test #125

Closed

4 tasks

dnandakumar-nv self-requested a review March 17, 2026 18:21

dnandakumar-nv approved these changes Mar 17, 2026

View reviewed changes

ericksoa merged commit 9bd473b into main Mar 17, 2026
4 checks passed

mafueee pushed a commit to mafueee/NemoClaw that referenced this pull request Mar 28, 2026

feat(cli): launch sandbox editors via managed ssh include (NVIDIA#226)

bcc6dad

cv mentioned this pull request May 17, 2026

chore(security): triage code scanning backlog by risk/reward #3654

Open

6 tasks

wscurran added the chore Build, CI, dependency, or tooling maintenance label Jun 8, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test: add full live E2E test (install → onboard → inference)#226

test: add full live E2E test (install → onboard → inference)#226
ericksoa merged 1 commit into
mainfrom
test/issue-205-install-path-e2e

ericksoa commented Mar 17, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

dnandakumar-nv commented Mar 17, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

ericksoa commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test phases

Running

Results

Notes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

dnandakumar-nv commented Mar 17, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ericksoa commented Mar 17, 2026 •

edited

Loading