Skip to content

fix(proxy): resolve axios + NODE_USE_ENV_PROXY double-proxy conflict#2110

Merged
ericksoa merged 7 commits into
NVIDIA:mainfrom
BenediktSchackenberg:fix/axios-double-proxy-2109
Apr 21, 2026
Merged

fix(proxy): resolve axios + NODE_USE_ENV_PROXY double-proxy conflict#2110
ericksoa merged 7 commits into
NVIDIA:mainfrom
BenediktSchackenberg:fix/axios-double-proxy-2109

Conversation

@BenediktSchackenberg

@BenediktSchackenberg BenediktSchackenberg commented Apr 20, 2026

Copy link
Copy Markdown
Contributor

Problem

axios requests fail inside NemoClaw sandboxes with ERR_BAD_RESPONSE: stream has been aborted. The L7 proxy logs HTTP:UNKNOWN DENIED UNKNOWN.

Root cause: The OpenShell base image sets NODE_USE_ENV_PROXY=1, which intercepts all https.request() calls in Node.js 22 and routes them via CONNECT tunnel. axios also reads HTTPS_PROXY and configures its own proxy pass-through — the request gets double-processed:

  1. axios resolves the proxy → builds a request to http://10.200.0.1:3128
  2. Node.js re-processes the request through NODE_USE_ENV_PROXY → produces https://clawhub.ai:3128/ (port leaked into host)
  3. L7 proxy cannot parse this → DENIED UNKNOWN

https.request() (Node.js core), curl, and jwks-rsa all work correctly because they go through the CONNECT tunnel once.

Fix

Add a preload script (scripts/axios-proxy-fix.js) that disables axios's own proxy handling (proxy: false) when NODE_USE_ENV_PROXY=1 is detected. nemoclaw-start.sh loads it via NODE_OPTIONS=--require at sandbox startup.

  • Only active when NODE_USE_ENV_PROXY=1 — no-op in other environments
  • curl, wget, gRPC continue to use HTTPS_PROXY as before
  • fetch() / undici benefit from the same fix since they share the axios fix path through https.request() after the patch

Fixes #2109

Signed-off-by: Benedikt Schackenberg 6381261+BenediktSchackenberg@users.noreply.github.com

Summary by CodeRabbit

  • New Features

    • Optional startup preload to ensure Axios uses environment-driven proxy behavior when enabled.
  • Bug Fixes

    • Consistent proxy handling across startup and interactive sessions so proxy settings reliably apply.
    • Robustized shell update logic for identity/ownership repairs while preserving existing success/failure behavior.
  • Tests

    • Added regression tests validating proxy env propagation and adjusted tests to accept equivalent shell formatting.

Node.js 22's NODE_USE_ENV_PROXY=1 (baked into the OpenShell base image)
intercepts all https.request() calls and routes them via CONNECT tunnel
through the L7 proxy. axios also reads HTTPS_PROXY from the environment
and configures its own proxy pass-through — resulting in the request
being double-processed:

  axios sets proxy → http://10.200.0.1:3128
  Node.js re-processes → https://clawhub.ai:3128/  ← port leaked
  L7 proxy: HTTP:UNKNOWN DENIED UNKNOWN

Fix: load a preload script (scripts/axios-proxy-fix.js) via NODE_OPTIONS
that disables axios's own proxy handling (proxy: false) when
NODE_USE_ENV_PROXY=1 is set. Node.js then handles the CONNECT tunnel
correctly and axios inherits it transparently through https.request().

The preload only activates when NODE_USE_ENV_PROXY=1 is present, so it
is safe to include unconditionally. curl, wget, and other tools continue
to use HTTPS_PROXY as before.

Fixes NVIDIA#2109

Signed-off-by: Benedikt Schackenberg <6381261+BenediktSchackenberg@users.noreply.github.com>
Copilot AI review requested due to automatic review settings April 20, 2026 19:09
@coderabbitai

coderabbitai Bot commented Apr 20, 2026

Copy link
Copy Markdown
Contributor

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Adds a Node.js preload that disables Axios’s built‑in proxy handling when enabled, updates the shell startup to preload that script via NODE_OPTIONS and emit it into the generated proxy env file for interactive sessions, reformats some shell conditionals, and adds/updates tests verifying the proxy env behavior and a model-override guard formatting variance.

Changes

Cohort / File(s) Summary
Shell startup script
scripts/nemoclaw-start.sh
Reformatted the apply_model_override() early-exit guard to accept shfmt variants. When NODE_USE_ENV_PROXY=1 and /opt/nemoclaw-blueprint/scripts/axios-proxy-fix.js exists, append --require <script> to NODE_OPTIONS for the main process and emit a corresponding export NODE_OPTIONS="...--require <script>" line into /tmp/nemoclaw-proxy-env.sh. Reflowed several shell conditionals (&&/`
Node.js proxy preload module
nemoclaw-blueprint/scripts/axios-proxy-fix.js
New preload script gated by NODE_USE_ENV_PROXY==='1'. Installs a temporary Module._load hook to intercept the first require('axios'), sets axios.defaults.proxy = false if unset, uses Symbol.for('nemoclaw.axiosProxyFix') for idempotency, and restores the original loader after patching.
Tests
test/service-env.test.ts, test/nemoclaw-start.test.ts
Adds regression tests that build/run temporary wrappers around scripts/nemoclaw-start.sh to assert /tmp/nemoclaw-proxy-env.sh includes NODE_OPTIONS with --require when NODE_USE_ENV_PROXY=1 and not otherwise. Updates the runtime model override test to accept either `

Sequence Diagram(s)

sequenceDiagram
    participant Shell as Shell startup script
    participant EnvFile as /tmp/nemoclaw-proxy-env.sh
    participant Node as Node process (NODE_OPTIONS)
    participant Preload as axios-proxy-fix.js
    participant App as Application (requires axios)

    Shell->>Shell: check NODE_USE_ENV_PROXY && script exists
    alt enabled & script present
        Shell->>Node: append --require /opt/.../axios-proxy-fix.js to NODE_OPTIONS
        Shell->>EnvFile: write export NODE_OPTIONS="... --require /opt/.../axios-proxy-fix.js"
    end
    Note over Node,Preload: process start or interactive shell sources NODE_OPTIONS
    Node->>Preload: preload script executes before app modules
    Preload->>Preload: install temporary Module._load hook for 'axios'
    App->>Preload: first require('axios') triggers hook
    Preload->>App: set axios.defaults.proxy = false (if unset)
    Preload->>Preload: restore original Module._load and mark patch installed
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Poem

🐰 A tiny preload hops in place,
It nudges axios to skip the race,
Shells share NODE_OPTIONS so sessions agree,
A symbol keeps the patch from repeating, you see,
Tunnels hum on, steady and ace.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The PR title clearly and specifically describes the main change: fixing a double-proxy conflict between axios and NODE_USE_ENV_PROXY.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
scripts/nemoclaw-start.sh (1)

769-772: Avoid duplicate --require entries in NODE_OPTIONS.

Line 771 always appends. If NODE_OPTIONS already contains this preload, Node can load it multiple times per process.

♻️ Suggested patch
 _AXIOS_FIX_SCRIPT="/opt/nemoclaw-blueprint/scripts/axios-proxy-fix.js"
 if [ -f "$_AXIOS_FIX_SCRIPT" ] && [ "${NODE_USE_ENV_PROXY:-}" = "1" ]; then
-  export NODE_OPTIONS="${NODE_OPTIONS:+$NODE_OPTIONS }--require $_AXIOS_FIX_SCRIPT"
+  case " ${NODE_OPTIONS:-} " in
+    *" --require $_AXIOS_FIX_SCRIPT "*) ;;
+    *)
+      export NODE_OPTIONS="${NODE_OPTIONS:+$NODE_OPTIONS }--require $_AXIOS_FIX_SCRIPT"
+      ;;
+  esac
 fi
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@scripts/nemoclaw-start.sh` around lines 769 - 772, The script unconditionally
appends the axios preload to NODE_OPTIONS which can cause duplicate "--require
/opt/nemoclaw-blueprint/scripts/axios-proxy-fix.js" entries; change the logic
that sets NODE_OPTIONS to first check whether the exact preload token is already
present (match the literal substring "--require $_AXIOS_FIX_SCRIPT" in the
current NODE_OPTIONS) and only append it when not found. Update the block that
references _AXIOS_FIX_SCRIPT and NODE_OPTIONS to perform this existence check
(e.g., using a grep/printf or a shell case string match against "$NODE_OPTIONS")
before exporting the new NODE_OPTIONS value.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@scripts/axios-proxy-fix.js`:
- Around line 38-39: Rename the unused parameters in the Module._load hook to
follow the repo rule by prefixing them with underscores: change the callback
signature from function (request, parent, isMain) to function (request, _parent,
_isMain) and update the call to originalLoad.apply(this, arguments) if you
prefer leaving arguments unchanged; ensure no other references to parent or
isMain remain in the Module._load function body so linting passes for the
unused-parameter rule.

---

Nitpick comments:
In `@scripts/nemoclaw-start.sh`:
- Around line 769-772: The script unconditionally appends the axios preload to
NODE_OPTIONS which can cause duplicate "--require
/opt/nemoclaw-blueprint/scripts/axios-proxy-fix.js" entries; change the logic
that sets NODE_OPTIONS to first check whether the exact preload token is already
present (match the literal substring "--require $_AXIOS_FIX_SCRIPT" in the
current NODE_OPTIONS) and only append it when not found. Update the block that
references _AXIOS_FIX_SCRIPT and NODE_OPTIONS to perform this existence check
(e.g., using a grep/printf or a shell case string match against "$NODE_OPTIONS")
before exporting the new NODE_OPTIONS value.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 11a67d44-1d7b-4c17-a88c-562cb35f1a83

📥 Commits

Reviewing files that changed from the base of the PR and between ed38d45 and b90c10c.

📒 Files selected for processing (2)
  • scripts/axios-proxy-fix.js
  • scripts/nemoclaw-start.sh

Comment thread scripts/axios-proxy-fix.js Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR aims to prevent axios HTTPS requests from being double-proxied in NemoClaw sandboxes when NODE_USE_ENV_PROXY=1 is set, by disabling axios’s own proxy handling via a Node preload script and enabling that preload from the sandbox entrypoint.

Changes:

  • Add a Node preload script that patches axios defaults to set proxy: false when NODE_USE_ENV_PROXY=1.
  • Update nemoclaw-start.sh to conditionally append --require for the preload script via NODE_OPTIONS.
  • Minor shell formatting/flow adjustments in nemoclaw-start.sh.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File Description
scripts/nemoclaw-start.sh Conditionally enables an axios proxy workaround by adding a preload via NODE_OPTIONS.
scripts/axios-proxy-fix.js Implements the preload-time axios patch by intercepting module loading and setting axios.defaults.proxy = false.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread scripts/nemoclaw-start.sh
# malformed URLs (https://host:3128/) rejected by the L7 proxy.
# The preload script disables axios's own proxy handling so NODE_USE_ENV_PROXY
# takes over — the correct path for all other Node.js HTTP clients.
_AXIOS_FIX_SCRIPT="/opt/nemoclaw-blueprint/scripts/axios-proxy-fix.js"

Copilot AI Apr 20, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_AXIOS_FIX_SCRIPT points to /opt/nemoclaw-blueprint/scripts/axios-proxy-fix.js, but the image build only copies nemoclaw-blueprint/ into /opt/nemoclaw-blueprint/ (no scripts/ dir), and this new file is added under repo scripts/. As a result the -f check will fail and the preload will never be enabled. Move the preload file into nemoclaw-blueprint/scripts/ (or another path that is actually present in the image) and/or adjust the build to copy it to the referenced location.

Suggested change
_AXIOS_FIX_SCRIPT="/opt/nemoclaw-blueprint/scripts/axios-proxy-fix.js"
#
# Write the preload at runtime to a path guaranteed to exist in the container
# instead of depending on an image-build copy step for a repository-level file.
_AXIOS_FIX_SCRIPT="/tmp/axios-proxy-fix.js"
cat >"$_AXIOS_FIX_SCRIPT" <<'EOF'
'use strict';
const Module = require('module');
const originalLoad = Module._load;
Module._load = function patchedAxiosLoad(request, parent, isMain) {
const exported = originalLoad.apply(this, arguments);
if (request === 'axios' && exported) {
const patchDefaults = function patchDefaults(target) {
if (target && target.defaults && Object.prototype.hasOwnProperty.call(target.defaults, 'proxy')) {
target.defaults.proxy = false;
}
};
patchDefaults(exported);
if (exported.default) {
patchDefaults(exported.default);
}
}
return exported;
};
EOF

Copilot uses AI. Check for mistakes.
Comment thread scripts/nemoclaw-start.sh
# takes over — the correct path for all other Node.js HTTP clients.
_AXIOS_FIX_SCRIPT="/opt/nemoclaw-blueprint/scripts/axios-proxy-fix.js"
if [ -f "$_AXIOS_FIX_SCRIPT" ] && [ "${NODE_USE_ENV_PROXY:-}" = "1" ]; then
export NODE_OPTIONS="${NODE_OPTIONS:+$NODE_OPTIONS }--require $_AXIOS_FIX_SCRIPT"

Copilot AI Apr 20, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sets NODE_OPTIONS in the entrypoint process, but openshell sandbox connect sessions source env from /tmp/nemoclaw-proxy-env.sh (see comment below) rather than inheriting exports from PID 1. If axios failures also occur in interactive sessions / user commands, this preload likely won’t be active there. Consider also emitting an export NODE_OPTIONS=... line into the /tmp/nemoclaw-proxy-env.sh file (or otherwise ensuring NODE_OPTIONS is present for connect sessions).

Suggested change
export NODE_OPTIONS="${NODE_OPTIONS:+$NODE_OPTIONS }--require $_AXIOS_FIX_SCRIPT"
export NODE_OPTIONS="${NODE_OPTIONS:+$NODE_OPTIONS }--require $_AXIOS_FIX_SCRIPT"
_TOOL_REDIRECTS+=("NODE_OPTIONS=$NODE_OPTIONS")

Copilot uses AI. Check for mistakes.
Comment thread scripts/nemoclaw-start.sh
Comment on lines +770 to +772
if [ -f "$_AXIOS_FIX_SCRIPT" ] && [ "${NODE_USE_ENV_PROXY:-}" = "1" ]; then
export NODE_OPTIONS="${NODE_OPTIONS:+$NODE_OPTIONS }--require $_AXIOS_FIX_SCRIPT"
fi

Copilot AI Apr 20, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New proxy behavior is introduced here, but the existing test suite validates the proxy env block and the generated /tmp env file (e.g., test/service-env.test.ts) without covering this new NODE_OPTIONS --require injection. Add/update tests to assert the preload flag is set when NODE_USE_ENV_PROXY=1 (and that it’s persisted to the connect-session env file if that’s the intended scope).

Copilot uses AI. Check for mistakes.
Comment thread scripts/axios-proxy-fix.js Outdated
- Move axios-proxy-fix.js to nemoclaw-blueprint/scripts/ so it is
  present in the image at /opt/nemoclaw-blueprint/scripts/ (the
  Dockerfile COPYs nemoclaw-blueprint/ → /opt/nemoclaw-blueprint/)
- Prefix unused Module._load params with _ to satisfy repo eslint rule
- Restore Module._load after axios is patched to avoid ongoing overhead
- Add idempotency guard via Symbol.for so the script is safe to require
  multiple times
- Also emit NODE_OPTIONS --require into /tmp/nemoclaw-proxy-env.sh so
  that interactive connect sessions (openshell sandbox connect) also
  benefit from the fix
- Add regression test (NVIDIA#2109) to service-env.test.ts asserting that
  proxy-env.sh contains NODE_OPTIONS with the --require flag when
  NODE_USE_ENV_PROXY=1 and the fix script exists

Per Copilot + CodeRabbit review on NVIDIA#2110.

Signed-off-by: Benedikt Schackenberg <6381261+BenediktSchackenberg@users.noreply.github.com>
@BenediktSchackenberg

Copy link
Copy Markdown
Contributor Author

Addressed all review points:

  1. File location — moved to nemoclaw-blueprint/scripts/ (Dockerfile: COPY nemoclaw-blueprint/ /opt/nemoclaw-blueprint/) ✅
  2. Unused params — prefixed _parent / _isMain
  3. Module._load overhead — restored to originalLoad immediately after axios is patched; added Symbol.for idempotency guard ✅
  4. Connect sessions — also emits NODE_OPTIONS --require into /tmp/nemoclaw-proxy-env.sh so interactive openshell sandbox connect sessions get the fix too ✅
  5. Tests — regression test in test/service-env.test.ts asserts that proxy-env.sh contains the NODE_OPTIONS preload when NODE_USE_ENV_PROXY=1 and the fix script exists; 33/33 pass ✅

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
test/service-env.test.ts (1)

572-576: Add an explicit non-empty persistBlock check for clearer failures.

If the sed anchors change, this test will currently fail later with less actionable output.

💡 Suggested tweak
         const persistBlock = execFileSync(
           "sed",
           ["-n", "/^_PROXY_URL=/,/^chmod 644/p", scriptPath],
           { encoding: "utf-8" },
         );
+        if (!persistBlock.trim()) {
+          throw new Error(
+            "Failed to extract proxy persistence block from scripts/nemoclaw-start.sh — " +
+              "the _PROXY_URL..chmod block may have been moved or renamed",
+          );
+        }
         const wrapper = [
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@test/service-env.test.ts` around lines 572 - 576, The test reads a
persistBlock via execFileSync into the variable persistBlock but doesn't verify
it's non-empty, causing later, less useful failures if the sed anchors change;
after the execFileSync call that assigns persistBlock (the line calling
execFileSync with "sed" and the "/^_PROXY_URL=/,/^chmod 644/p" args), add an
explicit assertion/check that persistBlock is not an empty string and fail fast
with a clear message (e.g., throw or use the test framework's assert/expect)
indicating the sed extraction returned no content so maintainers know the
anchors need updating.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@test/service-env.test.ts`:
- Around line 572-576: The test reads a persistBlock via execFileSync into the
variable persistBlock but doesn't verify it's non-empty, causing later, less
useful failures if the sed anchors change; after the execFileSync call that
assigns persistBlock (the line calling execFileSync with "sed" and the
"/^_PROXY_URL=/,/^chmod 644/p" args), add an explicit assertion/check that
persistBlock is not an empty string and fail fast with a clear message (e.g.,
throw or use the test framework's assert/expect) indicating the sed extraction
returned no content so maintainers know the anchors need updating.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 0e1252cc-3a5e-4cf2-b329-ed9718989f4b

📥 Commits

Reviewing files that changed from the base of the PR and between b90c10c and 4ddec05.

📒 Files selected for processing (3)
  • nemoclaw-blueprint/scripts/axios-proxy-fix.js
  • scripts/nemoclaw-start.sh
  • test/service-env.test.ts

Per CodeRabbit review on NVIDIA#2110.

Signed-off-by: Benedikt Schackenberg <6381261+BenediktSchackenberg@users.noreply.github.com>
@BenediktSchackenberg

Copy link
Copy Markdown
Contributor Author

Fixed — added the explicit persistBlock.trim() check so the test fails fast with a clear message if the sed anchors ever change. 33/33 pass.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
test/service-env.test.ts (1)

565-603: Add a negative-path assertion for no-op behavior.

Line 587 forces NODE_USE_ENV_PROXY=1, but this regression suite does not assert the opposite case (unset/not 1 should not add NODE_OPTIONS). Adding that test would better lock the intended no-op contract.

➕ Suggested test addition
+    it("does not inject NODE_OPTIONS when NODE_USE_ENV_PROXY is unset", () => {
+      const fakeDataDir = join(tmpdir(), `nemoclaw-axios-fix-negative-${process.pid}`);
+      const fakeFixScript = join(fakeDataDir, "axios-proxy-fix.js");
+      execFileSync("mkdir", ["-p", fakeDataDir]);
+      const tmpFile = join(tmpdir(), `nemoclaw-axios-fix-negative-env-${process.pid}.sh`);
+      try {
+        const scriptPath = join(import.meta.dirname, "../scripts/nemoclaw-start.sh");
+        const persistBlock = execFileSync("sed", ["-n", "/^_PROXY_URL=/,/^chmod 644/p", scriptPath], {
+          encoding: "utf-8",
+        });
+        writeFileSync(fakeFixScript, "// fake", { mode: 0o644 });
+        const wrapper = [
+          "#!/usr/bin/env bash",
+          "set -euo pipefail",
+          'PROXY_HOST="10.200.0.1"',
+          'PROXY_PORT="3128"',
+          "_TOOL_REDIRECTS=()",
+          persistBlock
+            .trimEnd()
+            .replaceAll("/tmp/nemoclaw-proxy-env.sh", `${fakeDataDir}/proxy-env.sh`)
+            .replaceAll("/opt/nemoclaw-blueprint/scripts/axios-proxy-fix.js", fakeFixScript),
+        ].join("\n");
+        writeFileSync(tmpFile, wrapper, { mode: 0o700 });
+        execFileSync("bash", [tmpFile], { encoding: "utf-8" });
+
+        const envFile = readFileSync(join(fakeDataDir, "proxy-env.sh"), "utf-8");
+        expect(envFile).not.toContain("NODE_OPTIONS");
+      } finally {
+        try {
+          execFileSync("rm", ["-rf", fakeDataDir, tmpFile]);
+        } catch {
+          /* ignore */
+        }
+      }
+    });
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@test/service-env.test.ts` around lines 565 - 603, Add a negative-path test
alongside the existing "regression `#2109`: proxy-env.sh includes NODE_OPTIONS
--require when NODE_USE_ENV_PROXY=1 and fix script exists" case that mirrors the
same setup but with NODE_USE_ENV_PROXY unset or set to "0" (use the same tmp
wrapper construction, persistBlock extraction and fakeDataDir/fakeFixScript
handling), run the wrapper, then read the generated proxy-env.sh and assert it
does NOT contain "NODE_OPTIONS", "--require", or the fakeFixScript path; ensure
you reference the same variables used in the positive test (NODE_USE_ENV_PROXY,
fakeFixScript, proxy-env.sh, persistBlock) and keep teardown in the same finally
block pattern.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@test/service-env.test.ts`:
- Around line 565-603: Add a negative-path test alongside the existing
"regression `#2109`: proxy-env.sh includes NODE_OPTIONS --require when
NODE_USE_ENV_PROXY=1 and fix script exists" case that mirrors the same setup but
with NODE_USE_ENV_PROXY unset or set to "0" (use the same tmp wrapper
construction, persistBlock extraction and fakeDataDir/fakeFixScript handling),
run the wrapper, then read the generated proxy-env.sh and assert it does NOT
contain "NODE_OPTIONS", "--require", or the fakeFixScript path; ensure you
reference the same variables used in the positive test (NODE_USE_ENV_PROXY,
fakeFixScript, proxy-env.sh, persistBlock) and keep teardown in the same finally
block pattern.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 47b66816-0698-4888-b65a-ce80d2938c50

📥 Commits

Reviewing files that changed from the base of the PR and between 4ddec05 and b400249.

📒 Files selected for processing (1)
  • test/service-env.test.ts

@wscurran wscurran added bug Something fails against expected or documented behavior OpenShell labels Apr 21, 2026
@wscurran

Copy link
Copy Markdown
Contributor

✨ Thanks for submitting this PR that proposes a fix for the axios request failure issue inside NemoClaw sandboxes. The changes and explanation you provided will help us review this further. Fixes #2109.


Possibly related open issues:

ericksoa
ericksoa previously approved these changes Apr 21, 2026

@ericksoa ericksoa left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Approach is sound — preload hook is self-cleaning, guarded by NODE_USE_ENV_PROXY=1, and fails safe if the script is missing. Connect sessions are covered via proxy-env.sh. Copilot review comments are either already addressed or incorrect about the file path mapping.

@ericksoa ericksoa dismissed their stale review April 21, 2026 04:27

Revoking — needs fixes to pass E2E.

@ericksoa

Copy link
Copy Markdown
Contributor

Thanks for this — the approach is solid and we'd like to merge this once CI is green. Three things to fix:

  1. axios-proxy-fix.js not executable — the file has a shebang but isn't marked +x. Run chmod +x nemoclaw-blueprint/scripts/axios-proxy-fix.js (or git add --chmod=+x).

  2. Test nemoclaw-start.test.ts failing — the apply_model_override formatting change (\ ||||\n) broke the '|| return 0' substring assertion in the "is a no-op when no override env vars are set" test. Either update the test to match the new formatting or revert the cosmetic reformatting (it's unrelated to the proxy fix).

  3. Test service-env.test.ts regression axios requests fail with ERR_BAD_RESPONSE inside NemoClaw sandbox — double proxy conflict with NODE_USE_ENV_PROXY #2109 failing_TOOL_REDIRECTS[@]: unbound variable on macOS. The test wrapper script needs _TOOL_REDIRECTS=() declared before the extracted block references it (it's declared but the sed extraction may not be capturing it).

The proxy fix itself looks great — once these are addressed we'll approve and merge.

@BenediktSchackenberg

Copy link
Copy Markdown
Contributor Author

Thanks @copilot-pull-request-reviewer — the explicit persistBlock.trim() guard is already present in the current branch at both sed-call sites, so the test fails fast if the anchors ever move. That part is covered.

1. Mark axios-proxy-fix.js executable (chmod +x via git)
2. Update nemoclaw-start.test.ts to match shfmt-reformatted 'return 0'
   (shfmt puts return 0 on its own line, test now uses regex)
3. Add 'set +u' in service-env test wrappers for macOS bash compat
   (empty array expansion triggers unbound variable with set -u on macOS)
4. Fix describe block nesting in new NVIDIA#2109 regression test
5. Add negative-path test: proxy-env.sh must NOT contain NODE_OPTIONS
   when NODE_USE_ENV_PROXY is unset

Per ericksoa review on NVIDIA#2110.

Signed-off-by: Benedikt Schackenberg <6381261+BenediktSchackenberg@users.noreply.github.com>
@BenediktSchackenberg BenediktSchackenberg force-pushed the fix/axios-double-proxy-2109 branch from e76943e to d0ebc0d Compare April 21, 2026 05:29
@BenediktSchackenberg

Copy link
Copy Markdown
Contributor Author

Addressed all three items from @ericksoa:

  1. axios-proxy-fix.js +x — added via git add --chmod=+x
  2. nemoclaw-start.test.ts failing — updated the return 0 assertion to a regex that matches both || return 0 and the shfmt-reformatted standalone return 0
  3. service-env.test.ts macOS unbound variable — added set +u in both test wrapper scripts before the sed-extracted block (empty array expansion with set -u triggers the error on macOS bash)

102/102 tests pass.

ericksoa and others added 3 commits April 21, 2026 08:16
The CI checks job failed because:
1. axios-proxy-fix.js has a shebang but was not marked executable
2. shfmt/Prettier reformatted files that were committed with
   non-canonical formatting (trailing || vs backslash continuation)

Run formatters on the PR's files and chmod +x the preload script.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

@ericksoa ericksoa left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI fixes verified locally — hooks pass, executable bit set. LGTM.

@ericksoa ericksoa merged commit ff18fb8 into NVIDIA:main Apr 21, 2026
12 checks passed
@BenediktSchackenberg BenediktSchackenberg deleted the fix/axios-double-proxy-2109 branch April 21, 2026 17:23
@cv cv added the v0.0.22 label Apr 21, 2026
@miyoungc miyoungc mentioned this pull request Apr 22, 2026
13 tasks
miyoungc added a commit that referenced this pull request Apr 22, 2026
## Summary

Bumps the published doc version to `0.0.22` and documents the
user-visible CLI behavior changes to `nemoclaw <name> connect` that
landed since v0.0.21. Drafted via the `nemoclaw-contributor-update-docs`
skill against commits in `v0.0.21..origin/main`, filtered through
`docs/.docs-skip`.

## Changes

- **`docs/project.json`** and **`docs/versions1.json`**: bump the
published version from `0.0.20` to `0.0.22`; insert a `0.0.21` entry
into the version list so the history stays contiguous.
- **`docs/reference/commands.md`** → `nemoclaw <name> connect`: document
two new behaviors.
- Readiness poll with `NEMOCLAW_CONNECT_TIMEOUT` (integer seconds;
default `120`) that replaces the silent hang when the sandbox is not yet
`Ready` — right after onboarding, while the 2.4 GB image is still
pulling (#466).
- Post-connect hint is now agent-aware, names the correct TUI command
for the sandbox's agent, and tells you to use `/exit` to leave the chat
before `exit` returns you to the host shell (#2080).

Feature PRs that shipped their own docs in the same commit are
intentionally not re-documented here:

- `channels list/add/remove` (#2139) — command reference and the
"`openclaw channels` blocked inside the sandbox" troubleshooting entry
landed with the feature.
- `nemoclaw gc` (#2176) — documented as part of the destroy/rebuild
image cleanup PR.

Skipped per `docs/.docs-skip`:

- `e6bad533 fix(shields): verify config lock and fail hard on re-lock
failure (#2066)` — matched `skip-features: src/lib/shields.ts`.

Other commits in the range (#2141 OpenShell version bump, #1819 plugin
banner live inference probe, #2085 / #2146 Slack Socket Mode fixes,
#2110 axios proxy fix, #1818 NIM curl timeouts, #1824 onboard gateway
bootstrap recovery, and assorted CI / test / install plumbing) are
internal behavior refinements with no doc-relevant surface change.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [x] Doc only (includes code sample changes)

## Verification

- [x] `npx prek run --all-files` passes for the modified files via the
pre-commit hook, including `Regenerate agent skills from docs` (source ↔
generated parity confirmed)
- [ ] `npm test` passes — skipped; the one pre-existing
`test/cli.test.ts > unknown command exits 1` failure on `origin/main` is
unrelated to these markdown/JSON-only changes
- [ ] Tests added or updated for new or changed behavior — n/a, doc-only
- [x] No secrets, API keys, or credentials committed
- [x] Docs updated for user-facing behavior changes
- [ ] `make docs` builds without warnings (doc changes only) — not run
locally
- [x] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)
— n/a, no new pages

## AI Disclosure

- [x] AI-assisted — tool: Claude Code

---
Signed-off-by: Miyoung Choi <miyoungc@nvidia.com>

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* `connect` now displays the sandbox phase while waiting for readiness
and honors a configurable timeout via NEMOCLAW_CONNECT_TIMEOUT (default
120s).
* TTY hints are agent-aware and instruct using `/exit` before returning
to the host shell.

* **Documentation**
  * Command docs updated to describe polling, timeout, and TTY guidance.
* Project/docs metadata updated for versions 0.0.21 and 0.0.22 (package
version bumped to 0.0.22).
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
lcsmontiel added a commit to lcsmontiel/NemoClaw that referenced this pull request Apr 23, 2026
PR NVIDIA#2110's axios-only Module._load preload never fired at runtime:

  1. nemoclaw-blueprint/scripts/ is excluded from the optimized sandbox
     build context (src/lib/sandbox-build-context.ts), so
     axios-proxy-fix.js was not baked into the sandbox image.
  2. Adding scripts/ to the build context cache-busts the
     `COPY nemoclaw-blueprint/` Dockerfile layer and hangs npm ci in
     the k3s Docker-in-Docker build, so the delivery gap cannot be
     closed by expanding the context.
  3. Even if the file had reached the image, intercepting
     require('axios') via Module._load cannot patch follow-redirects +
     proxy-from-env bundled as ESM in OpenClaw's dist/http-Bh-HtMAg.js
     — there are no require() calls to intercept. The Bot Connector
     reply path uses the bundled code.

Replace with an http.request() wrapper — the lowest common denominator
every HTTP library bottoms out at. Detect FORWARD-mode requests
(hostname = proxy IP, path = full https:// URL) and rewrite them to
https.request() against the real target, letting NODE_USE_ENV_PROXY
handle the CONNECT tunnel correctly. Works for any HTTP client,
including bundled ESM that makes no require() calls.

Delivery:
  - nemoclaw-blueprint/scripts/http-proxy-fix.js — canonical source for
    review and tests.
  - scripts/nemoclaw-start.sh embeds the same JS inline via a heredoc,
    writes it to /tmp/nemoclaw-http-proxy-fix.js through
    emit_sandbox_sourced_file (root:root 444, symlink-safe), and loads
    it via NODE_OPTIONS=--require. No changes to sandbox-build-context.
  - test/http-proxy-fix-sync.test.ts enforces byte-for-byte equality
    between the heredoc and the canonical file, so future edits cannot
    silently diverge.
  - validate_tmp_permissions is invoked with the new path on both the
    root and non-root boot paths (the fix JS is a trust-boundary file
    — tampering would inject arbitrary code into every Node process
    via NODE_OPTIONS).

Because the content ships inside nemoclaw-start.sh rather than as a
separately-deployed file, the fix fires on the very first sandbox boot
with no post-onboard deploy + restart dance required.

Verified end-to-end on 2026-04-23: EC2 t3.large (ca-central-1),
NemoClaw v0.0.22 + OpenShell v0.0.29, Node 22.22.1. Direct
axios.get('https://clawhub.ai') returns 200 inside the sandbox; full
Teams -> ALB -> OpenClaw -> LiteLLM/Bedrock -> Bot Connector ->
Teams round-trip succeeds. No `FORWARD rejected` entries in OpenShell
network logs. Comparison table and reproduction steps posted in the
PR description.

Scope:
  - Fixes the NVIDIA#2109 regression class (axios / follow-redirects /
    proxy-from-env FORWARD-mode rewrites on NODE_USE_ENV_PROXY=1).
  - Does NOT fix NVIDIA#1570 (Discord WebSocket via the ws library). That
    bug sits at a different layer — EnvHttpProxyAgent's FORWARD-vs-
    CONNECT decision for Upgrade: websocket requests — and needs the
    agent-swap treatment that NVIDIA#2296 applies. The http.request wrapper
    in this PR cannot safely handle that case (it would re-enter the
    same faulty agent logic).
  - Does NOT modify sandbox-build-context.ts.

Removes the superseded nemoclaw-blueprint/scripts/axios-proxy-fix.js
and updates the existing regression tests in service-env.test.ts to
the new variable name (_PROXY_FIX_SCRIPT).

Closes NVIDIA#2109.
@wscurran wscurran added area: sandbox OpenShell sandbox lifecycle, runtime, config, or recovery bug-fix PR fixes a bug or regression and removed OpenShell bug Something fails against expected or documented behavior labels Jun 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: sandbox OpenShell sandbox lifecycle, runtime, config, or recovery bug-fix PR fixes a bug or regression

Projects

None yet

Development

Successfully merging this pull request may close these issues.

axios requests fail with ERR_BAD_RESPONSE inside NemoClaw sandbox — double proxy conflict with NODE_USE_ENV_PROXY

5 participants