Skip to content

[Nemoclaw][All Platforms] Proxy test conflates wiring with proxy policy: Node https.get reports ERR_PROXY_TUNNEL when upstream proxy returns 403 #3836

@zNeill

Description

@zNeill

Description

Description
With NEMOCLAW_PROXY_HOST / NEMOCLAW_PROXY_PORT set to a custom HTTP proxy, NemoClaw correctly wires proxy environment variables and the Node ESM proxy wrapper, but the verification step using https.get("https://api.telegram.org/") fails with ERR_PROXY_TUNNEL because the upstream proxy returns HTTP/1.1 403 Forbidden on the CONNECT request. The existing test’s Expected section assumes a successful HTTP status (200/301/302) from Telegram and treats anything else as a failure, even though the proxy path is correct and the failure is due to proxy policy, not NemoClaw’s proxy wiring. This makes the test conflate “proxy wiring is broken” with “proxy is intentionally blocking the destination.”

Environment

  • Platform: Linux (e.g. Ubuntu 22.04 / 24.04 / 26.04), in a NemoClaw sandbox.
  • NemoClaw: Installed and working (e.g. via curl -fsSL https://www.nvidia.com/nemoclaw.sh | bash).
  • OpenShell gateway: Running as per NemoClaw Quickstart.
  • Proxy configuration:
    • NEMOCLAW_PROXY_HOST=10.200.0.1
    • NEMOCLAW_PROXY_PORT=3128
    • Proxy reachable from the sandbox, but configured to deny CONNECT tunnels to api.telegram.org:443 with HTTP 403 Forbidden.
  • Node: v22.x inside the sandbox (uses EnvHttpProxyAgent when HTTP_PROXY / HTTPS_PROXY are set).

Steps to Reproduce

Preconditions

  1. Set the proxy environment for NemoClaw:
     
    bash export NEMOCLAW_PROXY_HOST=10.200.0.1 export NEMOCLAW_PROXY_PORT=3128
  2. Install NemoClaw and onboard at least one sandbox (e.g. prachi-new-sb), ensuring the sandbox entrypoint completes successfully.
  3. Ensure no host‑level HTTP_PROXY / HTTPS_PROXY overrides are set beyond the NemoClaw proxy config.

Repro steps

  1. Connect to the sandbox:
     
    bash nemoclaw connect
  2. Inside the sandbox, verify proxy env and config files:
     
    bash echo "$HTTP_PROXY" echo "$HTTPS_PROXY" ls -la /etc/profile.d/ 2>/dev/null || ls -la /sandbox/.profile.d/ 2>/dev/null stat -c '%U:%G %a' $(ls /etc/profile.d/nemoclaw-proxy.sh 2>/dev/null || ls /sandbox/.profile.d/nemoclaw-proxy.sh 2>/dev/null) grep -i proxy ~/.bashrc ~/.profile 2>&1 echo "$NO_PROXY" echo "$no_proxy"
  3. Verify Node wrapper:
     
    bash echo "$NODE_OPTIONS" echo "$NODE_OPTIONS" | grep -o 'nemoclaw-http-proxy-fix.js'
  4. Run the Node HTTPS probe:
     
    bash node -e 'const https=require("https"); https.get("https://api.telegram.org/", r=>console.log("STATUS:"+r.statusCode)).on("error", e=>console.log("ERR:"+e.code));'
  5. Confirm proxy behavior with curl:
     
    bash curl -v --max-time 10 --proxy http://10.200.0.1:3128 https://api.telegram.org/

Expected Result (per current test case)

  • HTTP_PROXY and HTTPS_PROXY inside the sandbox reflect the custom proxy:
     
    bash http://10.200.0.1:3128 http://10.200.0.1:3128
  • A root‑owned, read‑only NemoClaw proxy script exists in a sourced config directory, e.g.:
     
    bash ls -la /etc/profile.d/ -r--r--r-- 1 root root 141 May 16 20:13 nemoclaw-proxy.sh stat -c '%U:%G %a' /etc/profile.d/nemoclaw-proxy.sh root:root 444
  • ~/.bashrc and ~/.profile do not contain inline proxy exports or # BEGIN NEMOCLAW PROXY markers; they only source a runtime env file, e.g.:
     
    bash /sandbox/.bashrc:# Source runtime proxy config /sandbox/.bashrc:[ -f /tmp/nemoclaw-proxy-env.sh ] && . /tmp/nemoclaw-proxy-env.sh /sandbox/.profile:# Source runtime proxy config /sandbox/.profile:[ -f /tmp/nemoclaw-proxy-env.sh ] && . /tmp/nemoclaw-proxy-env.sh
  • NO_PROXY and no_proxy contain local bypass hosts and match exactly:
     
    bash localhost,127.0.0.1,::1,10.200.0.1 localhost,127.0.0.1,::1,10.200.0.1
  • NODE_OPTIONS includes the proxy wrapper:
     
    bash --require /tmp/nemoclaw-http-proxy-fix.js ...

    and

     
    bash

    echo "$NODE_OPTIONS" | grep -o 'nemoclaw-http-proxy-fix.js' nemoclaw-http-proxy-fix.js


  • Node HTTPS probe:

    • Expected by the test: STATUS: (e.g. STATUS:200 or redirect) and not a low‑level error like ECONNREFUSED or WebSocket 1006.
  • curl probe:
    • Expected: HTTP:200/301/302, indicating successful proxy tunnel and request to Telegram.

Actual Result

In the sandbox:

  1. Env and proxy script:
     
    bash echo $HTTPS_PROXY # http://10.200.0.1:3128 echo $HTTP_PROXY # http://10.200.0.1:3128 ls -la /etc/profile.d/ total 12 drwxr-xr-x 1 root root 4096 May 16 20:13 . drwxr-xr-x 1 root root 4096 May 19 16:59 .. -r--r--r-- 1 root root 141 May 16 20:13 nemoclaw-proxy.sh stat -c '%U:%G %a' /etc/profile.d/nemoclaw-proxy.sh # root:root 444 grep -i proxy ~/.bashrc ~/.profile 2>&1 /sandbox/.bashrc:# Source runtime proxy config /sandbox/.bashrc:[ -f /tmp/nemoclaw-proxy-env.sh ] && . /tmp/nemoclaw-proxy-env.sh /sandbox/.profile:# Source runtime proxy config /sandbox/.profile:[ -f /tmp/nemoclaw-proxy-env.sh ] && . /tmp/nemoclaw-proxy-env.sh echo "$NO_PROXY" echo "$no_proxy" # localhost,127.0.0.1,::1,10.200.0.1 # localhost,127.0.0.1,::1,10.200.0.1
  2. Node wrapper:
     
    bash echo "$NODE_OPTIONS" --require /tmp/nemoclaw-sandbox-safety-net.js --require /tmp/nemoclaw-http-proxy-fix.js --require /tmp/nemoclaw-nemotron-inference-fix.js --require /tmp/nemoclaw-seccomp-guard.js --require /tmp/nemoclaw-ciao-network-guard.js --require /tmp/nemoclaw-sandbox-safety-net.js --require /tmp/nemoclaw-http-proxy-fix.js --require /tmp/nemoclaw-nemotron-inference-fix.js --require /tmp/nemoclaw-seccomp-guard.js --require /tmp/nemoclaw-ciao-network-guard.js echo "$NODE_OPTIONS" | grep -o 'nemoclaw-http-proxy-fix.js' nemoclaw-http-proxy-fix.js nemoclaw-http-proxy-fix.js
  3. Node HTTPS probe:
     
    bash node -e 'const https=require("https"); https.get("https://api.telegram.org/", r=>console.log("STATUS:"+r.statusCode)).on("error", e=>console.log("ERR:"+e.code));' (node:864) [UNDICI-EHPA] Warning: EnvHttpProxyAgent is experimental, expect them to change at any time. ERR:ERR_PROXY_TUNNEL
  4. curl via proxy:
     
    bash curl -v --max-time 10 --proxy http://10.200.0.1:3128 https://api.telegram.org/ * Uses proxy env variable no_proxy == 'localhost,127.0.0.1,::1,10.200.0.1' * Trying 10.200.0.1:3128... * CONNECT tunnel: HTTP/1.1 negotiated * Establish HTTP proxy tunnel to api.telegram.org:443 > CONNECT api.telegram.org:443 HTTP/1.1 > Host: api.telegram.org:443 > User-Agent: curl/8.14.1 > Proxy-Connection: Keep-Alive < < HTTP/1.1 403 Forbidden < Content-Type: application/json < Content-Length: 89 < Connection: close < * CONNECT tunnel failed, response 403 * closing connection #0 curl: (56) CONNECT tunnel failed, response 403

Key observations:

  • The proxy wiring is correct: Node and curl both route through the proxy; the CONNECT request and no_proxy behavior are as expected.
  • The failure (ERR_PROXY_TUNNEL and HTTP/1.1 403 Forbidden) is due to the upstream proxy deliberately denying tunnels to api.telegram.org:443.
  • The current Expected section for this test implicitly assumes that the proxy will allow Telegram and that a 2xx/3xx HTTP status will be returned; it treats a proxy‑generated 403 as a test failure even though the underlying NemoClaw/OpenShell behavior is correct.

In other words, the test conflates:

  1. “Proxy wiring is broken / bypassed” (which would be a NemoClaw bug), with
  2. “Proxy path works but proxy policy blocks this host” (which is an environment/proxy policy issue).

Because the test step 6 currently expects a non‑error STATUS and not an ERR_PROXY_TUNNEL, this scenario registers as a failure even though the NemoClaw/OpenShell proxy integration is functioning as designed.

A fix would be to adjust the test’s Expected result to:

  • Treat any outcome where Node/curl go through the proxy (CONNECT via 10.200.0.1:3128) as a wiring success, regardless of whether the proxy returns 2xx/3xx or 4xx, and
  • Reserve test failure for cases where Node/curl attempt direct connections to api.telegram.org that bypass the proxy (e.g., ECONNREFUSED to api.telegram.org:443 with no proxy CONNECT shown in the logs).

Bug Details

Field Value
Priority Unprioritized
Action Dev - Open - To fix
Disposition Open issue
Module Machine Learning - NemoClaw
Keyword NemoClaw, NEMOCLAW_GH_SYNC_APPROVAL

[NVB#6192899]

Metadata

Metadata

Assignees

Labels

NV QABugs found by the NVIDIA QA TeamPRRPRR recommendation trackingarea: networkingDNS, proxy, TLS, ports, host aliases, or connectivityv0.0.65Release target

Type

No fields configured for Bug.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions