Greptile Summary

This PR fixes a latent bug where fetchWithSsrFGuard unconditionally called resolvePinnedHostnameWithPolicy() before checking whether a trusted env proxy was configured. In sandboxed environments where UDP DNS is blocked, this caused all web_fetch tool calls to fail before the proxy dispatch was ever reached.

The fix correctly restructures the dispatcher-selection logic in fetch-guard.ts — moving assertExplicitProxySupportsPinnedDns, resolvePinnedHostnameWithPolicy, and createPinnedDispatcher into the else branch that runs only when no trusted env proxy is active. It also adds useEnvProxy: true to the web_fetch tool's call site so it opts into the new proxy-aware path. The safety gate (canUseTrustedEnvProxy requires both the explicit mode flag AND a configured proxy env var) is sound, and the fallback to full DNS pinning when no proxy is configured is verified by a new unit test.

Key findings:

• The core logic restructuring is correct. Dispatcher lifecycle (create → close on redirect → release on success/error) is unchanged.
• The first new test ("skips DNS resolution entirely in trusted env proxy mode") is complete and correct.
• The second new test ("falls back to DNS pinning when … no proxy env var exists") stubs only four of the six proxy env keys checked by hasProxyEnvConfigured() — ALL_PROXY and all_proxy are not cleared. On CI hosts or developer machines where either of those vars is set, the test will silently exercise the proxy path instead of the DNS-pinning path it was designed to guard, potentially masking regressions in the fallback behaviour.

Confidence Score: 4/5

• Safe to merge after fixing the missing ALL_PROXY/all_proxy stubs in the fallback test; the production code change is sound.
• The production logic change in fetch-guard.ts is correct and well-reasoned. The only concrete issue is in the new test: two of the six proxy env keys checked by hasProxyEnvConfigured() are not stubbed, which can cause the fallback test to silently validate the wrong code path in environments where ALL_PROXY or all_proxy is set. This is a P1 test-isolation gap that undermines the guardrail the PR explicitly adds to prevent future regressions.
• src/infra/net/fetch-guard.ssrf.test.ts — the fallback test needs ALL_PROXY and all_proxy stubs added to reliably isolate the DNS-pinning path.

This is a comment left during a code review.
Path: src/infra/net/fetch-guard.ssrf.test.ts
Line: 371-376

Comment:
**Missing `ALL_PROXY`/`all_proxy` stubs weaken test isolation**

`hasProxyEnvConfigured()` iterates over all six keys in `PROXY_ENV_KEYS`, which includes `ALL_PROXY` and `all_proxy`. This test only stubs four of them. If a CI host (or a developer's machine) has `ALL_PROXY` or `all_proxy` set, `hasProxyEnvConfigured()` returns `true`, `canUseTrustedEnvProxy` becomes `true`, and the code takes the proxy path instead of the DNS-pinning path the test expects — causing the test to either fail or silently validate wrong behavior.

Since the goal of this test is to lock in the fallback behaviour with high confidence ("no proxy env var exists"), all six keys need to be cleared:

```suggestion
    // Ensure no proxy vars leak from the test host
    vi.stubEnv("HTTP_PROXY", "");
    vi.stubEnv("HTTPS_PROXY", "");
    vi.stubEnv("ALL_PROXY", "");
    vi.stubEnv("http_proxy", "");
    vi.stubEnv("https_proxy", "");
    vi.stubEnv("all_proxy", "");
```

How can I resolve this? If you propose a fix, please make it concise.

_{Reviews (1): Last reviewed commit: "fix(net): skip DNS pinning when routing ..." | Re-trigger Greptile}

Note: #59007 was opened addressing the same root cause. That PR moves DNS resolution into the non-proxy branch (same structural fix as here) but does not include:

• Protocol-aware proxy detection (hasEnvHttpProxyConfigured vs hasProxyEnvConfigured)
• Pre-DNS hostname policy enforcement in the proxy path (assertHostnameAllowedByPolicy)
• NO_PROXY/no_proxy fallback to DNS pinning (isHostExcludedByNoProxy)
• The useEnvProxy: true opt-in for web_fetch

Codex flagged the hasProxyEnvConfigured gap as P1 on #59007 as well.

Size on this pull changed from S to M largely due to increased test coverage.

Heads up — #59007 (skip DNS pinning before trusted env proxy dispatch) merged on 2026-04-08, which covers the core DNS-before-proxy ordering fix that this PR also addresses.

This PR has additional protocol-aware detection logic that #59007 doesn't include. Is there interest in continuing with that extra coverage, or should this be closed as superseded? Happy either way — just want to make sure it doesn't go stale without a decision.

@mjamiv Yeah, I saw that. I replied on your comment here with the changes I'm currently working on for this branch:
#43821 (comment)

I’ve rebased this branch onto the latest main, which is moving pretty quickly.

The change to make web_fetch respect a trusted proxy for DNS resolution is now behind an explicit config value, so the default behaviour remains unchanged and SSRF protections stay in place unless it is deliberately enabled.

@mjamiv does this provide what you need?

This PR also closes out #63565 ("web_fetch bypasses env proxy; SSRF guard DNS pinning fails in proxy-only envs"). That issue describes the same proxy-only sandbox scenario this PR is hardening for — adding it to the Linked Issue/PR list may help with review routing. Happy to test the branch against a proxy-only sandbox deployment if that would help get this unstuck.

@mjamiv

Thanks, that makes sense. #63565 is the same proxy-only sandbox failure mode this branch is narrowing down, so I’ve added it to the Linked Issue/PR list for review routing.

I have now force-pushed the rebased branch again on top of current main and re-ran pnpm test on both main and this branch. After refreshing the stale generated base-schema file on the branch, the visible failing tests on this branch matched the current main baseline, with no extra visible branch-only failure left over from this PR.

I’d definitely welcome a proxy-only sandbox test on your side if you have time. That is the exact deployment shape this branch is intended to cover. The live verification I already ran against a completely fresh bleeding-edge NemoClaw / OpenShell / OpenClaw build and deploy showed that, once tools.web.fetch.useTrustedEnvProxy is enabled, web_fetch does route through the trusted env proxy and can reach upstream services. The remaining example.com-style failures I hit during that work traced to NVIDIA/OpenShell#813 rather than to this OpenClaw change.

Tested this branch end-to-end in a proxy-only sandbox deployment (HTTP CONNECT egress, UDP DNS blocked, HTTPS_PROXY/HTTP_PROXY routed through EnvHttpProxyAgent). Short version: it works.

• useTrustedEnvProxy: true flows cleanly through the zod schema, resolveFetchUseTrustedEnvProxy, and into runWebFetch / fetchWithSsrFGuard.
• Trusted-env path returns real HTTP responses for allowlisted hosts where the STRICT baseline fails at DNS as expected.
• assertHostnameAllowedByPolicy still fires pre-dispatch in the trusted-env branch — confirmed for allowlist misses and for the cloud-metadata IP (169.254.169.254). That pre-DNS block is the load-bearing reassurance for anyone worried about this path widening the SSRF envelope.
• NO_PROXY exclusion correctly flips canUseTrustedEnvProxy to false and falls through to the pinned-DNS branch.

This covers the shape of environment described in #63565. Rolled the test environment back to stock afterward, no residual changes. Happy to re-run if you rebase or want additional scenarios.

Hi @cosmicnet, just checking in on this PR. This fix seems important for users running OpenClaw behind Clash or other transparent proxies where UDP is blocked and all DNS must go through the proxy.

Could you let us know if there's anything blocking this from being merged, or if it needs any additional information from the maintainers?

@smallmj thanks for checking in.

From my side, I do not think there is any substantive author-side blocker left on this branch. The change is implemented, I have manually verified it in a live OpenShell/NemoClaw deployment, and @mjamiv has independently verified it end to end in a proxy-only sandbox.

I do think part of the confusion here is that #59007 was treated as superseding this PR, but it did not actually pull in all of the remaining work from this branch. It landed the narrower DNS-before-proxy ordering fix, but it did not include the explicit tools.web.fetch.useTrustedEnvProxy opt-in for web_fetch, the protocol-aware HTTP(S) proxy detection, the NO_PROXY fallback to the pinned-DNS path, or the pre-DNS hostname policy checks in the trusted-proxy path. In other words, it did not fully supersede this PR, and I suspect that may be part of why this has stalled.

So from my side, I believe this is ready to pull in. If there is still a specific maintainer concern blocking merge, I would appreciate a pointer to it.

Comprehensive Proxy Test Results (2026-04-13)

Test Environment

• OpenClaw Gateway running on Ubuntu, behind Clash (socks5 proxy at 127.0.0.1:7897)
• Gateway started via systemctl (SetLoginEnvironment=no), so shell env vars do NOT propagate to the process

Test 1: Without Tun mode (proxy env vars not set)

curl with explicit -x flag (proxy works):

curl -x http://127.0.0.1:7897 https://api.ip.sb/ip
→ 216.236.40.184 (Hong Kong exit)

curl without proxy (direct):

curl https://api.ip.sb/ip
→ 2408:8207:... (domestic IPv6, no proxy)

web_fetch tool:

web_fetch https://api.ip.sb/ip
→ 221.219.70.115 (Beijing, direct — no proxy) ❌

Conclusion: web_fetch does not use proxy even when proxy is working for curl.

Test 2: With Tun mode + DNS override enabled (no HTTPS_PROXY env var set)

All DNS now routes through Clash TUN, which returns 198.18.x.x (RFC 2544 benchmark IP range) for most domains.

curl direct (Tun catches all):

curl https://api.ip.sb/ip
→ 216.236.40.180 (proxy exit) ✅

web_fetch:

web_fetch https://icanhazip.com
→ Blocked: resolves to private/internal/special-use IP address ❌

web_fetch https://checkip.amazonaws.com
→ Blocked: resolves to private/internal/special-use IP address ❌

web_fetch https://cloudflare.com/cdn-cgi/trace
→ Blocked: resolves to private/internal/special-use IP address ❌

Root cause: Clash TUN DNS returns 198.18.x.x (benchmark IPs). OpenClaw SSRF guard correctly blocks these as special-use addresses.

Conclusion: Tun mode breaks web_fetch via DNS + SSRF interaction.

Test 3: Tun mode OFF, DNS override OFF, HTTPS_PROXY env var set in systemctl

Properly injected proxy env into Gateway:

systemctl --user set-environment HTTPS_PROXY=http://127.0.0.1:7897
systemctl --user set-environment ALL_PROXY=http://127.0.0.1:7897
systemctl --user restart openclaw-gateway

Verification:

systemctl --user show-environment | grep proxy
→ ALL_PROXY=http://127.0.0.1:7897 ✅
→ HTTPS_PROXY=http://127.0.0.1:7897 ✅

curl (no -x flag, Tun off):

curl https://api.ip.sb/ip
→ 216.236.40.177 (proxy exit) ✅

web_fetch:

web_fetch https://api.ip.sb/ip
→ 221.219.70.115 (direct, no proxy) ❌

Conclusion: Even with HTTPS_PROXY correctly set in the Gateway process environment, web_fetch still does not use the proxy. This confirms web_fetch itself does not read HTTPS_PROXY/ALL_PROXY env vars — it uses Node.js built-in fetch which goes direct.

Summary

This is why PR #58034 is needed. The core issue is not the environment or DNS — it's that web_fetch does not respect HTTPS_PROXY. The useEnvProxy: true flag in this PR would make web_fetch explicitly use the proxy configured in openclaw.json, solving all of the above.

Additional note from cosmicnet: PR #59007 merged the DNS-before-proxy ordering fix for the core network layer, but did NOT include the web_fetch opt-in. This PR (#58034) adds that missing piece.

Rebased onto current main and force-pushed the branch.

I reran the affected coverage for the edited paths only: the NO_PROXY matcher, fetch guard SSRF path, web_fetch trusted-env-proxy path, and the config-doc baseline check. Those all pass on the rebased branch. I have not rerun the full suite.

The branch still carries the web_fetch-specific trusted env proxy opt-in and docs on top of the upstream proxy/fetch changes already on main.

For context, I think this branch addresses a different case from #60785 (which was just closed).

#60785 is about fake-IP setups via tools.web.fetch.ssrfPolicy.allowRfc2544BenchmarkRange. That makes sense for deployments where local DNS still succeeds and returns RFC2544 198.18.0.0/15 addresses that are then handled by the network layer.

This branch is aimed at proxy-only environments where local DNS is blocked or unavailable and web_fetch needs the trusted HTTP(S) env proxy itself to resolve DNS. Current main still keeps web_fetch on the local pinned-DNS path by default even when HTTP_PROXY or HTTPS_PROXY is set. This branch adds an explicit tools.web.fetch.useTrustedEnvProxy opt-in for that path, while still keeping hostname policy checks before proxy dispatch and falling back to pinned DNS when NO_PROXY excludes the target or no matching proxy env var applies.

Codex review: needs maintainer review before merge.

What this changes:

The PR adds a default-off tools.web.fetch.useTrustedEnvProxy config/docs/schema surface, wires it into web_fetch guarded fetch calls, and adds trusted-proxy, NO_PROXY, hostname-policy, and config regression coverage.

Maintainer follow-up before merge:

This is useful active contributor work, but it is dirty against current main and changes SSRF/trusted-proxy network semantics, so maintainers should review the trust boundary and rebase before landing rather than sending it to autonomous repair.

Security review:

Security review cleared: The diff changes network routing, but the reviewed design keeps the new path default-off, protocol/NO_PROXY gated, and hostname-policy guarded; no concrete secrets, CI, dependency, or supply-chain regression was found.

Thanks. Latest push is bd87eae254.

I believe the concrete implementation pieces called out here are now present on the branch:

• tools.web.fetch.useTrustedEnvProxy is an explicit, default-off config field.
• createWebFetchTool wires that config through to fetchWithWebToolsNetworkGuard.
• Default web_fetch behavior still keeps strict DNS pinning even when HTTP_PROXY is configured.
• Opt-in trusted env proxy mode uses env proxy dispatch only when a matching HTTP(S) proxy applies.
• NO_PROXY exclusions and no matching proxy env both fall back to the pinned-DNS path.
• Hostname-policy checks still run before trusted-proxy dispatch.
• Docs, schema, generated config metadata, and regression tests are updated.

Focused local verification after the final push contents:

• pnpm exec oxlint src/agents/tools/web-fetch.ts src/agents/tools/web-fetch.ssrf.test.ts src/agents/tools/web-tools.fetch.test.ts src/infra/net/fetch-guard.ts src/infra/net/fetch-guard.ssrf.test.ts src/infra/net/proxy-env.ts src/infra/net/proxy-env.test.ts
• pnpm vitest run src/infra/net/proxy-env.test.ts src/infra/net/fetch-guard.ssrf.test.ts src/agents/tools/web-tools.fetch.test.ts src/agents/tools/web-fetch.ssrf.test.ts src/config/schema.test.ts
• pnpm config:docs:check

All passed locally.

On the Firecrawl/provider fallback question: I left that as an intentional scope boundary for this PR. This branch makes the built-in web_fetch page-fetch path work in proxy-only deployments while preserving default strict mode. Firecrawl is a provider/API fallback path with a separate trust boundary, so I think maintainers should decide whether tools.web.fetch.useTrustedEnvProxy should also apply to provider API calls, or whether that should be a follow-up with separate docs/tests.

Post-#76887 status: still relevant, but this remains a separate web_fetch trusted-env-proxy change. #76887 only landed scoped fake-IP allowances for trusted provider/web-search hosts and did not make web_fetch honor ambient proxy env routing.

Current branch state is still DIRTY / conflicting with failing stale CI, so the next useful step is a rebase against current main plus fresh focused SSRF/web_fetch tests and Testbox pnpm check:changed. The trust-boundary decision remains separate from #76887.

Landed the maintainer-side version on main in 66336bf.

Kept the setting default-off as tools.web.fetch.useTrustedEnvProxy, preserved strict DNS pinning by default, enforced hostname/IP policy before trusted env-proxy dispatch, corrected NO_PROXY wildcard handling to match Undici, and added docs/schema/changelog plus regression coverage.

Verification before landing:

• targeted local tests for proxy-env, fetch-guard SSRF, web-tools fetch, web-fetch SSRF, and schema
• pnpm config:docs:check
• git diff --check
• Testbox pnpm check:changed on the rebased commit: tbx_01kqqw5q42q6wkwmmxn9deezk6

Thanks @cosmicnet and @mjamiv.