feat(web_fetch): proxy support with SSRF-safe HTTP CONNECT / SOCKS5 tunnelling#3357
Closed
ningqipeng wants to merge 2 commits into
Closed
feat(web_fetch): proxy support with SSRF-safe HTTP CONNECT / SOCKS5 tunnelling#3357ningqipeng wants to merge 2 commits into
ningqipeng wants to merge 2 commits into
Conversation
…unnelling
## Problem
web_fetch intentionally bypasses netclient proxy settings (source comment
in netclient.go). This means users behind corporate firewalls or GFW
cannot use web_fetch to reach external sites (GitHub, npm, etc.), even
when they have a working proxy configured in [network].
## Solution
Wire web_fetch to the existing [network] proxy configuration while
preserving full SSRF protection:
1. netclient.ResolveProxyURL — new exported function that resolves
ProxySpec to URL string (reuses existing customProxyURL logic)
2. webFetch.proxyURL — new field set during tool registration
3. ssrfGuardedClient(proxyURL) — enhanced to tunnel through proxy:
- HTTP/HTTPS proxy: dial proxy -> CONNECT with ORIGINAL hostname
(proxy resolves DNS remotely, essential for GFW users) -> tunnel
- IP literal targets still get SSRF-checked via net.ParseIP
- SOCKS5/SOCKS5h: Transport.Proxy (Go standard library handles it)
- No proxy: existing direct SSRF-safe behavior unchanged
4. ConfineWebFetch + boot.go plumbing for both workspace and fallback
paths
## Files changed
| File | Change |
|------|--------|
| internal/netclient/netclient.go | +ResolveProxyURL() |
| internal/tool/builtin/webfetch.go | webFetch.proxyURL, ssrfGuardedClient proxy tunnel |
| internal/tool/builtin/workspace.go | Workspace.ProxyURL |
| internal/tool/builtin/confine.go | +ConfineWebFetch() |
| internal/boot/boot.go | addBuiltins proxyURL param + fallback fix |
| internal/boot/boot_test.go | test signature update |
| internal/tool/builtin/web_fetch_proxy_test.go | new proxy tests |
## Testing
- go build ./... compiles
- go test ./internal/tool/builtin/ PASS
- go test ./internal/boot/ PASS
- SSRF still blocks 169.254.169.254, 10.0.0.1, 192.168.1.1 through proxy
# Conflicts: # internal/boot/boot_test.go
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
web_fetchintentionally bypassesnetclientproxy settings (source comment innetclient.go):This means users behind corporate firewalls or GFW cannot use
web_fetchto reach external sites (GitHub, npm, etc.), even when they have a working proxy configured in[network].Solution
Wire
web_fetchto the existing[network]proxy configuration while preserving full SSRF protection:netclient.ResolveProxyURL— new exported function that resolvesProxySpec→ URL string (reuses existingcustomProxyURLlogic)webFetch.proxyURL— new field set during tool registrationssrfGuardedClient(proxyURL)— enhanced to tunnel through proxy:CONNECTwith original hostname (proxy resolves DNS remotely, essential for GFW users) → tunnelnet.ParseIPTransport.Proxy(Go standard library handles it)ConfineWebFetch+boot.go— plumbing for both workspace and fallback pathsFiles changed
internal/netclient/netclient.goResolveProxyURL()internal/tool/builtin/webfetch.gowebFetch.proxyURL,ssrfGuardedClientproxy tunnelinternal/tool/builtin/workspace.goWorkspace.ProxyURLinternal/tool/builtin/confine.goConfineWebFetch()internal/boot/boot.goaddBuiltinsproxyURL param + fallback fixinternal/boot/boot_test.gointernal/tool/builtin/web_fetch_proxy_test.goTesting
go build ./...compilesgo test ./internal/tool/builtin/PASSgo test ./internal/boot/PASS169.254.169.254,10.0.0.1,192.168.1.1through proxyConfiguration
After this change, users add to
config.toml:Then
web_fetch https://raw.githubusercontent.com/...works through the proxy.