how to properly set UpstreamTlsContext for forward proxy #15500
Description
Hi,
I am trying to configure envoy as a forward proxy with the following yaml config:
(modified from https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/dynamic_forward_proxy_filter)
admin:
access_log_path: /tmp/admin_access.log
address:
socket_address:
protocol: TCP
address: 0.0.0.0
port_value: 3990
static_resources:
listeners:
- name: listener_0
address:
socket_address:
protocol: TCP
address: 0.0.0.0
port_value: 3991
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: ingress_http
http_protocol_options:
accept_http_10: true
route_config:
name: local_route
virtual_hosts:
- name: local_service
domains: ["*"]
routes:
- match:
prefix: "/"
route:
cluster: dynamic_forward_proxy_cluster
- match:
connect_matcher: {}
route:
cluster: dynamic_forward_proxy_cluster
upgrade_configs:
- upgrade_type: CONNECT
http_filters:
- name: envoy.filters.http.dynamic_forward_proxy
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.dynamic_forward_proxy.v3.FilterConfig
dns_cache_config:
name: dynamic_forward_proxy_cache_config
dns_lookup_family: V4_ONLY
- name: envoy.filters.http.router
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
clusters:
- name: dynamic_forward_proxy_cluster
connect_timeout: 1s
lb_policy: CLUSTER_PROVIDED
cluster_type:
name: envoy.clusters.dynamic_forward_proxy
typed_config:
"@type": type.googleapis.com/envoy.extensions.clusters.dynamic_forward_proxy.v3.ClusterConfig
dns_cache_config:
name: dynamic_forward_proxy_cache_config
dns_lookup_family: V4_ONLY
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
common_tls_context:
validation_context:
trusted_ca: {filename: /etc/ssl/certs/ca-certificates.crt}
This config work fine when not setting the transport_socket field.
However when transport_socket is present and configured as above, tls connection to the upstream will fail, could you help me understand what may have led to the failure after setting UpstreamTlsContext in transport_socket?
curl error message
curl -x http://localhost:3991 https://fuhry.com -v
* Rebuilt URL to: https://fuhry.com/
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 3991 (#0)
* Establish HTTP proxy tunnel to fuhry.com:443
> CONNECT fuhry.com:443 HTTP/1.1
> Host: fuhry.com:443
> User-Agent: curl/7.47.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 OK
< date: Mon, 15 Mar 2021 22:19:46 GMT
< server: envoy
<
* Proxy replied OK to CONNECT request
* found 129 certificates in /etc/ssl/certs/ca-certificates.crt
* found 520 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* gnutls_handshake() failed: An unexpected TLS packet was received.
* Closing connection 0
curl: (35) gnutls_handshake() failed: An unexpected TLS packet was received.
envoy logs
[2021-03-15 22:19:46.691][22330][debug][conn_handler] [external/envoy/source/server/connection_handler_impl.cc:501] [C0] new connection
[2021-03-15 22:19:46.691][22330][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:254] [C0] new stream
[2021-03-15 22:19:46.692][22330][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:886] [C0][S12483657978785096747] request headers complete (end_stream=false):
':authority', 'fuhry.com:443'
':method', 'CONNECT'
'user-agent', 'curl/7.47.0'
'proxy-connection', 'Keep-Alive'
[2021-03-15 22:19:46.692][22330][debug][forward_proxy] [external/envoy/source/extensions/common/dynamic_forward_proxy/dns_cache_impl.cc:56] thread local lookup for host 'fuhry.com:443'
[2021-03-15 22:19:46.692][22330][debug][forward_proxy] [external/envoy/source/extensions/common/dynamic_forward_proxy/dns_cache_impl.cc:77] cache miss for host 'fuhry.com:443', posting to main thread
[2021-03-15 22:19:46.692][22330][debug][forward_proxy] [external/envoy/source/extensions/filters/http/dynamic_forward_proxy/proxy_filter.cc:142] [C0][S12483657978785096747] waiting to load DNS cache entry
[2021-03-15 22:19:46.692][22320][debug][forward_proxy] [external/envoy/source/extensions/common/dynamic_forward_proxy/dns_cache_impl.cc:215] starting main thread resolve for host='fuhry.com:443' dns='fuhry.com' port='443'
[2021-03-15 22:19:46.715][22320][debug][forward_proxy] [external/envoy/source/extensions/common/dynamic_forward_proxy/dns_cache_impl.cc:232] main thread resolve complete for host 'fuhry.com:443'. 1 results
[2021-03-15 22:19:46.715][22320][debug][forward_proxy] [external/envoy/source/extensions/common/dynamic_forward_proxy/dns_cache_impl.cc:270] host 'fuhry.com:443' address has changed
[2021-03-15 22:19:46.715][22320][debug][upstream] [external/envoy/source/extensions/clusters/dynamic_forward_proxy/cluster.cc:119] Adding host info for fuhry.com:443
[2021-03-15 22:19:46.715][22320][debug][upstream] [external/envoy/source/extensions/clusters/dynamic_forward_proxy/cluster.cc:98] adding new dfproxy cluster host 'fuhry.com:443'
[2021-03-15 22:19:46.715][22320][debug][upstream] [external/envoy/source/common/upstream/upstream_impl.cc:278] transport socket match, socket default selected for host with address 35.194.70.22:443
[2021-03-15 22:19:46.715][22327][debug][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:1201] membership update for TLS cluster dynamic_forward_proxy_cluster added 1 removed 0
[2021-03-15 22:19:46.715][22330][debug][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:1201] membership update for TLS cluster dynamic_forward_proxy_cluster added 1 removed 0
[2021-03-15 22:19:46.715][22332][debug][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:1201] membership update for TLS cluster dynamic_forward_proxy_cluster added 1 removed 0
[2021-03-15 22:19:46.715][22332][debug][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:1208] re-creating local LB for TLS cluster dynamic_forward_proxy_cluster
[2021-03-15 22:19:46.715][22330][debug][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:1208] re-creating local LB for TLS cluster dynamic_forward_proxy_cluster
[2021-03-15 22:19:46.715][22320][debug][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:1201] membership update for TLS cluster dynamic_forward_proxy_cluster added 1 removed 0
[2021-03-15 22:19:46.715][22327][debug][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:1208] re-creating local LB for TLS cluster dynamic_forward_proxy_cluster
[2021-03-15 22:19:46.715][22328][debug][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:1201] membership update for TLS cluster dynamic_forward_proxy_cluster added 1 removed 0
[2021-03-15 22:19:46.715][22328][debug][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:1208] re-creating local LB for TLS cluster dynamic_forward_proxy_cluster
[2021-03-15 22:19:46.715][22320][debug][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:1208] re-creating local LB for TLS cluster dynamic_forward_proxy_cluster
[2021-03-15 22:19:46.715][22320][debug][forward_proxy] [external/envoy/source/extensions/common/dynamic_forward_proxy/dns_cache_impl.cc:289] DNS refresh rate reset for host 'fuhry.com:443', refresh rate 60000 ms
[2021-03-15 22:19:46.715][22330][debug][forward_proxy] [external/envoy/source/extensions/filters/http/dynamic_forward_proxy/proxy_filter.cc:159] [C0][S12483657978785096747] load DNS cache complete, continuing
[2021-03-15 22:19:46.715][22330][debug][router] [external/envoy/source/common/router/router.cc:425] [C0][S12483657978785096747] cluster 'dynamic_forward_proxy_cluster' match for URL ''
[2021-03-15 22:19:46.716][22330][debug][router] [external/envoy/source/common/router/router.cc:582] [C0][S12483657978785096747] router decoding headers:
':authority', 'fuhry.com:443'
':method', 'CONNECT'
':scheme', 'https'
'user-agent', 'curl/7.47.0'
'x-forwarded-proto', 'http'
'x-request-id', '2fbab735-98c7-4076-b6b6-40c2f303528b'
'x-envoy-expected-rq-timeout-ms', '15000'
[2021-03-15 22:19:46.716][22330][debug][pool] [external/envoy/source/common/tcp/original_conn_pool.cc:98] creating a new connection
[2021-03-15 22:19:46.716][22330][debug][pool] [external/envoy/source/common/tcp/original_conn_pool.cc:383] [C1] connecting
[2021-03-15 22:19:46.716][22330][debug][connection] [external/envoy/source/common/network/connection_impl.cc:860] [C1] connecting to 35.194.70.22:443
[2021-03-15 22:19:46.716][22330][debug][connection] [external/envoy/source/common/network/connection_impl.cc:876] [C1] connection in progress
[2021-03-15 22:19:46.716][22330][debug][pool] [external/envoy/source/common/tcp/original_conn_pool.cc:125] queueing request due to no available connections
[2021-03-15 22:19:46.807][22330][debug][connection] [external/envoy/source/common/network/connection_impl.cc:666] [C1] connected
[2021-03-15 22:19:46.992][22330][debug][pool] [external/envoy/source/common/tcp/original_conn_pool.cc:303] [C1] assigning connection
[2021-03-15 22:19:46.992][22330][debug][router] [external/envoy/source/common/router/upstream_request.cc:354] [C0][S12483657978785096747] pool ready
[2021-03-15 22:19:46.992][22330][debug][router] [external/envoy/source/common/router/router.cc:1174] [C0][S12483657978785096747] upstream headers complete: end_stream=false
[2021-03-15 22:19:46.992][22330][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:1484] [C0][S12483657978785096747] encoding headers via codec (end_stream=false):
':status', '200'
'date', 'Mon, 15 Mar 2021 22:19:46 GMT'
'server', 'envoy'
[2021-03-15 22:19:47.123][22330][debug][router] [external/envoy/source/common/router/upstream_request.cc:296] [C0][S12483657978785096747] resetting pool request
[2021-03-15 22:19:47.123][22330][debug][connection] [external/envoy/source/common/network/connection_impl.cc:132] [C1] closing data_to_write=0 type=1
[2021-03-15 22:19:47.123][22330][debug][connection] [external/envoy/source/common/network/connection_impl.cc:241] [C1] closing socket: 1
[2021-03-15 22:19:47.123][22330][debug][connection] [external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:308] [C1] SSL shutdown: rc=1
[2021-03-15 22:19:47.123][22330][debug][connection] [external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:224] [C1]
[2021-03-15 22:19:47.123][22330][debug][pool] [external/envoy/source/common/tcp/original_conn_pool.cc:140] [C1] client disconnected
[2021-03-15 22:19:47.123][22330][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:189] [C0][S12483657978785096747] doEndStream() resetting stream
[2021-03-15 22:19:47.123][22330][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:1533] [C0][S12483657978785096747] stream reset
[2021-03-15 22:19:47.123][22330][debug][connection] [external/envoy/source/common/network/connection_impl.cc:132] [C0] closing data_to_write=421 type=2
[2021-03-15 22:19:47.123][22330][debug][connection] [external/envoy/source/common/network/connection_impl_base.cc:47] [C0] setting delayed close timer with timeout 1000 ms
[2021-03-15 22:19:47.123][22330][debug][pool] [external/envoy/source/common/tcp/original_conn_pool.cc:255] [C1] connection destroyed
[2021-03-15 22:19:47.123][22330][debug][connection] [external/envoy/source/common/network/connection_impl.cc:696] [C0] write flush complete
[2021-03-15 22:19:47.130][22330][debug][connection] [external/envoy/source/common/network/connection_impl.cc:696] [C0] write flush complete
[2021-03-15 22:19:48.122][22330][debug][connection] [external/envoy/source/common/network/connection_impl_base.cc:63] [C0] triggered delayed close
[2021-03-15 22:19:48.122][22330][debug][connection] [external/envoy/source/common/network/connection_impl.cc:241] [C0] closing socket: 1
[2021-03-15 22:19:48.122][22330][debug][conn_handler] [external/envoy/source/server/connection_handler_impl.cc:164] [C0] adding to cleanup list
[2021-03-15 22:19:51.539][22320][debug][main] [external/envoy/source/server/server.cc:199] flushing stats
[2021-03-15 22:19:56.538][22320][debug][main] [external/envoy/source/server/server.cc:199] flushing stats