-
Notifications
You must be signed in to change notification settings - Fork 5.3k
Description
Envoy acting as mTLS terminating TCP proxy makes cluster connections ahead of TLS verification (optimistic)
Any incoming connection in turn causes consumption of connections on the upstream services even when client do nothing but a simple tcp connection, without presenting any certificate
I think the behavior is an optimization for the "good case" where if the mtls handshake ends successfully, we have the upstream already connected, but that behavior should at least be configurable such as if my service doesn't want to handle a bunch of empty connections, and wants to rely on envoy proxy as a filter for passing only valid mtls verified connections, I should be able to?
To reproduce with attached config
Connect (with nc so no cert) to ingress, immediately envoy connects to the upstream cluster service, even if it later just closes said connection when mtls handshake fails
# Minimalistic mtls terminating tcp proxy config:
# Laurent Demailly
admin:
access_log_path: /tmp/admin_access.log
profile_path: /tmp/envoy.prof
address:
socket_address: { address: 0.0.0.0, port_value: 15555 }
static_resources:
listeners:
- address:
socket_address:
address: 0.0.0.0
port_value: 15443
filter_chains:
- filters:
- name: envoy.filters.network.tcp_proxy
config:
stat_prefix: ingress_tcp
cluster: service_cluster
tls_context:
require_client_certificate: true
common_tls_context:
validation_context:
trusted_ca:
filename: conf/cert-tmp/ca.crt
# more needed to fully verify the client
tls_certificates:
- certificate_chain:
filename: conf/cert-tmp/server.crt
private_key:
filename: conf/cert-tmp/server.key
clusters:
name: service_cluster
hosts:
socket_address:
address: service
port_value: 15111
connect_timeout: 15s
type: strict_dns
test script:
#! /bin/sh
# Demonstrate/test minimal mTLS terminating envoy TCP proxy config
# "ingress" is envoy
# "service" trivial tcp service (sends 'working + date of listening')
# "client" lets you connects directly, with certs/openssl or not through ingress
#
# issue: envoy connects to service as soon as client connects to ingress even if no valid tls handshake happens
# this should be tunable to avoid a lot of bogus empty socket connections on service from bad clients (clients without certs)
#
# Laurent Demailly
INGRESS_TAG=mtls-ingress-test1:latest
NETWORK=envoy-tls-test1
./cert-gen.sh
docker network create --driver bridge $NETWORK
function cleanup {
docker stop ingress; docker rm ingress
docker stop client; docker rm client
docker stop service; docker rm service
}
cleanup 2>&1 > /dev/null
set -e
docker build -f Dockerfile.ingress -t $INGRESS_TAG .
# Ingress / Envoy proxy verifying and terminating mTLS
docker run --name ingress --network $NETWORK -p 15555:15555 -p 15443:15443 -d $INGRESS_TAG
# Server
docker run --name service --network $NETWORK -d alpine sh -c 'while true; do date; echo working `date` | nc -v -l -p 15111; done'
echo "direct connection: nc -v service 15111"
echo "through envoy (no tls): nc -v ingress 15443"
echo "with tls:"
echo "apk add openssl"
echo "openssl s_client -verify_return_error -no_ticket -cert /data/cert-tmp/client1.crt -key /data/cert-tmp/client1.key -CAfile /data/cert-tmp/ca.crt -connect ingress:15443"
docker run --name client --network $NETWORK -v `pwd`:/data -it alpine ash
From security disclosure, this is fine to discuss in the open
Related: #9023 and #2800