Skip to content

(tcp proxy with mtls:) Delay connection to upstream until TLS handshake is complete #11694

@ldemailly

Description

@ldemailly

Envoy acting as mTLS terminating TCP proxy makes cluster connections ahead of TLS verification (optimistic)

Any incoming connection in turn causes consumption of connections on the upstream services even when client do nothing but a simple tcp connection, without presenting any certificate

I think the behavior is an optimization for the "good case" where if the mtls handshake ends successfully, we have the upstream already connected, but that behavior should at least be configurable such as if my service doesn't want to handle a bunch of empty connections, and wants to rely on envoy proxy as a filter for passing only valid mtls verified connections, I should be able to?

To reproduce with attached config
Connect (with nc so no cert) to ingress, immediately envoy connects to the upstream cluster service, even if it later just closes said connection when mtls handshake fails

# Minimalistic mtls terminating tcp proxy config:
# Laurent Demailly
admin:
  access_log_path: /tmp/admin_access.log
  profile_path: /tmp/envoy.prof
  address:
    socket_address: { address: 0.0.0.0, port_value: 15555 }

static_resources:
  listeners:
    - address:
        socket_address:
          address: 0.0.0.0
          port_value: 15443
      filter_chains:
      - filters:
        - name: envoy.filters.network.tcp_proxy
          config:
            stat_prefix: ingress_tcp
            cluster: service_cluster
        tls_context:
          require_client_certificate: true
          common_tls_context:
            validation_context:
              trusted_ca:
                filename: conf/cert-tmp/ca.crt
                # more needed to fully verify the client
            tls_certificates:
              - certificate_chain:
                  filename: conf/cert-tmp/server.crt
                private_key:
                  filename: conf/cert-tmp/server.key

  clusters:
    name: service_cluster
    hosts:
      socket_address:
        address: service
        port_value: 15111
    connect_timeout: 15s
    type: strict_dns

test script:

#! /bin/sh
# Demonstrate/test minimal mTLS terminating envoy TCP proxy config
# "ingress" is envoy
# "service" trivial tcp service (sends 'working + date of listening')
# "client" lets you connects directly, with certs/openssl or not through ingress
#
# issue: envoy connects to service as soon as client connects to ingress even if no valid tls handshake happens
# this should be tunable to avoid a lot of bogus empty socket connections on service from bad clients (clients without certs)
#
# Laurent Demailly

INGRESS_TAG=mtls-ingress-test1:latest

NETWORK=envoy-tls-test1

./cert-gen.sh

docker network create --driver bridge $NETWORK

function cleanup {
    docker stop ingress; docker rm ingress
    docker stop client; docker rm client
    docker stop service; docker rm service
}

cleanup 2>&1 > /dev/null

set -e
docker build -f Dockerfile.ingress -t $INGRESS_TAG .

# Ingress / Envoy proxy verifying and terminating mTLS
docker run --name ingress --network $NETWORK -p 15555:15555 -p 15443:15443 -d $INGRESS_TAG
# Server
docker run --name service --network $NETWORK -d alpine sh -c 'while true; do date; echo working `date` | nc -v -l -p 15111; done'
echo "direct connection: nc -v service 15111"
echo "through envoy (no tls): nc -v ingress 15443"
echo "with tls:"
echo "apk add openssl"
echo "openssl s_client -verify_return_error -no_ticket -cert /data/cert-tmp/client1.crt -key /data/cert-tmp/client1.key -CAfile /data/cert-tmp/ca.crt -connect ingress:15443"
docker run --name client --network $NETWORK -v `pwd`:/data -it alpine ash

From security disclosure, this is fine to discuss in the open
Related: #9023 and #2800

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions