mTLS connection between Gateway and Backend fails for TCPRoute

# Bug Report: mTLS connection between Gateway and Backend fails for TCPRoute

## Description

When configuring mTLS (mutual TLS) between Envoy Gateway and a backend service using `TCPRoute`, `BackendTLSPolicy`, and `EnvoyProxy.backendTLS.clientCertificateRef`, the mTLS connection fails. The gateway cannot establish a secure connection to the backend.

**Expected behavior:** The gateway should establish an mTLS connection to the backend by:

- Validating the backend's server certificate using the CA from BackendTLSPolicy
- Presenting the client certificate from EnvoyProxy.backendTLS.clientCertificateRef for mutual authentication

**Actual behavior:**

- BackendTLSPolicy shows `Accepted: True`, `ResolvedRefs: True`
- However, the mTLS connection fails - no TLS configuration is applied to the upstream cluster (`transport_socket: null`)
- Neither the server CA nor the client certificate is loaded into Envoy
- All connections to the backend fail
- When GatewayClass does NOT have `parametersRef` (no `clientCertificateRef` for mTLS), server-only TLS works correctly

**Key finding:** Server-only TLS (without client certificate) works correctly. The issue occurs specifically when `EnvoyProxy.backendTLS.clientCertificateRef` is configured via `GatewayClass.parametersRef` for mTLS.

## Repro steps

### 1. Install Prerequisites

```bash
# Install cert-manager
helm repo add jetstack https://charts.jetstack.io --force-update
helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager --create-namespace \
  --version v1.16.1 --set crds.enabled=true

# Install trust-manager
helm upgrade trust-manager jetstack/trust-manager \
  --install --namespace cert-manager --wait

# Install Gateway API CRDs (experimental channel)
kubectl apply --server-side -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.1/experimental-install.yaml

# Install Envoy Gateway
helm install envoy-gateway oci://docker.io/envoyproxy/gateway-helm \
  --version v1.6.3 -n envoy-gateway-system --create-namespace

# Create namespace
kubectl create namespace test
```

### 2. Create Certificates

```yaml
# certificates.yaml - Server CA, Client CA, and leaf certificates
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: server-ca-selfsigned
  namespace: cert-manager
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: server-ca-cert
  namespace: cert-manager
spec:
  isCA: true
  commonName: server-ca
  secretName: server-ca-secret
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    name: server-ca-selfsigned
    kind: Issuer
  secretTemplate:
    labels:
      app.kubernetes.io/name: server-ca
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: server-ca-issuer
spec:
  ca:
    secretName: server-ca-secret
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: client-ca-selfsigned
  namespace: cert-manager
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: client-ca-cert
  namespace: cert-manager
spec:
  isCA: true
  commonName: client-ca
  secretName: client-ca-secret
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    name: client-ca-selfsigned
    kind: Issuer
  secretTemplate:
    labels:
      app.kubernetes.io/name: client-ca
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: client-ca-issuer
spec:
  ca:
    secretName: client-ca-secret
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: collector-server-cert
  namespace: test
spec:
  secretName: collector-server-tls
  issuerRef:
    name: server-ca-issuer
    kind: ClusterIssuer
  commonName: syslog-collector
  dnsNames:
    - syslog-collector
    - syslog-collector.test
    - syslog-collector.test.svc
    - syslog-collector.test.svc.cluster.local
  usages: [server auth, digital signature, key encipherment]
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: gateway-client-cert
  namespace: test
spec:
  secretName: gateway-client-tls
  issuerRef:
    name: client-ca-issuer
    kind: ClusterIssuer
  commonName: envoy-gateway-client
  usages: [client auth, digital signature, key encipherment]
---
apiVersion: trust.cert-manager.io/v1alpha1
kind: Bundle
metadata:
  name: server-ca-bundle
spec:
  sources:
  - secret:
      key: "ca.crt"
      selector:
        matchLabels:
          app.kubernetes.io/name: server-ca
  target:
    configMap:
      key: "ca.crt"
    namespaceSelector:
      matchLabels:
        kubernetes.io/metadata.name: test
---
apiVersion: trust.cert-manager.io/v1alpha1
kind: Bundle
metadata:
  name: client-ca-bundle
spec:
  sources:
  - secret:
      key: "ca.crt"
      selector:
        matchLabels:
          app.kubernetes.io/name: client-ca
  target:
    configMap:
      key: "ca.crt"
    namespaceSelector:
      matchLabels:
        kubernetes.io/metadata.name: test
```

### 3. Deploy Backend (OTel Collector with mTLS)

```yaml
# collector.yaml
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: syslog-collector-config
  namespace: test
data:
  config.yaml: |
    receivers:
      syslog:
        tcp:
          listen_address: "0.0.0.0:514"
          tls:
            cert_file: /etc/collector/tls/tls.crt
            key_file: /etc/collector/tls/tls.key
            client_ca_file: /etc/collector/client-ca/ca.crt  # Requires client cert
        protocol: rfc5424
    exporters:
      debug:
        verbosity: detailed
    service:
      pipelines:
        logs:
          receivers: [syslog]
          exporters: [debug]
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: syslog-collector
  namespace: test
spec:
  serviceName: syslog-collector
  replicas: 1
  selector:
    matchLabels:
      app: syslog-collector
  template:
    metadata:
      labels:
        app: syslog-collector
    spec:
      containers:
      - name: collector
        image: otel/opentelemetry-collector-contrib:0.96.0
        args: ["--config=/etc/otel/config.yaml"]
        ports:
        - containerPort: 514
        volumeMounts:
        - name: config
          mountPath: /etc/otel
        - name: server-tls
          mountPath: /etc/collector/tls
          readOnly: true
        - name: client-ca
          mountPath: /etc/collector/client-ca
          readOnly: true
      volumes:
      - name: config
        configMap:
          name: syslog-collector-config
      - name: server-tls
        secret:
          secretName: collector-server-tls
      - name: client-ca
        configMap:
          name: client-ca-bundle
---
apiVersion: v1
kind: Service
metadata:
  name: syslog-collector
  namespace: test
spec:
  selector:
    app: syslog-collector
  ports:
  - port: 514
    targetPort: 514
```

### 4. Configure Gateway with mTLS

```yaml
# gateway.yaml
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: custom-proxy-config
  namespace: envoy-gateway-system
spec:
  backendTLS:
    clientCertificateRef:
      kind: Secret
      name: gateway-client-tls
      namespace: test
---
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: envoy-gateway-class-mtls
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
  parametersRef:
    group: gateway.envoyproxy.io
    kind: EnvoyProxy
    name: custom-proxy-config
    namespace: envoy-gateway-system
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: tcp-gateway
  namespace: test
spec:
  gatewayClassName: envoy-gateway-class-mtls
  listeners:
  - name: tcp-syslog
    protocol: TCP
    port: 514
---
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: TCPRoute
metadata:
  name: syslog-route
  namespace: test
spec:
  parentRefs:
  - name: tcp-gateway
    sectionName: tcp-syslog
  rules:
  - backendRefs:
    - name: syslog-collector
      port: 514
---
apiVersion: gateway.networking.k8s.io/v1
kind: BackendTLSPolicy
metadata:
  name: syslog-backend-tls
  namespace: test
spec:
  targetRefs:
  - group: ""
    kind: Service
    name: syslog-collector
  validation:
    caCertificateRefs:
    - group: ""
      kind: ConfigMap
      name: server-ca-bundle
    hostname: syslog-collector.test.svc.cluster.local
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
  name: allow-envoy-gateway-client-cert
  namespace: test
spec:
  from:
  - group: gateway.networking.k8s.io
    kind: Gateway
    namespace: envoy-gateway-system
  to:
  - group: ""
    kind: Secret
    name: gateway-client-tls
```

### 5. Apply and Test

```bash
kubectl apply -f certificates.yaml
kubectl apply -f collector.yaml
kubectl apply -f gateway.yaml

# Wait for gateway IP
GATEWAY_IP=$(kubectl get gateway tcp-gateway -n test -o jsonpath='{.status.addresses[0].value}')

# Send test message
echo "<14>1 2026-01-31T12:00:00Z host test - - - Test message" | nc $GATEWAY_IP 514

# Check Envoy config dump - shows transport_socket: null
```bash
ENVOY_POD=$(kubectl get pods -n envoy-gateway-system -l gateway.envoyproxy.io/owning-gateway-name=tcp-gateway -o jsonpath='{.items[0].metadata.name}')
kubectl port-forward -n envoy-gateway-system $ENVOY_POD 19000:19000 &
curl -s curl -s localhost:19000/config_dump?resource=dynamic_active_clusters | \
  jq '.configs[].cluster | select(.name | contains("tcproute")) | {name, transport_socket}'
```

## Environment

- **Envoy Gateway Version**: v1.6.3
- **Envoy Version**: (bundled with Envoy Gateway v1.6.3)
- **Kubernetes Version**: 1.30
- **Gateway API CRDs**: Experimental channel v1.4.1
  - BackendTLSPolicy: v1 (GA)
  - TCPRoute: v1alpha2
- **cert-manager**: v1.16.1
- **trust-manager**: v0.7.0

## Logs

### Envoy Gateway Controller Logs

```bash
kubectl logs -n envoy-gateway-system deployment/envoy-gateway --tail=50
```

```text
INFO    gateway-api     runner/runner.go:137    received an update      {"runner": "gateway-api", "key": "gateway.envoyproxy.io/gatewayclass-controller"}
INFO    infrastructure  runner/runner.go:109    received an update      {"runner": "infrastructure", "key": "test/tcp-gateway", "delete": false}
INFO    provider        kubernetes/controller.go:306    reconciling gateways    {"runner": "provider"}
INFO    provider        kubernetes/controller.go:1129   processing OIDC HMAC Secret     {"runner": "provider", "namespace": "envoy-gateway-system", "name": "envoy-oidc-hmac"}
INFO    provider        kubernetes/controller.go:1210   processing Secret       {"runner": "provider", "namespace": "test", "name": "gateway-client-tls"}
INFO    provider        kubernetes/controller.go:1151   processing Envoy TLS Secret     {"runner": "provider", "namespace": "envoy-gateway-system", "name": "envoy"}
INFO    provider        kubernetes/controller.go:1559   processing Gateway      {"runner": "provider", "namespace": "test", "name": "tcp-gateway"}
INFO    provider        kubernetes/routes.go:427        processing TCPRoute     {"runner": "provider", "namespace": "test", "name": "syslog-route"}
INFO    provider        kubernetes/controller.go:663    processing Backend      {"runner": "provider", "kind": "Service", "namespace": "test", "name": "syslog-collector"}
INFO    provider        kubernetes/controller.go:681    added Service to resource tree  {"runner": "provider", "kind": "Service", "namespace": "test", "name": "syslog-collector"}
INFO    provider        kubernetes/controller.go:586    reconciled gateways successfully        {"runner": "provider"}
```

### Envoy Proxy Logs (Data Plane)

```bash
kubectl logs -n envoy-gateway-system -l gateway.envoyproxy.io/owning-gateway-name=tcp-gateway -c envoy --tail=20
```

```text
{":authority":null,"bytes_received":0,"bytes_sent":0,"connection_termination_details":null,"downstream_local_address":"10.224.1.3:10514","downstream_remote_address":"<REDACTED_PUBLIC_IP>:63770","duration":0,"method":null,"protocol":null,"requested_server_name":null,"response_code":0,"response_code_details":null,"response_flags":"UH","route_name":null,"start_time":"2026-01-31T23:47:47.026Z","upstream_cluster":"tcproute/test/syslog-route/rule/-1","upstream_host":null,"upstream_local_address":null,"upstream_transport_failure_reason":null}
{":authority":null,"bytes_received":0,"bytes_sent":0,"connection_termination_details":null,"downstream_local_address":"10.224.1.3:10514","downstream_remote_address":"70.37.26.50:55129","duration":0,"method":null,"protocol":null,"requested_server_name":null,"response_code":0,"response_code_details":null,"response_flags":"UH","route_name":null,"start_time":"2026-01-31T23:47:48.045Z","upstream_cluster":"tcproute/test/syslog-route/rule/-1","upstream_host":null,"upstream_local_address":null,"upstream_transport_failure_reason":null}
```

### Envoy Config Shows No TLS

When checking the Envoy config dump, the upstream cluster has no TLS configuration:

```bash
ENVOY_POD=$(kubectl get pods -n envoy-gateway-system -l gateway.envoyproxy.io/owning-gateway-name=tcp-gateway -o jsonpath='{.items[0].metadata.name}')
kubectl port-forward -n envoy-gateway-system $ENVOY_POD 19000:19000 &
curl -s localhost:19000/config_dump?resource=dynamic_active_clusters | \
  jq '.configs[].cluster | select(.name | contains("tcproute")) | {name, transport_socket}'
```

```json
{
  "name": "tcproute/test/syslog-route/rule/-1",
  "transport_socket": null
}
```

### No client or server CA certificate volume mounts in Gateway Proxy pod

```bash
kubectl describe pod -n envoy-gateway-system -l gateway.envoyproxy.io/owning-gateway-name=tcp-gateway | grep -A 10 "Volumes:"
Volumes:
  certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  envoy
    Optional:    false
  sds:
    Type:        ConfigMap (a volume populated by a ConfigMap)
    Name:        envoy-test-tcp-gateway-6c227073
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
```

### Collector Logs show no logs have been received

```bash
kubectl logs -n test syslog-collector-0
```


Provide feedback

Saved searches

Use saved searches to filter your results more quickly

mTLS connection between Gateway and Backend fails for TCPRoute #8143

Bug Report: mTLS connection between Gateway and Backend fails for TCPRoute

Description

Repro steps

1. Install Prerequisites

2. Create Certificates

3. Deploy Backend (OTel Collector with mTLS)

4. Configure Gateway with mTLS

5. Apply and Test

Environment

Logs

Envoy Gateway Controller Logs

Envoy Proxy Logs (Data Plane)

Envoy Config Shows No TLS

No client or server CA certificate volume mounts in Gateway Proxy pod

Collector Logs show no logs have been received

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

mTLS connection between Gateway and Backend fails for TCPRoute #8143

Description

Bug Report: mTLS connection between Gateway and Backend fails for TCPRoute

Description

Repro steps

1. Install Prerequisites

2. Create Certificates

3. Deploy Backend (OTel Collector with mTLS)

4. Configure Gateway with mTLS

5. Apply and Test

Environment

Logs

Envoy Gateway Controller Logs

Envoy Proxy Logs (Data Plane)

Envoy Config Shows No TLS

No client or server CA certificate volume mounts in Gateway Proxy pod

Collector Logs show no logs have been received

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions