Cannot call https/tls upstream for grpc - wrong scheme header #16255
Description
Wrong scheme header sent to upstream grpc service
Description:
I understand that when using grpc, tls over http2 is preferred for performance reasons. I have setup an upstream grpc service for cluster discovery and this works with http2 and tls when called from a test client.
However, when it gets called from envoy, the grpc web service logs
The request :scheme header 'http' does not match the transport scheme 'https'
Which is correct because the service is https only. This error stops processing of the request.
I can see the following in the Envoy logs, which shows that despite being configured for https (as far as I can tell!), it is still sending the scheme header as http. If it is relevant, I don't have TLS enabled on the listeners, I am just trying to use it for the grpc service for CDS. I can't find any documentation that says that I can override the scheme in the configuration for the cluster.
[2021-04-30 15:55:05.297][1][debug][router] [source/common/router/router.cc:631] [C0][S7382542047916721558] router decoding headers:
':method', 'POST'
':path', '/envoy.service.cluster.v3.ClusterDiscoveryService/StreamClusters'
':authority', 'xds_cluster'
':scheme', 'http'
'te', 'trailers'
'content-type', 'application/grpc'
'x-envoy-internal', 'true'
'x-forwarded-for', '192.168.48.3'
Relevant part of the envoy.yaml file:
static_resources:
clusters:
- name: xds_cluster
connect_timeout: 15s
per_connection_buffer_limit_bytes: 32768 # 32 KiB
type: STRICT_DNS
lb_policy: ROUND_ROBIN
typed_extension_protocol_options:
envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
"@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
explicit_http_config:
http2_protocol_options:
connection_keepalive:
interval: 30s
timeout: 5s
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
sni: manager.example.io
common_tls_context:
tls_certificates:
- certificate_chain: { "filename": "/etc/ssl/certs/servercert.pem" }
private_key: { "filename": "/etc/ssl/certs/serverkey.pem" }
alpn_protocols:
- h2
validation_context:
match_subject_alt_names:
- suffix: "example.io"
trusted_ca:
filename: /etc/ssl/certs/ca-certificates.crt
load_assignment:
cluster_name: xds_cluster
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: manager.example.io
port_value: 443
dynamic_resources:
cds_config:
resource_api_version: V3
api_config_source:
api_type: GRPC
transport_api_version: V3
grpc_services:
- envoy_grpc:
cluster_name: xds_cluster