Nomad version

Nomad v1.4.3 (f464aca)

Issue

Consul 1.14 introduced changes to the gRPC listener used for Envoy xDS configuration, splitting support for plaintext and TLS-enabled listeners for the protocol across two ports.

An initial fix for discovering the TLS-enabled listener via Consul's /agent/self endpoint landed in #15309, however when utilizing gRPC over TLS the consul connect envoy [...] command internally invoked by Nomad to bootstrap Envoy requires the addition of the -grpc-ca-file or -grpc-ca-path flags (or environment variable equivalents) to inject a trusted CA certificate into the generated Envoy config.

When the gRPC CA has not been explicitly passed, Envoy sidecar proxy tasks will initially deploy but will never retrieve their running configuration from the local Consul agent, sitting in a fail/retry loop unable to negotiate a connection with the gRPC-TLS listener..

To support this, Nomad will likely need to make a CA certificate available within the allocation filesystem wherever a sidecar_service{} stanza has been defined, on disk on each scheduler, or the Consul CLI will need to be extended to allow other means of providing the CA certificate (i.e. as an inline PEM rather than file path).

As a temporary workaround to restore proxy connectivity, the plaintext Consul HTTP and gRPC endpoints may be enabled and bound to a loopback interface within the local Consul agent, as Nomad will default to discovering the plaintext gRPC listener as long as HTTP is the URL schema in the consul.address configuration.