Is there an existing issue for this?
Version
equal or higher than v1.18.6 and lower than v1.19.0
What happened?
Adding HTTPS listeners with specific hostnames to a Gateway that also has a wildcard HTTP listener causes the wildcard HTTP listener to stop routing traffic entirely. The CiliumEnvoyConfig is not generated at all, and all HTTP requests through the wildcard listener return 404 Not Found from Envoy.
As soon as the HTTP listener is changed from a wildcard (*.lab.inqbeo.org) to a specific hostname (testweb2.lab.inqbeo.org), routing works immediately.
This effectively makes cert-manager HTTP-01 ACME challenges impossible with Cilium Gateway API. The combination of a wildcard HTTP listener with per-hostname HTTPS listeners is the only architecturally viable pattern for HTTP-01 challenges, because:
- HTTPS listeners require specific hostnames — each hostname that needs TLS termination must have its own listener with its own
certificateRefs.
- A ClusterIssuer's HTTP-01 solver must reference a single HTTP listener — the solver's
gatewayHTTPRoute.parentRefs points to one listener by sectionName. This listener must be able to serve challenges for any hostname the issuer manages:
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
spec:
acme:
solvers:
- http01:
gatewayHTTPRoute:
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: main-gateway
namespace: gateway-system
sectionName: catchall-http # ← must work for ALL hostnames
- A wildcard HTTP listener is therefore required — without it, you would need a separate HTTP listener and a separate ClusterIssuer per hostname, which is unmanageable.
This is not an exotic configuration — it is the standard pattern recommended for Gateway API with cert-manager: a single wildcard HTTP listener for ACME challenges (and HTTP-to-HTTPS redirects), alongside per-hostname HTTPS listeners for TLS termination. This bug makes that pattern completely non-functional.
How can we reproduce the issue?
Working configuration — wildcard HTTP listener only:
Gateway:
listeners:
- name: catchall-http
port: 80
protocol: HTTP
hostname: "*.lab.inqbeo.org"
allowedRoutes:
namespaces:
from: All
HTTPRoute (in namespace default):
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: testweb
namespace: default
spec:
parentRefs:
- name: main-gateway
namespace: gateway-system
sectionName: catchall-http
hostnames:
- testweb2.lab.inqbeo.org
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- name: testweb
port: 80
Result: ✅ curl http://testweb2.lab.inqbeo.org returns nginx welcome page.
Broken configuration — add HTTPS listeners:
Gateway:
listeners:
- name: catchall-http
port: 80
protocol: HTTP
hostname: "*.lab.inqbeo.org"
allowedRoutes:
namespaces:
from: All
- name: testweb-https
port: 443
protocol: HTTPS
hostname: testweb.lab.inqbeo.org
tls:
mode: Terminate
certificateRefs:
- name: testweb-inqbeo-org-tls
allowedRoutes:
namespaces:
from: All
- name: testweb2-https
port: 443
protocol: HTTPS
hostname: testweb2.lab.inqbeo.org
tls:
mode: Terminate
certificateRefs:
- name: testweb2-inqbeo-org-tls
allowedRoutes:
namespaces:
from: All
HTTPRoute unchanged — still explicitly targets sectionName: catchall-http.
Result: ❌ curl http://testweb2.lab.inqbeo.org returns HTTP/1.1 404 Not Found with server: envoy. At the same time, cert-manager's HTTP-01 challenge for the HTTPS listener's certificate also fails — the cert-manager logs show wrong status code '404', expected '200' for the /.well-known/acme-challenge/ path. Neither the application HTTPRoute nor the solver HTTPRoute created by cert-manager appear in any CiliumEnvoyConfig:
$ kubectl -n gateway-system get ciliumenvoyconfig -l gateway.networking.k8s.io/owning-gateway=main-gateway -o yaml
apiVersion: v1
items: []
kind: List
The entire Envoy configuration for the Gateway is simply not generated.
Not a workaround — change HTTP listener to specific hostname:
Changing the HTTP listener hostname from *.lab.inqbeo.org to testweb2.lab.inqbeo.org immediately restores routing for that hostname and allows cert-manager to complete its HTTP-01 challenge.
However, this is not a viable workaround — it only works for a single hostname at a time. In any real deployment with multiple hostnames, this would require a separate HTTP listener per hostname, a separate ClusterIssuer per hostname, and manually coordinating them all. It merely confirms where the bug lies.
Impact on cert-manager HTTP-01 challenges
The ClusterIssuer is configured to use a single HTTP listener for all HTTP-01 challenges:
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: inqbeo.org-stepca
spec:
acme:
server: https://ca.inqbeo.org/acme/acme/directory
solvers:
- http01:
gatewayHTTPRoute:
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: main-gateway
namespace: gateway-system
sectionName: catchall-http
When cert-manager needs to validate a certificate, it creates a solver HTTPRoute targeting this HTTP listener with the specific hostname being validated:
spec:
hostnames:
- testweb2.lab.inqbeo.org
parentRefs:
- name: main-gateway
namespace: gateway-system
sectionName: catchall-http
rules:
- matches:
- path:
type: Exact
value: /.well-known/acme-challenge/<token>
backendRefs:
- name: cm-acme-http-solver-ch76n
namespace: gateway-system
port: 8089
This route is accepted (status.parents[].conditions: Accepted: True) but never programmed into Envoy, causing the ACME propagation check to fail with wrong status code '404', expected '200' in a loop until it times out.
The solver pod and service are running and have valid endpoints — the problem is purely that the route is not translated to Envoy configuration. There is no workaround that preserves a single ClusterIssuer for all hostnames.
What did you expect to happen?
Per the Gateway API specification, a wildcard listener hostname (*.lab.inqbeo.org) should match HTTPRoutes with specific hostnames (testweb2.lab.inqbeo.org). The presence of HTTPS listeners on port 443 should not affect HTTP routing on port 80, especially when the HTTPRoute explicitly targets the HTTP listener via sectionName.
Cilium Version
1.18.1
Kernel Version
6.12
Kubernetes Version
1.35.0+rke2r1
Regression
No response
Sysdump
No response
Relevant log output
Anything else?
AI-assisted analysis: The root cause appears to be in the route filtering or Envoy config translation logic. When HTTPS listeners with specific hostnames are added, the entire CiliumEnvoyConfig disappears — suggesting the translation pipeline encounters an error or conflict and drops the config entirely, rather than just failing for individual routes.
Cilium Users Document
Code of Conduct
Is there an existing issue for this?
Version
equal or higher than v1.18.6 and lower than v1.19.0
What happened?
Adding HTTPS listeners with specific hostnames to a Gateway that also has a wildcard HTTP listener causes the wildcard HTTP listener to stop routing traffic entirely. The
CiliumEnvoyConfigis not generated at all, and all HTTP requests through the wildcard listener return404 Not Foundfrom Envoy.As soon as the HTTP listener is changed from a wildcard (
*.lab.inqbeo.org) to a specific hostname (testweb2.lab.inqbeo.org), routing works immediately.This effectively makes cert-manager HTTP-01 ACME challenges impossible with Cilium Gateway API. The combination of a wildcard HTTP listener with per-hostname HTTPS listeners is the only architecturally viable pattern for HTTP-01 challenges, because:
certificateRefs.gatewayHTTPRoute.parentRefspoints to one listener bysectionName. This listener must be able to serve challenges for any hostname the issuer manages:This is not an exotic configuration — it is the standard pattern recommended for Gateway API with cert-manager: a single wildcard HTTP listener for ACME challenges (and HTTP-to-HTTPS redirects), alongside per-hostname HTTPS listeners for TLS termination. This bug makes that pattern completely non-functional.
How can we reproduce the issue?
Working configuration — wildcard HTTP listener only:
Gateway:
HTTPRoute (in namespace
default):Result: ✅
curl http://testweb2.lab.inqbeo.orgreturns nginx welcome page.Broken configuration — add HTTPS listeners:
Gateway:
HTTPRoute unchanged — still explicitly targets
sectionName: catchall-http.Result: ❌
curl http://testweb2.lab.inqbeo.orgreturnsHTTP/1.1 404 Not Foundwithserver: envoy. At the same time, cert-manager's HTTP-01 challenge for the HTTPS listener's certificate also fails — the cert-manager logs showwrong status code '404', expected '200'for the/.well-known/acme-challenge/path. Neither the application HTTPRoute nor the solver HTTPRoute created by cert-manager appear in anyCiliumEnvoyConfig:The entire Envoy configuration for the Gateway is simply not generated.
Not a workaround — change HTTP listener to specific hostname:
Changing the HTTP listener hostname from
*.lab.inqbeo.orgtotestweb2.lab.inqbeo.orgimmediately restores routing for that hostname and allows cert-manager to complete its HTTP-01 challenge.However, this is not a viable workaround — it only works for a single hostname at a time. In any real deployment with multiple hostnames, this would require a separate HTTP listener per hostname, a separate ClusterIssuer per hostname, and manually coordinating them all. It merely confirms where the bug lies.
Impact on cert-manager HTTP-01 challenges
The ClusterIssuer is configured to use a single HTTP listener for all HTTP-01 challenges:
When cert-manager needs to validate a certificate, it creates a solver HTTPRoute targeting this HTTP listener with the specific hostname being validated:
This route is accepted (
status.parents[].conditions: Accepted: True) but never programmed into Envoy, causing the ACME propagation check to fail withwrong status code '404', expected '200'in a loop until it times out.The solver pod and service are running and have valid endpoints — the problem is purely that the route is not translated to Envoy configuration. There is no workaround that preserves a single ClusterIssuer for all hostnames.
What did you expect to happen?
Per the Gateway API specification, a wildcard listener hostname (
*.lab.inqbeo.org) should match HTTPRoutes with specific hostnames (testweb2.lab.inqbeo.org). The presence of HTTPS listeners on port 443 should not affect HTTP routing on port 80, especially when the HTTPRoute explicitly targets the HTTP listener viasectionName.Cilium Version
1.18.1
Kernel Version
6.12
Kubernetes Version
1.35.0+rke2r1
Regression
No response
Sysdump
No response
Relevant log output
Anything else?
AI-assisted analysis: The root cause appears to be in the route filtering or Envoy config translation logic. When HTTPS listeners with specific hostnames are added, the entire
CiliumEnvoyConfigdisappears — suggesting the translation pipeline encounters an error or conflict and drops the config entirely, rather than just failing for individual routes.Cilium Users Document
Code of Conduct