Skip to content

Cilium wildcard http listener broken by https listeners with more specific hostnames #44123

@fibbs

Description

@fibbs

Is there an existing issue for this?

  • I have searched the existing issues

Version

equal or higher than v1.18.6 and lower than v1.19.0

What happened?

Adding HTTPS listeners with specific hostnames to a Gateway that also has a wildcard HTTP listener causes the wildcard HTTP listener to stop routing traffic entirely. The CiliumEnvoyConfig is not generated at all, and all HTTP requests through the wildcard listener return 404 Not Found from Envoy.

As soon as the HTTP listener is changed from a wildcard (*.lab.inqbeo.org) to a specific hostname (testweb2.lab.inqbeo.org), routing works immediately.

This effectively makes cert-manager HTTP-01 ACME challenges impossible with Cilium Gateway API. The combination of a wildcard HTTP listener with per-hostname HTTPS listeners is the only architecturally viable pattern for HTTP-01 challenges, because:

  1. HTTPS listeners require specific hostnames — each hostname that needs TLS termination must have its own listener with its own certificateRefs.
  2. A ClusterIssuer's HTTP-01 solver must reference a single HTTP listener — the solver's gatewayHTTPRoute.parentRefs points to one listener by sectionName. This listener must be able to serve challenges for any hostname the issuer manages:
    apiVersion: cert-manager.io/v1
    kind: ClusterIssuer
    spec:
      acme:
        solvers:
        - http01:
            gatewayHTTPRoute:
              parentRefs:
              - group: gateway.networking.k8s.io
                kind: Gateway
                name: main-gateway
                namespace: gateway-system
                sectionName: catchall-http   # ← must work for ALL hostnames
  3. A wildcard HTTP listener is therefore required — without it, you would need a separate HTTP listener and a separate ClusterIssuer per hostname, which is unmanageable.

This is not an exotic configuration — it is the standard pattern recommended for Gateway API with cert-manager: a single wildcard HTTP listener for ACME challenges (and HTTP-to-HTTPS redirects), alongside per-hostname HTTPS listeners for TLS termination. This bug makes that pattern completely non-functional.

How can we reproduce the issue?

Working configuration — wildcard HTTP listener only:

Gateway:

listeners:
- name: catchall-http
  port: 80
  protocol: HTTP
  hostname: "*.lab.inqbeo.org"
  allowedRoutes:
    namespaces:
      from: All

HTTPRoute (in namespace default):

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: testweb
  namespace: default
spec:
  parentRefs:
  - name: main-gateway
    namespace: gateway-system
    sectionName: catchall-http
  hostnames:
  - testweb2.lab.inqbeo.org
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: testweb
      port: 80

Result: ✅ curl http://testweb2.lab.inqbeo.org returns nginx welcome page.


Broken configuration — add HTTPS listeners:

Gateway:

listeners:
- name: catchall-http
  port: 80
  protocol: HTTP
  hostname: "*.lab.inqbeo.org"
  allowedRoutes:
    namespaces:
      from: All
- name: testweb-https
  port: 443
  protocol: HTTPS
  hostname: testweb.lab.inqbeo.org
  tls:
    mode: Terminate
    certificateRefs:
    - name: testweb-inqbeo-org-tls
  allowedRoutes:
    namespaces:
      from: All
- name: testweb2-https
  port: 443
  protocol: HTTPS
  hostname: testweb2.lab.inqbeo.org
  tls:
    mode: Terminate
    certificateRefs:
    - name: testweb2-inqbeo-org-tls
  allowedRoutes:
    namespaces:
      from: All

HTTPRoute unchanged — still explicitly targets sectionName: catchall-http.

Result: ❌ curl http://testweb2.lab.inqbeo.org returns HTTP/1.1 404 Not Found with server: envoy. At the same time, cert-manager's HTTP-01 challenge for the HTTPS listener's certificate also fails — the cert-manager logs show wrong status code '404', expected '200' for the /.well-known/acme-challenge/ path. Neither the application HTTPRoute nor the solver HTTPRoute created by cert-manager appear in any CiliumEnvoyConfig:

$ kubectl -n gateway-system get ciliumenvoyconfig -l gateway.networking.k8s.io/owning-gateway=main-gateway -o yaml
apiVersion: v1
items: []
kind: List

The entire Envoy configuration for the Gateway is simply not generated.


Not a workaround — change HTTP listener to specific hostname:

Changing the HTTP listener hostname from *.lab.inqbeo.org to testweb2.lab.inqbeo.org immediately restores routing for that hostname and allows cert-manager to complete its HTTP-01 challenge.

However, this is not a viable workaround — it only works for a single hostname at a time. In any real deployment with multiple hostnames, this would require a separate HTTP listener per hostname, a separate ClusterIssuer per hostname, and manually coordinating them all. It merely confirms where the bug lies.

Impact on cert-manager HTTP-01 challenges

The ClusterIssuer is configured to use a single HTTP listener for all HTTP-01 challenges:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: inqbeo.org-stepca
spec:
  acme:
    server: https://ca.inqbeo.org/acme/acme/directory
    solvers:
    - http01:
        gatewayHTTPRoute:
          parentRefs:
          - group: gateway.networking.k8s.io
            kind: Gateway
            name: main-gateway
            namespace: gateway-system
            sectionName: catchall-http

When cert-manager needs to validate a certificate, it creates a solver HTTPRoute targeting this HTTP listener with the specific hostname being validated:

spec:
  hostnames:
  - testweb2.lab.inqbeo.org
  parentRefs:
  - name: main-gateway
    namespace: gateway-system
    sectionName: catchall-http
  rules:
  - matches:
    - path:
        type: Exact
        value: /.well-known/acme-challenge/<token>
    backendRefs:
    - name: cm-acme-http-solver-ch76n
      namespace: gateway-system
      port: 8089

This route is accepted (status.parents[].conditions: Accepted: True) but never programmed into Envoy, causing the ACME propagation check to fail with wrong status code '404', expected '200' in a loop until it times out.

The solver pod and service are running and have valid endpoints — the problem is purely that the route is not translated to Envoy configuration. There is no workaround that preserves a single ClusterIssuer for all hostnames.

What did you expect to happen?

Per the Gateway API specification, a wildcard listener hostname (*.lab.inqbeo.org) should match HTTPRoutes with specific hostnames (testweb2.lab.inqbeo.org). The presence of HTTPS listeners on port 443 should not affect HTTP routing on port 80, especially when the HTTPRoute explicitly targets the HTTP listener via sectionName.

Cilium Version

1.18.1

Kernel Version

6.12

Kubernetes Version

1.35.0+rke2r1

Regression

No response

Sysdump

No response

Relevant log output

Anything else?

AI-assisted analysis: The root cause appears to be in the route filtering or Envoy config translation logic. When HTTPS listeners with specific hostnames are added, the entire CiliumEnvoyConfig disappears — suggesting the translation pipeline encounters an error or conflict and drops the config entirely, rather than just failing for individual routes.

Cilium Users Document

  • Are you a user of Cilium? Please add yourself to the Users doc

Code of Conduct

  • I agree to follow this project's Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/agentCilium agent related.area/servicemeshGH issues or PRs regarding servicemeshfeature/k8s-gateway-apikind/bugThis is a bug in the Cilium logic.kind/community-reportThis was reported by a user in the Cilium community, eg via Slack.need-more-infoMore information is required to further debug or fix the issue.needs/triageThis issue requires triaging to establish severity and next steps.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions