Skip to content

Ingress controller can only listen on a single host port and not on BOTH 80 and 443 #39422

@kevinvalk

Description

@kevinvalk

(First started with feature proposal issue and then with bug report, but both templates felt wrong as this feels exactly between bug and feature request.)

I am facing the exact problem as #32768 which became stale and was automatically closed.

My intend is to replace NGINX Ingress DaemonSet deployment that listens on host ports 80 and 443 with Cilium. This seemed supported with the recently (1.16+) introduced hostNetwork flag for both Ingress and Gateway. However, as I can only give a single sharedListenerPort, this will not fly as far as I can see.

In the GatewayAPI case, you configure the listen ports in the Gateway resources and it will bind on the host ports were required. Sadly, I cannot use Gateway due to difficulty in managing many certificates for different domains https://gateway-api.sigs.k8s.io/geps/gep-1713/#yaml

In the ingress controller case the docs states https://docs.cilium.io/en/stable/network/servicemesh/ingress/

Shared Ingress: Globally via Helm flags

ingressController.hostNetwork.sharedListenerPort: Host network port to expose the Cilium ingress controller
Envoy listener. The default port is 8080. If you change it, you should choose a port number higher than 1023 (see Bind to privileged port).

Which makes sense if you look at the code:

func (r *ingressReconciler) getSharedListenerPorts() (uint32, uint32, uint32) {
if !r.hostNetworkEnabled {
return defaultPassthroughPort, defaultInsecureHTTPPort, defaultSecureHTTPPort
}
if r.hostNetworkSharedPort > 0 {
return r.hostNetworkSharedPort, r.hostNetworkSharedPort, r.hostNetworkSharedPort
}
return defaultHostNetworkListenerPort, defaultHostNetworkListenerPort, defaultHostNetworkListenerPort
}

However, I would expect to be able to provide HTTP AND HTTPS ports (and maybe even separate passthrough port?).

I did not fully analyze the remaining implementation, so I am not sure if it is as simple as allowing for three different helm values and plugging those in getSharedListenerPorts or if more logic has to be changed in other layers.


The relevant Cilium configuration:

  ingressController:
    enabled: true
    loadbalancerMode: shared # As we are using host port we can only have a single ingress controller
    hostNetwork:
        enabled: true
        sharedListenerPort: 80 # <---- how can we also listen on 443 to serve both HTTP and HTTPS traffic

  # We want to be able to bind our Ingress and/or Gateway to host port 80 and 443 (privileged), so we have to make some
  # changes.
  envoy:
    enabled: true
    securityContext:
      capabilities:
        keepCapNetBindService: true # Will pass the NET_BIND_SERVICE to the actual forked process
        envoy:
          # NET_ADMIN, SYS_ADMIN are defaults and taken from `envoy.securityContext.capabilities.envoy` Helm values
          - NET_ADMIN
          - SYS_ADMIN
          - NET_BIND_SERVICE # To run our Ingress/Gateway directly on 80, 443 we need this

Metadata

Metadata

Labels

area/agentCilium agent related.area/servicemeshGH issues or PRs regarding servicemeshfeature/k8s-gateway-apikind/bugThis is a bug in the Cilium logic.kind/community-reportThis was reported by a user in the Cilium community, eg via Slack.needs/triageThis issue requires triaging to establish severity and next steps.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions