CFP: IngressController configurable cloud provider related LoadBalancer

Cilium Feature Proposal

Is your feature request related to a problem?

Cilium adds ingress service without any option to pass additional annotations to it. These annotations are used for various cloud and other controllers. For instance, configuring AWS LoadBalancer or cert-manager:

service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "TCP"
service.beta.kubernetes.io/aws-load-balancer-internal: "true"
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"

Also, there's no way to re-use the same cloud LoadBalancer for multiple ingress services. It'd be great to have just a few types(e.g. public and internal) LoadBalancers for different purposes shared across multiple services. Currently, for each service controller runs a separate service backed by a dedicated LoadBalancer.

Controller should respect ingress object annotations and pass its annotations to the ingress service:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "TCP"
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"

(Optional) Describe your proposed solution

I see a few options:

• have an option to create a static service used for cilium ingress and disable service auto-creation
• pass all annotations from the ingress object to the auto-created ingress service

I also need it.

+1

Sorry for late reply, this seems reasonable to me, one simple approach to to propagate the (all or filtered) annotations from Ingress objects to (at least) Service child resource.

PR is very much welcome, feel free to let us know if you are keen to work on this. Also, please don't hesitate to reach out in #development Cilium slack channel.

@sayboras I would like to work on this. Actually, I am new to the code base, so could you please share some idea or code pointers for adding this feature.

Basically, CIlium Ingress Controller will create below child resources for each Ingress object:

• LB service
• Kubernetes Endpoint
• CiliumEnvoyConfig

This issue is mainly about propagating required annotations from Ingress to LB service. This docs can give you some idea of which annotations are used, from the first look, below annotations prefixes are useful:

• service.beta.kubernetes.io/*
• service.kubernetes.io/*
• cloud.google.com (only used in GCP)

Thanks for your interest, I have assigned the issue to you, feel free to reach out in #development Cilium OSS slack channel.

Related code to start with https://github.com/cilium/cilium/blob/bea793bbc6e8489a308d0bcb6fd796b368d2494c/operator/pkg/ingress/service.go#L186

How to reuse the loadbalancer? From the PR and the helm docs it is not clear for me how to configure it properly @NikhilSharmaWe @aditighag

👋

LB reuse work is actively under development now, it's tracked by https://github.com/cilium/cilium/issues/21270. Please feel free to subscribe to this github issue for update. Thanks.