Description

This PR is a proposal for supporting static IP allocation as described in #34094.
The proposed abstraction is Cloud Provider agnostic. This PR implements the code path for AWS which is fully functional. The code for other Cloud Providers can be implemented later separately.

In this version of the implementation, we assume that users manually manage pools of static IP addresses (in this particular example, pools of AWS Elastic IP addresses) and assign different tags to differentiate the pools.
The main idea of this proposal is to add a new field (static-ip-tags) to the ipam section of the CNI spec. This will allow users to indicate that they want a static IP with given tags to be assigned to the node.
Example:

"ipam": {
  "static-ip-tags": {
    "pool": "pool-a",
    "kubernetes_cluster": "cluster-1"
  }
}

When the Operator sees that static-ip-tags is set on the CiliumNode object, it makes the Cloud Provider API calls to retrieve the IPs with the given tags and associates the first found IP with the instance. It then updates the status of the CiliumNode object.

Testing

I added some unit tests but also tried out the PR in a real AWS environment.
At first I manually allocated an EIP with tags "anton": "test" and "kubernetes_cluster": "<MY_CLUSTER_NAME>" in our AWS account.
I then provisioned a node with the following CNI spec:

$ sudo cat /etc/cni/net.d/10-generic-veth.conflist
{
  "cniVersion": "0.3.1",
  "name": "cilium",
  "plugins": [
    {
      "name": "cilium",
      "type": "cilium-cni",
      "eni": {
        "delete-on-termination": true,
        "first-interface-index": 1,
        "use-primary-address": false,
        "pre-allocate": 1,
        "min-allocate": 3
      },
      "ipam": {
        "pre-allocate": 1,
        "min-allocate": 3,
        "static-ip-tags": {
          "anton": "test",
          "kubernetes_cluster": ""
        }
      }
    }
  ]
} 

I then confirmed that the IPAM settings were successfully passed on to the CiliumNode CRD object:

$ kubectl get ciliumnode $MY_NODE_NAME -o=jsonpath='{.spec.ipam}' | jq
{
  "min-allocate": 3,
  "pool": {
    "100.xxx.xxx.xxx": {
      "resource": "eni-xxx"
    },
    "100.xxx.xxx.xxx": {
      "resource": "eni-xxx"
    }
  },
  "pools": {},
  "pre-allocate": 1,
  "static-ip-tags": {
    "anton": "test",
    "kubernetes_cluster": "<MY_CLUSTER_NAME>"
  }
}

The Operator saw that IPAM field and associated the EIP I created. Logs:

Found 1 EIPs corresponding to tags map[anton:test kubernetes_cluster:<MY_CLUSTER_NAME>]

Associated EIP 18.213.xxx.xxx with Instance i-xxxxf7517 (association ID: eipassoc-xxx)

It also updated the CRD resource status with the right IP address:

$ k get ciliumnode $MY_NODE_NAME -o=jsonpath='{.status.ipam}' | jq
{
  "assigned-static-ip": "18.213.xxx.xxx",
  "operator-status": {},
  "used": [...]
}

The EIP is successfully associated with the EC2 instance as well: