ipam: Switch ENI IPAM from CRD to multi-pool allocator#45154
Draft
HadrienPatte wants to merge 3 commits intomainfrom
Draft
ipam: Switch ENI IPAM from CRD to multi-pool allocator#45154HadrienPatte wants to merge 3 commits intomainfrom
HadrienPatte wants to merge 3 commits intomainfrom
Conversation
e392e3d to
d0fe107
Compare
Member
Author
|
/test |
d0fe107 to
f10aaae
Compare
|
/test |
1 similar comment
Member
Author
|
/test |
f10aaae to
9464536
Compare
Member
Author
|
/test |
9464536 to
76ae495
Compare
|
/test |
1 similar comment
Member
Author
|
/test |
In #45110 I added logic to have the operator write `Spec.IPAM.Pools.Allocated` alongside `Spec.IPAM.Pool`, but this is actually not correct as this `Allocated` field is supposed to be written to by the agent. This commit removes the logic writing `Allocated` from the operator and moves and adapt the supporting functions so they can be used by the agent in a following commit. Signed-off-by: Hadrien Patte <hadrien.patte@datadoghq.com>
Replace the CRD allocator with the multi-pool allocator for ENI IPAM mode on the agent side. Previous PR #45124 ensures the operator already supports this new agent setup. The new `eniMultiPoolAllocator` is a light wrapper on the standard `multiPoolAllocator` that enriches `AllocationResult` with ENI-specific required metadata via `buildENIAllocationResult` (see #45089). Key differences from the standard multi-pool allocator: * `AllowFirstLastIPs` is enabled so /28 prefix delegation ranges are fully allocatable (see #45025 and #45082). * `LinearPreAlloc` uses a simple `inUse + preAlloc` formula for demand computation instead of `neededIPCeil` rounding. This matches the CRD allocator's `calculateNeededIPs` semantics and is necessary to ensure the operator can recover the exact IP usage from the demand signal (requested - preAllocate) (see #45124). * No dependency on `CiliumPodIPPool` CRDs, pools are instead populated by the agent from `Status.ENI.ENIs` which is maintained by the operator. The agents now read CIDRs from `Status.ENI.ENIs`, allocate IPs locally, and write aggregate demand to `Spec.IPAM.Pools.Requested`. They no longer writes per-IP usage to `Status.IPAM.Used`, achieving a reduction of kubernetes API pressure. Relates to cilium/design-cfps#87 Signed-off-by: Hadrien Patte <hadrien.patte@datadoghq.com>
Now that ENI IPAM is using the multi-pool allocator, this is deadcode. Signed-off-by: Hadrien Patte <hadrien.patte@datadoghq.com>
76ae495 to
78edb9f
Compare
Member
Author
|
/test |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replace the CRD allocator with the multi-pool allocator for ENI IPAM mode on the agent side. Previous PR #45124 ensures the operator already supports this new agent setup.
The new
eniMultiPoolAllocatoris a light wrapper on the standardmultiPoolAllocatorthat enrichesAllocationResultwith ENI-specific required metadata viabuildENIAllocationResult(see #45089).Key differences from the standard multi-pool allocator:
AllowFirstLastIPsis enabled so /28 prefix delegation ranges are fully allocatable (see ipam: addAllowFirstLastIPsoption toipallocator.NewCIDRRange#45025 and ipam: WireAllowFirstLastIPsoption intocidrPool#45082).LinearPreAllocuses a simpleinUse + preAllocformula for demand computation instead ofneededIPCeilrounding. This matches the CRD allocator'scalculateNeededIPssemantics and is necessary to ensure the operator can recover the exact IP usage from the demand signal (requested - preAllocate) (see ipam: Read demand fromSpec.IPAM.Pools.Requestedfor ENI multi-pool #45124).CiliumPodIPPoolCRDs, pools are instead populated by the agent fromStatus.ENI.ENIswhich is maintained by the operator.The agents now read CIDRs from
Status.ENI.ENIs, allocate IPs locally, and write aggregate demand toSpec.IPAM.Pools.Requested. They no longer writes per-IP usage toStatus.IPAM.Used, achieving a reduction of kubernetes API pressure.Relates to cilium/design-cfps#87
Testing
There's a matrix of scenarios depending on the relative versions of the agents and the operator:
Case A
This case is both agent and operator on 1.19, we didn't touch anything there, nothing to say.
Case B
This case has agents updated to 1.20 before the operator. This case is not expected to be supported. The operator should be updated to 1.20 before the agents. Updating both together with a single helm upgrade is fine as the new operator deployment should be ready before most 1.20 agents become ready themselves.
In this situation, a 1.20 agent will be able to start with the new ENI multi-pool allocator as all it needs from the operator is
Status.ENI.ENIswhich the 1.19 operator already maintains. The only issue is that the agents won't be able to request more IPs as the 1.19 operator doesn't know where to look for the new demand signal.Case C
This case has 1.19 agents with a 1.20 operator, since the operator still maintains and updates
spec.ipam.pool(see #45110) which is the only field 1.19 agents read, they continue to work with the CRD allocator without any issue, expressing their demand viastatus.ipam.used, which the operator still watches (see #45124).Case D
This is the case where both agents and operator are running with the new 1.20 logic: the operator maintains
Status.ENI.ENIs, agents watch that and constructspec.ipam.pools.allocatedfrom it, using the new ENI multipool allocator, they express their needs throughspec.ipam.pools.requestedand zero outstatus.ipam.usedto clearly indicate to the operator that they're using multipool.Here's what a ciliumnode looks like in this case:
kubectl get ciliumnode $node -ojson | jq '.spec.ipam'
{ "min-allocate": 3, "pool": { "100.120.116.112": { "resource": "eni-024ee4a1e521cda8c" }, "100.120.116.113": { "resource": "eni-024ee4a1e521cda8c" }, "100.120.116.114": { "resource": "eni-024ee4a1e521cda8c" }, "100.120.116.115": { "resource": "eni-024ee4a1e521cda8c" }, "100.120.116.116": { "resource": "eni-024ee4a1e521cda8c" }, "100.120.116.117": { "resource": "eni-024ee4a1e521cda8c" }, "100.120.116.118": { "resource": "eni-024ee4a1e521cda8c" }, "100.120.116.119": { "resource": "eni-024ee4a1e521cda8c" }, "100.120.116.120": { "resource": "eni-024ee4a1e521cda8c" }, "100.120.116.121": { "resource": "eni-024ee4a1e521cda8c" }, "100.120.116.122": { "resource": "eni-024ee4a1e521cda8c" }, "100.120.116.123": { "resource": "eni-024ee4a1e521cda8c" }, "100.120.116.124": { "resource": "eni-024ee4a1e521cda8c" }, "100.120.116.125": { "resource": "eni-024ee4a1e521cda8c" }, "100.120.116.126": { "resource": "eni-024ee4a1e521cda8c" }, "100.120.116.127": { "resource": "eni-024ee4a1e521cda8c" }, "100.120.26.176": { "resource": "eni-024ee4a1e521cda8c" }, "100.120.26.177": { "resource": "eni-024ee4a1e521cda8c" }, "100.120.26.178": { "resource": "eni-024ee4a1e521cda8c" }, "100.120.26.179": { "resource": "eni-024ee4a1e521cda8c" }, "100.120.26.180": { "resource": "eni-024ee4a1e521cda8c" }, "100.120.26.181": { "resource": "eni-024ee4a1e521cda8c" }, "100.120.26.182": { "resource": "eni-024ee4a1e521cda8c" }, "100.120.26.183": { "resource": "eni-024ee4a1e521cda8c" }, "100.120.26.184": { "resource": "eni-024ee4a1e521cda8c" }, "100.120.26.185": { "resource": "eni-024ee4a1e521cda8c" }, "100.120.26.186": { "resource": "eni-024ee4a1e521cda8c" }, "100.120.26.187": { "resource": "eni-024ee4a1e521cda8c" }, "100.120.26.188": { "resource": "eni-024ee4a1e521cda8c" }, "100.120.26.189": { "resource": "eni-024ee4a1e521cda8c" }, "100.120.26.190": { "resource": "eni-024ee4a1e521cda8c" }, "100.120.26.191": { "resource": "eni-024ee4a1e521cda8c" } }, "pools": { "allocated": [ { "cidrs": [ "100.120.116.112/28", "100.120.26.176/28" ], "pool": "default" } ], "requested": [ { "needed": { "ipv4-addrs": 20 }, "pool": "default" } ] }, "pre-allocate": 1 }kubectl get ciliumnode $node -ojson | jq '.status.eni.enis["eni-024ee4a1e521cda8c"]'
{ "addresses": [ "100.120.26.176", "100.120.26.177", "100.120.26.178", "100.120.26.179", "100.120.26.180", "100.120.26.181", "100.120.26.182", "100.120.26.183", "100.120.26.184", "100.120.26.185", "100.120.26.186", "100.120.26.187", "100.120.26.188", "100.120.26.189", "100.120.26.190", "100.120.26.191", "100.120.116.112", "100.120.116.113", "100.120.116.114", "100.120.116.115", "100.120.116.116", "100.120.116.117", "100.120.116.118", "100.120.116.119", "100.120.116.120", "100.120.116.121", "100.120.116.122", "100.120.116.123", "100.120.116.124", "100.120.116.125", "100.120.116.126", "100.120.116.127" ], "description": "Cilium-CNI (i-XXXX)", "id": "eni-024ee4a1e521cda8c", "ip": "100.120.117.66", "mac": "0e:d5:7d:00:64:d5", "number": 1, "prefixes": [ "100.120.26.176/28", "100.120.116.112/28" ], "security-groups": [ "sg-XXXX", "sg-YYYY" ], "subnet": { "cidr": "100.120.0.0/17", "id": "subnet-XXXX" }, "tags": { "io.cilium/cilium-managed": "true", "io.cilium/cluster-name": "redacted" }, "vpc": { "cidrs": [ "100.120.0.0/16" ], "id": "vpc-XXXX", "primary-cidr": "10.0.0.0/19" } }kubectl get ciliumnode $node -ojson | jq '.status.ipam'
{ "operator-status": {} }