Skip to content

Create new ENIs in initial subnet by default#22000

Merged
tklauser merged 1 commit intocilium:masterfrom
bimmlerd:pr/bimmlerd/create-eni-in-known-subnet-by-default
Nov 18, 2022
Merged

Create new ENIs in initial subnet by default#22000
tklauser merged 1 commit intocilium:masterfrom
bimmlerd:pr/bimmlerd/create-eni-in-known-subnet-by-default

Conversation

@bimmlerd
Copy link
Copy Markdown
Member

@bimmlerd bimmlerd commented Nov 4, 2022

When no Subnet IDs/tags are configured, Cilium used to allocate new ENIs in the subnet which the most addresses available that is in the same VPC and AZ. This can lead to suprises, see #20553. Instead, try to allocate the new ENI in the same subnet as the primary one.

In order to do so, save the initial subnet ID in the Spec to have access to it while making the decision where to allocate the new ENI.

Fixes: #20553

In ENI IPAM mode, try to allocate new ENIs in the same subnet as the primary ENI instead of the subnet with the most available addresses.

@bimmlerd bimmlerd added kind/enhancement This would improve or streamline existing functionality. release-note/minor This PR changes functionality that users may find relevant to operating Cilium. area/eni Impacts ENI based IPAM. sig/ipam labels Nov 4, 2022
@bimmlerd bimmlerd force-pushed the pr/bimmlerd/create-eni-in-known-subnet-by-default branch from b15e0b3 to f85fc60 Compare November 4, 2022 11:10
@bimmlerd
Copy link
Copy Markdown
Member Author

bimmlerd commented Nov 4, 2022

/test

@bimmlerd bimmlerd force-pushed the pr/bimmlerd/create-eni-in-known-subnet-by-default branch from f85fc60 to c5a0416 Compare November 8, 2022 08:52
@bimmlerd bimmlerd marked this pull request as ready for review November 8, 2022 09:38
@bimmlerd bimmlerd requested review from a team as code owners November 8, 2022 09:38
@bimmlerd bimmlerd requested a review from gandro November 8, 2022 09:38
Copy link
Copy Markdown
Member

@gandro gandro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome work 🚀

Two very minor nits, otherwise this looks excellent to me.

@pchaigno pchaigno removed the request for review from a team November 8, 2022 10:48
@bimmlerd bimmlerd force-pushed the pr/bimmlerd/create-eni-in-known-subnet-by-default branch from c5a0416 to 757c6ac Compare November 8, 2022 15:08
@bimmlerd bimmlerd requested review from a team as code owners November 8, 2022 15:08
@bimmlerd bimmlerd force-pushed the pr/bimmlerd/create-eni-in-known-subnet-by-default branch from 757c6ac to 7cdaed2 Compare November 8, 2022 15:15
@bimmlerd
Copy link
Copy Markdown
Member Author

bimmlerd commented Nov 8, 2022

/test

Job 'Cilium-PR-K8s-1.16-kernel-4.9' failed:

Click to show.

Test Name

K8sDatapathVerifier Runs the kernel verifier against Cilium's BPF datapath

Failure Output

FAIL: Expected clean target to succeed

If it is a flake and a GitHub issue doesn't already exist to track it, comment /mlh new-flake Cilium-PR-K8s-1.16-kernel-4.9 so I can create one.

@bimmlerd bimmlerd force-pushed the pr/bimmlerd/create-eni-in-known-subnet-by-default branch from 7cdaed2 to 2250167 Compare November 14, 2022 10:47
@bimmlerd
Copy link
Copy Markdown
Member Author

bimmlerd commented Nov 14, 2022

I realised that https://docs.cilium.io/en/v1.12/concepts/networking/ipam/eni/#eni-creation exists, and had to update that too- It was mildly out of date already, but should be correct now. Worthy of a quick glance though, maybe the wording could be improved.

Manually requesting reviews since GH didn't.

Copy link
Copy Markdown
Member

@christarazi christarazi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work!

Copy link
Copy Markdown
Member

@joestringer joestringer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

docs LGTM, thanks!

@joestringer
Copy link
Copy Markdown
Member

/ci-eks

@bimmlerd
Copy link
Copy Markdown
Member Author

/test

@maintainer-s-little-helper maintainer-s-little-helper bot added the ready-to-merge This PR has passed all tests and received consensus from code owners to merge. label Nov 18, 2022
In the ENI mode of IPAM, when all IPs of an ENI are used, a new ENI is
created to provide more IPs. Cilium chooses the subnet for this ENI
based on SubnetIDs and SubnetTags, but if none are configured it needs
to choose a subnet nonetheless.

In this fallback case, the heuristic used was to choose the subnet in
the same VPC and Availability Zone with the most available addresses.
This led to surprising results, however, since the first ENI and future
ENIs were not necessariliy in the same subnet.

Instead, as part of node discovery, remember the subnet of the primary
ENI as part of the Spec and try to create future ENIs in it as well, as
long as there are sufficient addresses available.

Note that explicitly configured subnet IDs and/or tags still take
precedence, this changes the unconfigured default to be a bit more
reasonable.

Fixes: cilium#20553

Signed-off-by: David Bimmler <david.bimmler@isovalent.com>
@bimmlerd bimmlerd force-pushed the pr/bimmlerd/create-eni-in-known-subnet-by-default branch from 2250167 to 43c62dc Compare November 18, 2022 11:47
@bimmlerd bimmlerd removed the ready-to-merge This PR has passed all tests and received consensus from code owners to merge. label Nov 18, 2022
@bimmlerd
Copy link
Copy Markdown
Member Author

Rebased on master due to conflcit in docs (which exposed that I had accidentially put the upgrade notes into the section for 1.12 instead of 1.13. Thanks Joe!)

ConformanceKind1.19 and "Smoke test / conformance-test (pull_request)" flake on #22217, the rest of CI was green before the rebase. Setting ready to merge.

@bimmlerd bimmlerd added the ready-to-merge This PR has passed all tests and received consensus from code owners to merge. label Nov 18, 2022
@tklauser tklauser merged commit c5cbf40 into cilium:master Nov 18, 2022
@alloveras
Copy link
Copy Markdown

Is there any chance this PR can be backported in the next v1.12 release?

@christarazi
Copy link
Copy Markdown
Member

@alloveras As I understand, this would be a breaking change or at least a change in default behavior. We typically don't do that in a stable release.

@alloveras
Copy link
Copy Markdown

Yeah, I understand 👍 I saw this one more as a bugfix than anything else but I guess you could argue it both ways 😄

@gandro
Copy link
Copy Markdown
Member

gandro commented Jan 12, 2023

Yeah, I understand +1 I saw this one more as a bugfix than anything else but I guess you could argue it both ways smile

As a note: You can achieve a very similar behavior on v1.12 by setting a subnet tag filter via a custom CNI config:

https://docs.cilium.io/en/v1.12/concepts/networking/ipam/eni/#eni-creation
https://docs.cilium.io/en/v1.12/concepts/networking/ipam/eni/#create-a-cni-configuration

@alloveras
Copy link
Copy Markdown

As a note: You can achieve a very similar behavior on v1.12 by setting a subnet tag filter via a custom CNI config:

Yeah, absolutely 👍 That's exactly the workaround that we are using at the moment 😄 Thanks for the suggestion, though 🙇

@gandro
Copy link
Copy Markdown
Member

gandro commented Jan 12, 2023

Another thing to note: The subnet tag filters might still be needed even with this PR applied - the PR here will use prefer the initial subnet, but not force it. So once the initial subnet is exhausted, Cilium will allocate from any other available subnets, unless a tag filter is set. That may or may not be what you want.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/eni Impacts ENI based IPAM. kind/enhancement This would improve or streamline existing functionality. ready-to-merge This PR has passed all tests and received consensus from code owners to merge. release-note/minor This PR changes functionality that users may find relevant to operating Cilium.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ENI mode attaches secondary ENIs in dubious subnets by default

6 participants