alibabacloud: ENI leak when IP addresses run out in a subnet #21747
Description
Is there an existing issue for this?
- I have searched the existing issues
What happened?
In a cilium cluster with ipam=alibabacloud, we have seen ENI leakage with the following operator log:
level=info msg="Attached ENI to instance" eniID=eni-xxx instanceID=i-xxx name=nodexxx securityGroupIDs="[sg-xxx]" subsys=ipam toAllocate=2 vSwitchID=vsw-xxx
level=info msg="Synchronized ENI information" numInstances=348 numSecurityGroups=7 numVPCs=7 numVSwitches=30 subsys=eni
level=info msg="Synchronized ENI information" numInstances=348 numSecurityGroups=7 numVPCs=7 numVSwitches=30 subsys=eni
level=info msg="Resolving IP deficit of node" available=2 availableForAllocation=0 availableInterfaces=1 instanceID=i-xxx maxIPsToAllocate=2 name=nodexxx neededIPs=2 remainingInterfaces=1 selectedInterface= selectedPoolID= subsys=ipam used=2
level=info msg="No more IPs available, creating new ENI" instanceID=i-xxx name=nodexxx securityGroupIDs="[sg-xxx]" subsys=ipam toAllocate=2 vSwitchID=vsw-xxx
level=info msg="Created new ENI" eniID=eni-xxx instanceID=i-xxx name=nodexxx securityGroupIDs="[sg-xxx]" subsys=ipam toAllocate=2 vSwitchID=vsw-xxx
level=warning msg="Unable to create interface on instance: unable to attach ENI SDK.ServerError\nErrorCode: EniPerInstanceLimitExceeded\nRecommend: https://next.api.aliyun.com/troubleshoot?q=EniPerInstanceLimitExceeded&product=Ecs\nRequestId: xxx\nMessage: The number of ENI exceeds the limit(2) for the type of instance you are trying to launch." instanceID=i-xxx name=nodexxx subsys=ipam
level=info msg="Created new ENI" eniID=eni-xxx instanceID=i-xxx name=nodexxx securityGroupIDs="[sg-xxx]" subsys=ipam toAllocate=2 vSwitchID=vsw-xxx
level=warning msg="Unable to create interface on instance: unable to attach ENI SDK.ServerError\nErrorCode: EniPerInstanceLimitExceeded\nRecommend: https://next.api.aliyun.com/troubleshoot?q=EniPerInstanceLimitExceeded&product=Ecs\nRequestId: xxx\nMessage: The number of ENI exceeds the limit(2) for the type of instance you are trying to launch." instanceID=i-xxx name=nodexxx subsys=ipam
the following log repeats until all IP addresses in all subnets are consumed, and ENIs created here are leaked in Available status (not attached to any instance):
level=info msg="Created new ENI" eniID=eni-xxx instanceID=i-xxx name=nodexxx securityGroupIDs="[sg-xxx]" subsys=ipam toAllocate=2 vSwitchID=vsw-xxx
level=warning msg="Unable to create interface on instance: unable to attach ENI SDK.ServerError\nErrorCode: EniPerInstanceLimitExceeded\nRecommend: https://next.api.aliyun.com/troubleshoot?q=EniPerInstanceLimitExceeded&product=Ecs\nRequestId: xxx\nMessage: The number of ENI exceeds the limit(2) for the type of instance you are trying to launch." instanceID=i-xxx name=nodexxx subsys=ipam
Cilium Version
v1.10.7
Kernel Version
5.10.56
Kubernetes Version
v1.19.3
Sysdump
No response
Relevant log output
No response
Anything else?
No response
Code of Conduct
- I agree to follow this project's Code of Conduct