Azure IPAM Support by tgraf · Pull Request #10089 · cilium/cilium

tgraf · 2020-02-07T14:18:49Z

This PR implements IPAM using the Azure API similarly to how AWS ENI is currently supported:

The PR is split into two parts:

Part 1 - Refactoring of existing ENI code to prepare Azure IPAM:

      eni: Move Limits type to pkg/ipam
      ipam: Factor out IPAM related metrics
      eni: Unexport GetLimits
      k8s: CiliumNode: Move generic IPAM parmeters into Spec.IPAM
      api: Factor out metrics accounting for external API usage
      operator: Rename --eni-parallel-workers to --parallel-alloc-workers
      operator: Rename --aws-client-burst and --aws-client-qps options
      api: Factor out API helpers into pkg/api/helpers
      ipam: Move generic types from pkg/aws/eni to pkg/ipam/types
      ec2/mock: Support returning different ENIs, subnets and security groups
      eni: Refactor ENI IPAM into generic ipam.NodeManager
      operator: Move CiliumNode update logic out of ENI
      operator: Decopule IPAM and node interactions from ENI

Part 2 - Implementation of Azure IPAM

      ipam: Support for Azure IPAM
      cni: Add Azure IPAM support
      doc: Azure installation instructions & concepts

The function is similar to how AWS ENI mode operates with a couple of key differences:

A single interface provides enough IPs to host all local pods so support to create additional interfaces is not required.
The Azure API does not support allocating IP itself. It can only assign IPs to interfaces. The cilium-operator thus maintains a small allocator to keep track of available IPs in all subnets.

This change is

maintainer-s-little-helper · 2020-02-07T14:18:54Z

Commit 13d7eb6 does not contain "Signed-off-by".

Please follow instructions provided in https://docs.cilium.io/en/stable/contributing/contributing/#developer-s-certificate-of-origin

pkg/k8s/apis/cilium.io/v2/types.go

pkg/ipam/node_manager.go

pkg/aws/eni/node.go

pkg/azure/ipam/metrics/mock/mock.go

pkg/azure/ipam/metrics/metrics.go

pkg/azure/ipam/instances.go

pkg/azure/api/mock/mock.go

pkg/azure/api/api_test.go

pkg/azure/api/api.go

pkg/aws/eni/instances.go

pkg/aws/ec2/mock/mock.go

pkg/api/helpers/rate_limit.go

pkg/api/helpers/metrics/mock/mock.go

pkg/api/helpers/metrics/metrics.go

coveralls · 2020-02-07T14:47:28Z

Coverage increased (+0.1%) to 45.681% when pulling fa8aa88 on pr/tgraf/azure-ipam into 21fa852 on master.

tgraf · 2020-02-21T14:19:13Z

test-me-please

pkg/api/metrics/metrics.go

pkg/api/metrics/mock/mock.go

pkg/api/metrics/metrics.go

pkg/aws/ec2/mock/mock.go

pkg/api/metrics/mock/mock.go

pkg/api/metrics/metrics.go

pkg/aws/ec2/mock/mock.go

pkg/ipam/metrics/metrics.go

The concept of node specific IPAM and the relevant management required to do so has been part of pkg/eni and tightly coupled with it. This commit decouples the node management and base IPAM logic from pkg/eni and moves it into pkg/ipam to prepare support for additional IPAM backends. A couple of changes are required to do so: - Slight modifications to the calculations of needed addresses due to a split in code between pkg/ipam and pkg/eni - New tests in pkg/ipam/node_manager_test.go which does not assume interface creation is required prior to allocation and with removed address per interface limits. - The instances manager mock can be removed as the eni.InstancesManager now serves as the abstraction to the NodeManager and the NodeManager no longer requires an EC2 mock as all of those details are now isolated in pkg/eni. The existing tests in pkg/aws/eni/node_manager_test.go are mostly unchanged to verify that the refactoring does not change behavior of the existing ENI allocator. Signed-off-by: Thomas Graf <thomas@cilium.io>

This is a pure refactoring to move the CiliumNode update logic implementation out of being ENI specific. Signed-off-by: Thomas Graf <thomas@cilium.io>

Signed-off-by: Thomas Graf <thomas@cilium.io>

tgraf · 2020-02-28T15:29:48Z

test-docs-please

pkg/azure/api/mock/mock.go

tgraf · 2020-02-28T16:37:44Z

test-me-please

Following the refactoring of the AWS ENI IPAM logic to be generic. This commit adds support for Azure IPAM. The function is similar to how AWS ENI mode operates with a couple of key differences: - A single interface provides enough IPs to host all local pods so support to create additional interfaces is not required. - The Azure API does not support allocating IP itself. It can only assign IPs to interfaces. The cilium-operator thus maintains a small allocator to keep track of available IPs in all subnets. In order to access the Azure API, a service principal is required. The credentials for the service principal is passed to the operator as environment variables or command line arguments. Signed-off-by: Thomas Graf <thomas@cilium.io>

Signed-off-by: Thomas Graf <thomas@cilium.io>

Add helm chart option to enable Azure support along with Documentation on how to use it. Add a concepts section to explain how Azure mode functions. Signed-off-by: Thomas Graf <thomas@cilium.io>

tgraf · 2020-02-28T18:06:53Z

test-me-please

tgraf · 2020-02-28T20:44:05Z

Also, @tgraf I don't know why but I have a diff in the generated code if I run make generate-k8s-api

Ran make generate-k8s-api again after moving Azure bits into pkg/azure/types Diff looks good now.

tgraf · 2020-02-29T00:15:55Z

test-docs-please

pkg/ipam/metrics/metrics.go

As suggested in #10089 (comment) Signed-off-by: Tobias Klauser <tklauser@distanz.ch>

Remove deprecated fields that have been unused for 5+ years: * Azure `AzureInterface.GatewayIP` (obsolete since v1.10, see #15182) * AWS `ENISpec.InstanceID` (obsolete since v1.8, see #10569) * AWS `ENISpec.{MinAllocate,PreAllocate,MaxAboveWatermark}` (obsolete since v1.8, see #10089) Signed-off-by: Hadrien Patte <hadrien.patte@datadoghq.com>

tgraf added wip release-note/major This PR introduces major new functionality to Cilium. labels Feb 7, 2020

tgraf requested review from a team as code owners February 7, 2020 14:18

tgraf requested a review from a team February 7, 2020 14:18

tgraf requested a review from a team as a code owner February 7, 2020 14:18

maintainer-s-little-helper bot added the dont-merge/needs-sign-off The author needs to add signoff to their commits before merge. label Feb 7, 2020

houndci-bot reviewed Feb 7, 2020

View reviewed changes

pkg/api/helpers/metrics/metrics.go Outdated Show resolved Hide resolved

pkg/api/helpers/metrics/metrics.go Show resolved Hide resolved

tgraf force-pushed the pr/tgraf/azure-ipam branch from 4fe88fa to 47a4807 Compare February 9, 2020 09:51

maintainer-s-little-helper bot removed the dont-merge/needs-sign-off The author needs to add signoff to their commits before merge. label Feb 9, 2020

tgraf mentioned this pull request Feb 11, 2020

Refactor ENI IPAM to support other IPAMs #9890

Closed

aanm added the dont-merge/wait-until-release Freeze window for current release is blocking non-bugfix PRs label Feb 14, 2020

aanm added this to the 1.8 milestone Feb 18, 2020

aanm removed the dont-merge/wait-until-release Freeze window for current release is blocking non-bugfix PRs label Feb 18, 2020

tgraf force-pushed the pr/tgraf/azure-ipam branch from 47a4807 to e3d039c Compare February 20, 2020 15:49

tgraf requested a review from a team February 20, 2020 15:49

tgraf force-pushed the pr/tgraf/azure-ipam branch from e3d039c to ba48b37 Compare February 21, 2020 13:26

tgraf force-pushed the pr/tgraf/azure-ipam branch from ba48b37 to e589796 Compare February 27, 2020 23:44

houndci-bot reviewed Feb 27, 2020

View reviewed changes

pkg/api/metrics/metrics.go Outdated Show resolved Hide resolved

houndci-bot reviewed Feb 27, 2020

View reviewed changes

pkg/api/metrics/mock/mock.go Outdated Show resolved Hide resolved

pkg/api/metrics/metrics.go Outdated Show resolved Hide resolved

houndci-bot reviewed Feb 27, 2020

View reviewed changes

pkg/aws/ec2/mock/mock.go Outdated Show resolved Hide resolved

pkg/aws/ec2/mock/mock.go Outdated Show resolved Hide resolved

pkg/aws/ec2/mock/mock.go Outdated Show resolved Hide resolved

houndci-bot reviewed Feb 27, 2020

View reviewed changes

tgraf added 3 commits February 28, 2020 16:16

operator: Move CiliumNode update logic out of ENI

e08d00c

This is a pure refactoring to move the CiliumNode update logic implementation out of being ENI specific. Signed-off-by: Thomas Graf <thomas@cilium.io>

operator: Decopule IPAM and node interactions from ENI

66a9aea

Signed-off-by: Thomas Graf <thomas@cilium.io>

tgraf force-pushed the pr/tgraf/azure-ipam branch from 956b08e to 833b349 Compare February 28, 2020 16:36

houndci-bot reviewed Feb 28, 2020

View reviewed changes

pkg/azure/api/mock/mock.go Outdated Show resolved Hide resolved

pkg/azure/api/mock/mock.go Outdated Show resolved Hide resolved

pkg/azure/api/mock/mock.go Outdated Show resolved Hide resolved

houndci-bot reviewed Feb 28, 2020

View reviewed changes

pkg/azure/api/mock/mock.go Outdated Show resolved Hide resolved

pkg/azure/api/mock/mock.go Outdated Show resolved Hide resolved

pkg/azure/api/mock/mock.go Outdated Show resolved Hide resolved

tgraf added 3 commits February 28, 2020 17:49

cni: Add Azure IPAM support

72b7a50

Signed-off-by: Thomas Graf <thomas@cilium.io>

doc: Azure installation instructions & concepts

fa8aa88

Add helm chart option to enable Azure support along with Documentation on how to use it. Add a concepts section to explain how Azure mode functions. Signed-off-by: Thomas Graf <thomas@cilium.io>

tgraf force-pushed the pr/tgraf/azure-ipam branch from 833b349 to fa8aa88 Compare February 28, 2020 17:08

tgraf merged commit 6e5be2f into master Mar 2, 2020

tgraf deleted the pr/tgraf/azure-ipam branch March 2, 2020 09:34

tklauser reviewed Mar 2, 2020

View reviewed changes

pkg/ipam/metrics/metrics.go Show resolved Hide resolved

tklauser mentioned this pull request Mar 2, 2020

ipam/metrics: mention interfaces instead of ENI #10406

Merged

tklauser added a commit that referenced this pull request Mar 2, 2020

ipam/metrics: mention interfaces instead of ENI

f15532d

As suggested in #10089 (comment) Signed-off-by: Tobias Klauser <tklauser@distanz.ch>

aanm pushed a commit that referenced this pull request Mar 3, 2020

ipam/metrics: mention interfaces instead of ENI

94e6ccd

As suggested in #10089 (comment) Signed-off-by: Tobias Klauser <tklauser@distanz.ch>

joestringer mentioned this pull request Mar 9, 2020

Azure IPAM #10411

Closed

4 tasks

christarazi mentioned this pull request May 29, 2020

Refactor Azure IPAM and ENI IPAM to deduplicate similar logic #11787

Open

tklauser mentioned this pull request Jun 3, 2020

v1.7 backports 2020-06-03 #11855

Merged

HadrienPatte mentioned this pull request Jan 8, 2025

Update azure go sdk #36751

Merged

8 tasks

HadrienPatte mentioned this pull request Apr 2, 2026

ipam: Remove obsolete fields from cloud provider IPAM types #45138

Open

Conversation

tgraf commented Feb 7, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Part 1 - Refactoring of existing ENI code to prepare Azure IPAM:

Part 2 - Implementation of Azure IPAM

Uh oh!

maintainer-s-little-helper bot commented Feb 7, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

coveralls commented Feb 7, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tgraf commented Feb 21, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tgraf commented Feb 28, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tgraf commented Feb 28, 2020

Uh oh!

tgraf commented Feb 28, 2020

Uh oh!

tgraf commented Feb 28, 2020

Uh oh!

tgraf commented Feb 7, 2020 •

edited

Loading

coveralls commented Feb 7, 2020 •

edited

Loading