Skip to content

addons.adoc

Latest commit

 

History

History
358 lines (275 loc) · 11 KB

addons.adoc

File metadata and controls

358 lines (275 loc) · 11 KB

Addons

This topic describes how to manage Amazon EKS Add-Ons for your Amazon EKS clusters using eksctl. EKS Add-Ons is a feature that lets you enable and manage Kubernetes operational software through the EKS API, simplifying the process of installing, configuring, and updating cluster add-ons.

Warning

eksctl now installs default addons (vpc-cni, coredns, kube-proxy) as EKS addons instead of self-managed addons. This means you should use eksctl update addon instead of eksctl utils update-* commands for clusters created with eksctl v0.184.0 and above.

You can create clusters without any default networking addons when you want to use alternative CNI plugins like Cilium and Calico.

EKS Add-ons now support receiving IAM permissions via EKS Pod Identity Associations, allowing them to connect with AWS services outside of the cluster

Creating addons

Eksctl provides more flexibility for managing cluster addons:

In your config file, you can specify the addons you want and (if required) the role or policies to attach to them:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: example-cluster
  region: us-west-2

iam:
  withOIDC: true

addons:
- name: vpc-cni
  # all below properties are optional
  version: 1.7.5
  tags:
    team: eks
  # you can specify at most one of:
  attachPolicyARNs:
  - arn:aws:iam::account:policy/AmazonEKS_CNI_Policy
  # or
  serviceAccountRoleARN: arn:aws:iam::account:role/AmazonEKSCNIAccess
  # or
  attachPolicy:
    Statement:
    - Effect: Allow
      Action:
      - ec2:AssignPrivateIpAddresses
      - ec2:AttachNetworkInterface
      - ec2:CreateNetworkInterface
      - ec2:DeleteNetworkInterface
      - ec2:DescribeInstances
      - ec2:DescribeTags
      - ec2:DescribeNetworkInterfaces
      - ec2:DescribeInstanceTypes
      - ec2:DetachNetworkInterface
      - ec2:ModifyNetworkInterfaceAttribute
      - ec2:UnassignPrivateIpAddresses
      Resource: '*'

You can specify at most one of attachPolicy, attachPolicyARNs and serviceAccountRoleARN.

If none of these are specified, the addon will be created with a role that has all recommended policies attached.

Note

In order to attach policies to addons your cluster must have OIDC enabled. If it’s not enabled we ignore any policies attached.

You can then either have these addons created during the cluster creation process:

eksctl create cluster -f config.yaml

Or create the addons explicitly after cluster creation using the config file or CLI flags:

eksctl create addon -f config.yaml
eksctl create addon --name vpc-cni --version 1.7.5 --service-account-role-arn <role-arn>
eksctl create addon --name aws-ebs-csi-driver --namespace-config 'namespace=custom-namespace'
Tip

Use the --namespace-config flag to deploy addons to a custom namespace instead of the default namespace.

During addon creation, if a self-managed version of the addon already exists on the cluster, you can choose how potential configMap conflicts shall be resolved by setting resolveConflicts option via the config file, e.g.

addons:
- name: vpc-cni
  attachPolicyARNs:
    - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
  resolveConflicts: overwrite

For addon create, the resolveConflicts field supports three distinct values:

  • none - EKS doesn’t change the value. The create might fail.

  • overwrite - EKS overwrites any config changes back to EKS default values.

  • preserve - EKS doesn’t change the value. The create might fail. (Similarly to none, but different from preserve in updating addons).

Listing enabled addons

You can see what addons are enabled in your cluster by running:

eksctl get addons --cluster <cluster-name>

or

eksctl get addons -f config.yaml

Setting the addon’s version

Setting the version of the addon is optional. If the version field is left empty eksctl will resolve the default version for the addon. More information about which version is the default version for specific addons can be found in the AWS documentation about EKS. Note that the default version might not necessarily be the latest version available.

The addon version can be set to latest. Alternatively, the version can be set with the EKS build tag specified, such as v1.7.5-eksbuild.1 or v1.7.5-eksbuild.2. It can also be set to the release version of the addon, such as v1.7.5 or 1.7.5, and the eksbuild suffix tag will be discovered and set for you.

See the section below on how to discover available addons and their versions.

Discovering addons

You can discover what addons are available to install on your cluster by running:

eksctl utils describe-addon-versions --cluster <cluster-name>

This will discover your cluster’s kubernetes version and filter on that. Alternatively if you want to see what addons are available for a particular kubernetes version you can run:

eksctl utils describe-addon-versions --kubernetes-version <version>

You can also discover addons by filtering on their type, owner and/or publisher. For e.g., to see addons for a particular owner and type you can run:

eksctl utils describe-addon-versions --kubernetes-version 1.22 --types "infra-management, policy-management" --owners "aws-marketplace"

The types, owners and publishers flags are optional and can be specified together or individually to filter the results.

Discovering the configuration schema for addons

After discovering the addon and version, you can view the customization options by fetching its JSON configuration schema.

eksctl utils describe-addon-configuration --name vpc-cni --version v1.12.0-eksbuild.1

This returns a JSON schema of the various options available for this addon.

Working with configuration values

ConfigurationValues can be provided in the configuration file during the creation or update of addons. Only JSON and YAML formats are supported.

For eg.,

addons:
- name: coredns
  configurationValues: |-
    replicaCount: 2
addons:
- name: coredns
  version: latest
  configurationValues: "{\"replicaCount\":3}"
  resolveConflicts: overwrite
Note

Bear in mind that when addon configuration values are being modified, configuration conflicts will arise.

 
Thus, we need to specify how to deal with those by setting the `resolveConflicts` field accordingly.
As in this scenario we want to modify these values, we'd set `resolveConflicts: overwrite`.

Additionally, the get command will now also retrieve ConfigurationValues for the addon. e.g.

eksctl get addon --cluster my-cluster --output yaml
- ConfigurationValues: '{"replicaCount":3}'
  IAMRole: ""
  Issues: null
  Name: coredns
  NewerVersion: ""
  Status: ACTIVE
  Version: v1.8.7-eksbuild.3

Using custom namespace

A custom namespace can be provided in the configuration file during the creation of addons. A namespace can’t be updated once an addon is created.

Using config file

addons:
  - name: aws-ebs-csi-driver
    version: latest
    namespaceConfig:
      namespace: custom-namespace

Using CLI flag

Alternatively, you can specify a custom namespace using the --namespace-config flag:

eksctl create addon --cluster my-cluster --name aws-ebs-csi-driver --namespace-config 'namespace=custom-namespace'

The get command will also retrieve the namespace value for the addon

- ConfigurationValues: ""
  IAMRole: ""
  Issues: null
  Name: aws-ebs-csi-driver
  NamespaceConfig:
    namespace: custom-namespace
  NewerVersion: ""
  PodIdentityAssociations: null
  Status: ACTIVE
  Version: v1.47.0-eksbuild.1

Updating addons

You can update your addons to newer versions and change what policies are attached by running:

eksctl update addon -f config.yaml
eksctl update addon --name vpc-cni --version 1.8.0 --service-account-role-arn <new-role>
Note
 The namespace configuration cannot be updated once an addon is created. The --namespace-config flag is only available during addon creation.

Similarly to addon creation, When updating an addon, you have full control over the config changes that you may have previously applied on that add-on’s configMap. Specifically, you can preserve, or overwrite them. This optional functionality is available via the same config file field resolveConflicts. e.g.,

addons:
- name: vpc-cni
  attachPolicyARNs:
    - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
  resolveConflicts: preserve

For addon update, the resolveConflicts field accepts three distinct values:

  • none - EKS doesn’t change the value. The update might fail.

  • overwrite - EKS overwrites any config changes back to EKS default values.

  • preserve - EKS preserves the value. If you choose this option, we recommend that you test any field and value changes on a non-production cluster before updating the add-on on your production cluster.

Deleting addons

You can delete an addon by running:

eksctl delete addon --cluster <cluster-name> --name <addon-name>

This will delete the addon and any IAM roles associated to it.

When you delete your cluster all IAM roles associated to addons are also deleted.

Cluster creation flexibility for default networking addons

When a cluster is created, EKS automatically installs VPC CNI, CoreDNS and kube-proxy as self-managed addons. To disable this behavior in order to use other CNI plugins like Cilium and Calico, eksctl now supports creating a cluster without any default networking addons. To create such a cluster, set addonsConfig.disableDefaultAddons, as in:

addonsConfig:
  disableDefaultAddons: true
eksctl create cluster -f cluster.yaml

To create a cluster with only CoreDNS and kube-proxy and not VPC CNI, specify the addons explicitly in addons and set addonsConfig.disableDefaultAddons, as in:

addonsConfig:
  disableDefaultAddons: true
addons:
  - name: kube-proxy
  - name: coredns
eksctl create cluster -f cluster.yaml

As part of this change, eksctl now installs default addons as EKS addons instead of self-managed addons during cluster creation if addonsConfig.disableDefaultAddons is not explicitly set to true. As such, eksctl utils update-* commands can no longer be used for updating addons for clusters created with eksctl v0.184.0 and above:

  • eksctl utils update-aws-node

  • eksctl utils update-coredns

  • eksctl utils update-kube-proxy

Instead, eksctl update addon should be used now.