identity: add node-selector-labels by oblazek · Pull Request #26924 · cilium/cilium

oblazek · 2023-07-19T12:12:13Z

This is revival of #22208, based on latest main.

Please take a look at CFP for more details.

When one enables policy-cidr-match-mode=nodes it allows nodes to be selected by CIDR network policies which is great, but hides the fact that each node in the end has a unique identity from IdentityScopeRemoteNode.

$ k --context infra8.ng -n cilium exec -it cilium-4r24l -- cilium identity list
Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init), install-cni-binaries (init)
ID         LABELS
1          cidr:2a02:598:245::c8c/128
           reserved:host
2          reserved:world
3          reserved:unmanaged
4          reserved:health
5          reserved:init
6          reserved:remote-node
7          reserved:kube-apiserver
           reserved:remote-node
8          reserved:ingress
9          reserved:world-ipv4
10         reserved:world-ipv6
129        reserved:kube-dns
.
.
.
33563866   cidr:2a02:598:128::4616/128
           reserved:remote-node
33563867   cidr:10.246.52.30/32
           reserved:remote-node
33563868   cidr:2a02:598:128::4630/128
           reserved:remote-node
33563869   cidr:10.246.52.22/32
           reserved:remote-node
33563870   cidr:2a02:598:128::4622/128
           reserved:remote-node
33563871   cidr:10.246.52.34/32
           reserved:remote-node
33563872   cidr:2a02:598:128::4634/128
           reserved:remote-node
33563873   cidr:10.246.3.13/32
           reserved:remote-node
33563874   cidr:2a02:598:128::313/128
           reserved:remote-node
33563875   cidr:10.246.52.20/32
           reserved:remote-node
33563876   cidr:2a02:598:128::4620/128
           reserved:remote-node
33563877   cidr:10.246.52.14/32
           reserved:remote-node
33563878   cidr:2a02:598:128::4614/128
           reserved:remote-node
33563879   cidr:10.246.52.12/32
           reserved:remote-node
33563880   cidr:2a02:598:128::4612/128
           reserved:remote-node
33563881   cidr:10.246.52.24/32
           reserved:remote-node

In cases where we have a big clustermesh this results in a lot of identities to be used. Instead we could use this new feature to allow nodes to be selectable by their labels. Which might be filtered out by --node-labels list.
The code also adds a new label source LabelSourceNode="node" which adds node: prefix to node labels.

$ k --context infra8.ng -n cilium exec -it cilium-4r24l -- cilium identity list
Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init), install-cni-binaries (init)
ID         LABELS
33554434   node:scif.cz/node=openstack-worker
           reserved:remote-node
33554435   node:scif.cz/node=labrador
           reserved:remote-node

With that one could specify this new fromNodes/toNodes in their CNPs and allow access only from/to specific nodes:

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "from-nodes"
spec:
  endpointSelector:
    matchLabels:
      {}
  ingress:
  - fromNodes:
    - matchLabels:
        cilium.io/ci-node: k8s1

instead of

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "from-nodes"
spec:
  endpointSelector:
    matchLabels:
      {}
  ingress:
  - fromEntities:
    - remote-node

or

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "from-nodes"
spec:
  endpointSelector:
    matchLabels:
      {}
  ingress:
    - fromCIDR:
      - 10.246.47.36/32
      - 10.249.76.38/32
      - 10.249.189.38/32
      - 10.249.43.31/32
      - 10.249.5.12/32
      - 10.249.81.12/32
      - 10.249.82.12/32
      - 10.249.83.12/32
      - 10.249.95.14/32
      - 10.249.80.54/32
      - 10.249.94.43/32
      - 10.249.96.30/32
      - 10.249.99.38/32
      - 10.249.92.40/32
      - 10.231.3.34/32
      - 10.249.73.21/32
      - 10.249.95.55/32
      - 10.249.97.11/32
      - 10.249.101.36/32
      - 10.249.99.12/32
      - 10.249.102.36/32
      - 10.249.104.36/32
      - 10.249.108.57/32
      - 10.249.38.57/32
      - 10.249.106.48/32
      - 10.249.123.34/32
      - 10.249.72.19/32
      - 10.249.105.36/32
      - 10.249.107.12/32
      - 10.249.110.34/32
      - 10.249.108.12/32
      - 10.249.38.36/32
      - 10.249.106.46/32
      - 10.249.104.38/32
      - 10.249.111.46/32
      - 10.249.121.34/32

where one would need to define all node IPs and keep track of them perhaps in a CiliumCIDRGroup resource.

Fixes: #21615
Fixes: #19121

identity: Allow nodes to be selectable by their labels instead of CIDR and/or remote-node entity.

oblazek · 2023-07-20T13:10:49Z

/test

github-actions · 2023-08-20T01:44:26Z

This pull request has been automatically marked as stale because it
has not had recent activity. It will be closed if no further activity
occurs. Thank you for your contributions.

oblazek · 2023-08-23T07:20:26Z

currently working on the conflicts

github-actions · 2023-09-23T01:44:03Z

This pull request has been automatically marked as stale because it
has not had recent activity. It will be closed if no further activity
occurs. Thank you for your contributions.

oblazek · 2023-09-26T13:32:43Z

reworked code based on #27464 which simplifies the code in a huge way

joestringer · 2023-10-06T16:46:46Z

I think this also fixes #19121 right?

oblazek · 2023-10-06T18:33:15Z

I think this also fixes #19121 right?

yeah it does 😊

oblazek · 2023-10-11T14:13:04Z

/test

oblazek · 2023-10-13T20:28:21Z

/test

nathanjsweet

Thanks! Looks great!

oblazek · 2024-01-09T08:06:12Z

rebased from latest main

oblazek · 2024-01-10T10:05:27Z

/test

joestringer

LGTM. I think the main questions I would have are:

How should we document this new policy construct? Typically we may add some examples under examples directory and reference them in Documentation/security/policy/language.rst, presumably also with the optional flags.
How should we regression test that the functionality here works?

daemon/cmd/daemon_main.go

oblazek · 2024-01-26T07:01:29Z

yeah, I totally want to add a documentation to this; I was thinking about the same part of the docs as you Joe (should I add the docs as part of this PR or a next one?)
I want to finish up this - connectivity: Add per node identity test cilium-cli#2130, but true, this will run only on envs where this feature is enabled

joestringer · 2024-01-26T21:04:53Z

Sounds good to me. Given the Slack discussion I think we agreed on a small change to the user-facing flag, so after resolving that issue and getting CI to pass I think this should be good to merge.

oblazek · 2024-01-29T09:57:26Z

/test

maintainer-s-little-helper · 2024-01-29T11:32:36Z

Commit bb6219a does not match "(?m)^Signed-off-by:".

Please follow instructions provided in https://docs.cilium.io/en/stable/contributing/development/contributing_guide/#developer-s-certificate-of-origin

oblazek · 2024-01-29T11:34:10Z

/test

oblazek · 2024-01-30T06:14:57Z

/test

oblazek · 2024-01-30T13:07:53Z

/test

oblazek · 2024-01-31T08:49:17Z

/test

This commit adds a new flag `--enable-node-selector-labels` which takes advantage of the new IdentityScopeRemoteNode. If this flag is enabled it will filter out node labels based on `--node-labels` list and will append that to remote-node label. This also adds a new label source LabelSourceNode="node" which is prepended to all node labels. Signed-off-by: Ondrej Blazek <ondrej.blazek@firma.seznam.cz>

When node-selector labels are enabled this means that entity 'remote-node' still works but now each node also has custom identity based on its labels (including reserved:remote-node). This mean user has now more fine grained control over what kind of remote-node is allowed/blocked. Signed-off-by: Ondrej Blazek <ondrej.blazek@firma.seznam.cz>

oblazek · 2024-02-05T08:20:08Z

/test

oblazek · 2024-02-05T12:57:12Z

/test

oblazek · 2024-02-05T14:07:25Z

/ci-e2e

oblazek · 2024-02-05T14:07:32Z

/ci-ginkgo

oblazek · 2024-02-05T14:07:37Z

/ci-ipsec-upgrade

maintainer-s-little-helper bot added the dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. label Jul 19, 2023

oblazek force-pushed the ob-per-node-identities branch 7 times, most recently from 078f2a1 to 49d7832 Compare July 20, 2023 09:04

github-actions bot added the stale The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale. label Aug 20, 2023

github-actions bot removed the stale The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale. label Aug 24, 2023

github-actions bot added the stale The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale. label Sep 23, 2023

oblazek force-pushed the ob-per-node-identities branch from 49d7832 to 0d31d38 Compare September 26, 2023 13:31

oblazek force-pushed the ob-per-node-identities branch 4 times, most recently from c9e30c7 to 9a4af8c Compare September 26, 2023 14:17

github-actions bot removed the stale The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale. label Sep 27, 2023

oblazek force-pushed the ob-per-node-identities branch 2 times, most recently from 0a8ce70 to ecb791a Compare October 11, 2023 14:08

oblazek force-pushed the ob-per-node-identities branch from ecb791a to 659ae7b Compare October 13, 2023 14:26

oblazek mentioned this pull request Oct 16, 2023

Add a new option to exclude unwanted k8s node labels from CiliumNode #28290

Merged

nathanjsweet approved these changes Dec 6, 2023

View reviewed changes

joestringer removed the dont-merge/wait-until-release Freeze window for current release is blocking non-bugfix PRs label Dec 15, 2023

oblazek force-pushed the ob-per-node-identities branch from be21684 to b07f887 Compare January 9, 2024 08:06

joestringer added release-note/major This PR introduces major new functionality to Cilium. and removed release-note/minor This PR changes functionality that users may find relevant to operating Cilium. labels Jan 24, 2024

joestringer reviewed Jan 26, 2024

View reviewed changes

daemon/cmd/daemon_main.go Outdated Show resolved Hide resolved

joestringer enabled auto-merge January 26, 2024 21:01

auto-merge was automatically disabled January 29, 2024 09:48
Head branch was pushed to by a user without write access

oblazek force-pushed the ob-per-node-identities branch from b07f887 to 8d44dec Compare January 29, 2024 09:48

oblazek changed the title ~~identity: add per-node labels~~ identity: add node-selector-labels Jan 29, 2024

oblazek added 2 commits February 5, 2024 09:19

oblazek mentioned this pull request Mar 6, 2024

docs: Document enable-node-selector-labels flag #31188

Merged

oblazek mentioned this pull request Dec 17, 2024

helm: add configuration for node-labels #36662

Merged

squeed mentioned this pull request Mar 13, 2026

policy: Fix node selector handling for local host in KCNP #44446

Merged

Conversation

oblazek commented Jul 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

oblazek commented Jul 20, 2023

Uh oh!

github-actions bot commented Aug 20, 2023

Uh oh!

oblazek commented Aug 23, 2023

Uh oh!

github-actions bot commented Sep 23, 2023

Uh oh!

oblazek commented Sep 26, 2023

Uh oh!

joestringer commented Oct 6, 2023

Uh oh!

oblazek commented Oct 6, 2023

Uh oh!

oblazek commented Oct 11, 2023

Uh oh!

oblazek commented Oct 13, 2023

Uh oh!

nathanjsweet left a comment

Choose a reason for hiding this comment

Uh oh!

oblazek commented Jan 9, 2024

Uh oh!

oblazek commented Jan 10, 2024

Uh oh!

joestringer left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

oblazek commented Jan 26, 2024

Uh oh!

joestringer commented Jan 26, 2024

Uh oh!

oblazek commented Jan 29, 2024

Uh oh!

maintainer-s-little-helper bot commented Jan 29, 2024

Uh oh!

oblazek commented Jan 29, 2024

Uh oh!

oblazek commented Jan 30, 2024

Uh oh!

oblazek commented Jan 30, 2024

Uh oh!

oblazek commented Jan 31, 2024

Uh oh!

oblazek commented Feb 5, 2024

Uh oh!

oblazek commented Feb 5, 2024

Uh oh!

oblazek commented Feb 5, 2024

Uh oh!

oblazek commented Feb 5, 2024

Uh oh!

oblazek commented Feb 5, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

oblazek commented Jul 19, 2023 •

edited

Loading

joestringer left a comment •

edited

Loading