Note

I have started splitting the draft PR into smaller chunks.
Here's the PR for the datapath - [GH-41867][Part-1][eBPF] Add support for hybrid routing in datapath by anubhabMajumdar · Pull Request #41868 · cilium/cilium .

Description

The motivation behind adding a new routing mode is discussed in this CFP . This PR provides an overview for how this change will be implemented. Given the size and complexity, the changes have been grouped into logical commits. Depending on feedback, will probably create separate smaller PRs to actually merge in the changes to main.

Overview

This feature would allow user to configure Cilium cluster(s) with multiple subnets, and benefit from both tunnel and native routing mode for maximum efficiency (no overhead of tunneling when not needed).

There's a more verbose description here. It goes into much more detail about the changes and how testing was performed, along with diagram explaining the setup.

Motivation

1. Given that tunneling can always route a packet as long as nodes have connectivity, the entire change is designed to use tunnel as default, with exceptions built in to skip tunneling when IPs belong to same subnet.
2. The change allows users to dynamically change the subnet topology. This will help skip agent restart.
3. I have tried to adhere to existing patterns that I could identify

• the eBPF map and lookup function closely resembles the IPCache library. Will clean them up to remove fields not needed (ex - cluster ID) before merging.
• the userspace controller similarly resembles dynamic controllers like dynamic exporter in Hubble.

Changes Overview

Datapath

This covers all the C code changes. Files edited/added:

1. bpf/lib/subnet.h - This has the LPM trie based subnet map, similar to the IPCache map. Has functions for lookup.
2. bpf_lxc.c - Added an extra check to skip_tunnel. Now, skip tunneling if IPs belong to same subnet.
3. bpf_host.c - Added an extra check to skip_tunnel. Now, skip tunneling if IPs belong to same subnet.
4. Added two debug keys - DBG_SUBNET_CHECK and DBG_TUNNEL_TRACE for debugging purposes. Will remove these before merging.

Userspace

This covers the user side code that watches, reads, parses and update the eBPF map with subnets. All files are under pkg/subnettopology. The pattern closely resembles other dynamic controllers like dynamic exporter, which reads a file every so many seconds and reconciles the change if it finds any.

Configuration

This covers all changes needed to add a new option under routing-mode. Also. when hybrid routing mode, allow settings for both tunnel and native routing. Files edited:

1. daemon/cmd/daemon_main.go
2. pkg/option/config.go

Install

YAML file changes to add the new hybrid routing related options.
NOTE: Will revert back to runnel mode as default routing option in values.yaml.

Testing Done

I haven't added any new tests to verify this change yet. Part of the reason for this change to have a discussion here as to how this should be tested going forward. I think we would probably need to setup new infrastructure to test it in Cluster Mesh with dynamic subnets.

Here's how I have tested this change in single/multi-cluster scenarios. All testing were done on following

• Cluster Type - AKS
• Networking - Pod and Node subnets for each cluster in mesh is deployed within the same Azure Virtual Network (VNET)

Single cluster

1. Deploy a cluster with no CNI installed. The Pod/Node CIDR is backed by two subnets belonging to same VNET
2. Install Cilium with hybrid routing. However, configure no subnets
3. Deploy a client-server on two different nodes
4. Check Hubble flows between a client-server setup. We see to-overlay flows on the client node.
5. Check cilium debug messages on the node to verify same_subnet=false.
6. Now, configure the pod subnet and check the eBPF map to see if the updates have sync'd
7. Check Hubble flows between a client-server setup. We see to-stack/network flows on the client node.
8. Check cilium debug messages on the node to verify same_subnet=true.
9. Remove the subnet config again. Repeat Steps 4 and 5.

Cluster Mesh

1. Setup 2 clusters as above with Cilium installed. Setup VNET peering between the two VNETs. Configure subnet on both cluster with the pod CIDRs of both clusters (ex: podcidr1,podcidr2).
2. Setup client on one cluster and server on the other. Start traffic from client->server.
3. Check Hubble flows. We see to-stack/network flows on the client node.
4. Check cilium debug messages on the node to verify same_subnet=true.
5. Edit the subnet config to podcidr1;podcidr2 (means no direct routing possible between two CIDR range).
6. Check Hubble flows. We see to-overlay flows on the client node.
7. Check cilium debug messages on the node to verify same_subnet=false.

Notes

• Haven't implemented atomic read of the subnet map while making routing decision yet
• Will remove the debug traces from data path before merging the PR
• Missing unit tests, will add after I get some initial feedback
• Will revert back to runnel mode as default routing option in values.yaml
• Will remove the README before merging
• Marking PR as draft. This PR will provide an overview of exactly what I am trying to achieve. Given its size and complexity, I will break it up into pieces for easier review and merging.

Please ensure your pull request adheres to the following guidelines:

• For first time contributors, read Submitting a pull request
• All code is covered by unit and/or runtime tests where feasible.
• All commits contain a well written commit description including a title,
  description and a Fixes: #XXX line if the commit addresses a particular
  GitHub issue.
• If your commit description contains a Fixes: <commit-id> tag, then
  please add the commit author[s] as reviewer[s] to this issue.
• All commits are signed off. See the section Developer’s Certificate of Origin
• Provide a title or release-note blurb suitable for the release notes.
• Are you a user of Cilium? Please add yourself to the Users doc
• Thanks for contributing!

Fixes: #issue-number

<!-- Enter the release note text here if needed or remove this section! -->