Subtasks

• CI: Add policy test to testPodConnectivityAcrossNodes #11486 - Direct-routing + (L7) policy

Dimensions

Included in matrix below

• Endpoint devices - veth / ipvlan
• Cluster connectivity - direct-routing (endpoint routes) / direct-routing (tailcall) / tunnel-mode
• Service connectivity - kube-proxy nodeport / BPF nodeport
• Traffic security - encryption / no encryption
• Endpoint security - L7 / no L7
• Socket acceleration - sockops / no sockops

Excluded from matrix below

The below are excluded to keep the matrix to a manageable size; their impact on dataplane connectivity is lower than those in the above list.

• Host services - host-reachable / none
• Host security - encrypt-node / none
• IP connectivity - IPv4 / IPv6 / both

Commandline arguments for configuration

• --datapath-mode (veth / ipvlan)
• --ipam (This actually controls not only IPAM, but also ENI mode for CNI)
• --tunnel, --auto-direct-node-routes
• --enable-node-port
• --masquerade
• --enable-host-reachable-services
• --install-iptables-rules (impacts ipsec, L7 policy, NodePort via kube-proxy)
• --enable-ipsec
• --encrypt-interface (should be covered by --enable-ipsec)
• --encrypt-node
• --single-cluster-route (incompatible with tunnel disabled)
• --enable-endpoint-routes
• --sockops-enable
• --enable-ipv4 / --enable-ipv6

Configuration Matrix

Reading the matrix

• The top of the matrix determines settings related to security
• The left of the matrix determines settings related to connectivity

Key

• X -- Invalid combination
• N -- No known testing
• M -- Manually tested during 1.6 cycle
• A -- Automatically regression-tested by CI (basic connectivity)

Matrix

Note, eni is the cluster connectivity device type + configuration, not endpoint device type. ENI is typically onfigured with veth devices. This matrix needs updating.

Footnotes

• [0] via https://github.com/cilium/cilium/blob/master/test/runtime/connectivity.go
• [1] via https://github.com/cilium/cilium/blob/master/test/runtime/Policies.go
• [2] via https://github.com/cilium/cilium/blob/master/test/k8sT/DatapathConfiguration.go
• [3] via https://github.com/cilium/cilium/blob/master/test/k8sT/Services.go
• [4] I can't find any documentation related to this, but I don't think BPF nodeport works with tunneling or veth.
• [5] via https://docs.cilium.io/en/latest/gettingstarted/ipvlan/#ipvlan-based-networking-beta
• [6] I can't find any documentation related to this, but I don't think ENI + Tunnel works. ENI + Tunnel works but it's pretty pointless. I've manually tested it but don't expect anybody to use it.

Notes and limitations

• This matrix is focused on configuration elements that affect datapath configuration such as BPF programs, iptables rules, rule and route configurations. It does not attempt to distinguish policy cases such as in-cluster/out-cluster; L3/L4/L7 Policy (other than for the L7 redirects); etc.
• The datapath configuration tests configure different combinations of the above dimensions, however they do not cover the matrix with L7 policy; nor do they make calls through services; only directly via pod IPs.
• Per-endpoint routes is considered orthogonal to tunnel and only compatible with direct-routing, hence only an option for direct routing in the table above.