Description

The latest flatcar move to systemd 245 introduces this problem with cilium: cilium/cilium#10645

A quick fix is to create a systemd override: echo 'net.ipv4.conf.lxc*.rp_filter = 0' | sudo tee -a /etc/sysctl.d/90-override.conf && sudo systemctl restart systemd-sysctl

Or via ignition:

    - path: /etc/sysctl.d/90-override.conf
      filesystem: root
      mode: 420
      contents:
        inline: net.ipv4.conf.lxc*.rp_filter = 0

Impact

This broke / breaks our production clusters after moving to the channel. A previously healthy / functioning cluster after this update can no longer respond to inbound traffic (can't ack back) and components that require external connectivity to function no longer work. Inter-node / inter-cluster traffic stays the same.

Environment and steps to reproduce

1. Set-up:

Cilium 1.8.2
Flatcar Beta or Alpha channel 2605
Kubernetes 1.18.8

2. Task:

Pod egress traffic fails

3. Action(s):

Moving from previous flatcar version to new 2605 channel versions with systemd 245

4. Error:

No direct errors are reported by the system directly; traffic failing to egress only generates more synthetic / business-specific alerts regarding service connectivity

Expected behavior

I would have expected these filters to be dropped from sysctl settings similar to debian:

$ grep -R rp_filter systemd_245.4-4ubuntu3.2/debian/patches/*
systemd_245.4-4ubuntu3.2/debian/patches/debian/UBUNTU-drop-kernel.-settings-from-sysctl-defaults-shipped.patch:-net.ipv4.conf.default.rp_filter = 2
systemd_245.4-4ubuntu3.2/debian/patches/debian/UBUNTU-drop-kernel.-settings-from-sysctl-defaults-shipped.patch:-net.ipv4.conf.*.rp_filter = 2
systemd_245.4-4ubuntu3.2/debian/patches/debian/UBUNTU-drop-kernel.-settings-from-sysctl-defaults-shipped.patch:--net.ipv4.conf.all.rp_filter