Can't install Kubernetes >=1.22 with RKE due to missing SELinux custom policies #598
Description
This is not a "bug" on Flatcar side. The issue is more about the way to deal with custom SE Linux policies but I was not sure how to contextualize the issue so feel free to remove the "bug" label. Still, the impact is not trivial.
Description
Using RKE, installing (or upgrading) Kubernetes >= 1.22 fails with the following error message
Failed running cluster err:[[selinux] Host [10.130.0.241] does not recognize SELinux label [label=type:rke_container_t]. This is required for Kubernetes version [>=1.22.0-rancher0]. Please install rancher-selinux RPM package and try again]
Starting with 1.22, RKE (and RKE2) chose to use custom SE Linux policies for their setup. These can be installed through dedicated RPMs: rancher-selinux for RKE and rke2-selinux for RKE2 but these can't be used as is with Flatcar.
I also opened an issue on RKE side to get their opinion on this : rancher/rke#2788
Impact
It's not possible to use RKE (or RKE2) with Flatcar Linux starting with Kubernetes 1.22.
Environment and steps to reproduce
- Set-up
Flatcar version :
NAME="Flatcar Container Linux by Kinvolk"
ID=flatcar
ID_LIKE=coreos
VERSION=3033.2.0
VERSION_ID=3033.2.0
BUILD_ID=2021-12-10-1820
PRETTY_NAME="Flatcar Container Linux by Kinvolk 3033.2.0 (Oklo)"
ANSI_COLOR="38;5;75"
HOME_URL="https://flatcar-linux.org/"
BUG_REPORT_URL="https://issues.flatcar-linux.org"
FLATCAR_BOARD="amd64-usr"
I'm using RKE through terraform :
resource "rke_cluster" "main" {
kubernetes_version = "v1.22.4-rancher1-1"
cluster_name = "test-cluster"
authentication {
strategy = "x509"
sans = "<...redacted...>"
}
dynamic "nodes" {
for_each = flatten([local.rke_cluster_master_nodes, local.rke_cluster_worker_nodes])
content {
address = nodes.value["address"]
ssh_key = nodes.value["id_rsa"]
labels = nodes.value["labels"]
role = nodes.value["roles"]
hostname_override = nodes.value["name"]
user = nodes.value["user"]
}
}
dns {
provider = "coredns"
}
ingress {
provider = "none"
}
network {
plugin = "calico"
options = {
"calico_cloud_provider" : "none",
"calico_flex_volume_plugin_dir" : "/var/lib/kubelet/volumeplugins"
}
}
services {
kube_api {
audit_log {
enabled = true
}
secrets_encryption_config {
enabled = true
}
}
}
upgrade_strategy {
drain = false
max_unavailable_worker = 1
max_unavailable_controlplane = 1
}
}
- Task
Run terraform apply to deploy (or upgrade) your cluster to 1.22
- Action(s)
Wait for the error
- Error: [describe the error that was triggered]
Error is triggered early in the process as it's a pre-check done by RKE before doing the actual installation
Failed running cluster err:[[selinux] Host [10.130.0.241] does not recognize SELinux label [label=type:rke_container_t]. This is required for Kubernetes version [>=1.22.0-rancher0]. Please install rancher-selinux RPM package and try again]
Expected behavior / Additional information
I am a complete newbie when it comes to SE Linux and I don't really know of a way to work around this. As mentioned in the RKE issue, I manually tried to import the SE module from the RPM in my Flatcar instance but failed because of /usr being read-only.
I didn't find any documentation about adding custom SE Linux configuration on a Flatcar instance. It feels like it's not easily doable without maintaining a custom Flatcar image which seems overkill for this kind of small configuration tweak and I'd like to avoid it.
What would be the best way to tackle this ? As RKE (and RKE2) are popular tools for deploying Kubernetes, does it make sense to request for new packages based on their RPMs ?
Thanks