Can't install Kubernetes >=1.22 on Flatcar Linux due to missing SELinux custom policies #2788
Description
RKE version:
1.3.3 (using terraform RKE provider v1.3.0)
Docker version: (docker version,docker info preferred)
Client:
Context: default
Debug Mode: false
Server:
Containers: 17
Running: 9
Paused: 0
Stopped: 8
Images: 13
Server Version: 20.10.11
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: false
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: cde01e96ed658bc5050abe1bb601b4b4510ba7a2
runc version: e4bccdbd64361ac5ea8ba90bb8845add78f957a6
init version:
Security Options:
seccomp
Profile: default
selinux
cgroupns
Kernel Version: 5.10.84-flatcar
Operating System: Flatcar Container Linux by Kinvolk 3033.2.0 (Oklo)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.807GiB
Name: worker-01
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Operating system and kernel: (cat /etc/os-release, uname -r preferred)
NAME="Flatcar Container Linux by Kinvolk"
ID=flatcar
ID_LIKE=coreos
VERSION=3033.2.0
VERSION_ID=3033.2.0
BUILD_ID=2021-12-10-1820
PRETTY_NAME="Flatcar Container Linux by Kinvolk 3033.2.0 (Oklo)"
ANSI_COLOR="38;5;75"
HOME_URL="https://flatcar-linux.org/"
BUG_REPORT_URL="https://issues.flatcar-linux.org"
FLATCAR_BOARD="amd64-usr"
5.10.84-flatcar
Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO)
Master/Worker nodes provisionned by terraform using the RKE provider v1.3.0. Nodes are vSphere virtual machines based on the Flatcar OVA.
cluster.yml file:
As I'm using the terraform provider, here's the tf rke_cluster declaration
resource "rke_cluster" "main" {
kubernetes_version = "v1.22.4-rancher1-1"
cluster_name = "test-cluster"
authentication {
strategy = "x509"
sans = "<...redacted...>"
}
dynamic "nodes" {
for_each = flatten([local.rke_cluster_master_nodes, local.rke_cluster_worker_nodes])
content {
address = nodes.value["address"]
ssh_key = nodes.value["id_rsa"]
labels = nodes.value["labels"]
role = nodes.value["roles"]
hostname_override = nodes.value["name"]
user = nodes.value["user"]
}
}
dns {
provider = "coredns"
}
ingress {
provider = "none"
}
network {
plugin = "calico"
options = {
"calico_cloud_provider" : "none",
"calico_flex_volume_plugin_dir" : "/var/lib/kubelet/volumeplugins"
}
}
services {
kube_api {
audit_log {
enabled = true
}
secrets_encryption_config {
enabled = true
}
}
}
upgrade_strategy {
drain = false
max_unavailable_worker = 1
max_unavailable_controlplane = 1
}
}
Steps to Reproduce:
Try to update a kubernetes cluster from 1.21 (or possibly earlier versions) to 1.22 when using Flatcar OS 3033.2.0. I imagine that a fresh 1.22 installation would lead to the same result.
Results:
The following error occurs:
Failed running cluster err:[[selinux] Host [10.130.0.241] does not recognize SELinux label [label=type:rke_container_t]. This is required for Kubernetes version [>=1.22.0-rancher0]. Please install rancher-selinux RPM package and try again]
As shown in docker info above, SELinux is enabled on dockerd, triggering this specific step from RKE. Starting from 1.22, a dedicated custom SELinux policy must be installed on SELinux-enabled nodes. As I'm using Flatcar Linux, it's not possible to deploy this RPM as-is.
I'm quite a newbie when it comes to SELinux and I don't see how I can easily work around this as disabling SELinux on the docker daemon is not an option for me. Is there any plan on RKE side to better integrate this with Flatcar Linux ? I may be missing a simple way to circumvent this so don't hesitate to tell me ^^
Thanks