This repository was archived by the owner on May 12, 2021. It is now read-only.
This repository was archived by the owner on May 12, 2021. It is now read-only.
9P/virtio-fs: inotify does not work: ConfigMap Updates #1505
Description
ConfigMap Updates: inotify doesn't work on 9p/virtio-fs filesystem mounts
A common design pattern in Kubernetes is to watch for changes to files/directories passes in as Configmaps or Secrets. Sidecar's normally use inotify to watch for changes and then signal the main container or perform some other type of reconfiguration.
Given that Kata today uses 9p to pass in both ConfigMaps as well as Secrets into the VM this will not work.
Note: This will be an issue for virtio-fs too
Expected result
Here is an example for a sidecar used to inject configuration change
apiVersion: v1
data:
authorized_keys: |
ssh-rsa USE_SIDECAR_TO_ADD_KEYS_kubectl_cm_edit_ssh-pub-key
kind: ConfigMap
metadata:
name: ssh-pub-key
---
apiVersion: v1
kind: Pod
metadata:
name: footfed
spec:
runtimeClassName: kata
volumes:
- name: runv
emptyDir:
medium: "Memory"
- name: runlockv
emptyDir:
medium: "Memory"
- name: tmpv
emptyDir:
medium: "Memory"
- name: fakecgroup
hostPath:
path: /sys/fs/cgroup
- name: ssh-dir
emptyDir:
medium: "Memory"
- name: ssh-pub-key
configMap:
name: ssh-pub-key
defaultMode: 384
containers:
# This sidecar is used to constantly update the keys allowing the VM
# keys to be revoked or updated
- name: key-sidecar
image: pstauffer/inotify
command: ["sh", "-xc", "chmod 700 /root/.ssh; cp /tmp/keys/authorized_keys /root/.ssh/authorized_keys; inotifywait -m -r -e create /tmp/ | while read IGNOREME; do echo /tmp/keys/authorized_keys; cp /tmp/keys/authorized_keys /root/.ssh/authorized_keys; chmod 600 /root/.ssh/authorized_keys; done"]
volumeMounts:
- name: ssh-dir
mountPath: /root/.ssh
- name: ssh-pub-key
mountPath: /tmp/keys
- name: footfed
image: quay.io/footloose/fedora29:latest
command: ["/sbin/init"]
volumeMounts:
- name: runv
mountPath: /run
- name: runlockv
mountPath: /run/lock
- name: tmpv
mountPath: /tmp
- name: fakecgroup
readOnly: true
mountPath: /sys/fs/cgroup
- name: ssh-dir
mountPath: /root/.ssh
Once the pod is fully up and running
- Edit the config map using
kubectl edit cm ssh-pub-keyand modify the key
Watch for the key to change (it will take a few seconds
watch kubectl exec footfed -c footfed -- cat /root/.ssh/authorized_keys
You should see the new key being setup
Actual result
The key is never updates as the config map change is never detected.
You can also see this in the side-car logs when things work
Every 2.0s: kubectl logs footfed -c key-sidecar bored-pelinor: Tue Apr 9 01:05:06 2019
+ chmod 700 /root/.ssh
+ cp /tmp/keys/authorized_keys /root/.ssh/authorized_keys
+ inotifywait -m -r -e create /tmp/
+ read IGNOREME
Setting up watches. Beware: since -r was given, this may take a while!
Watches established.
+ echo /tmp/keys/authorized_keys
+ cp /tmp/keys/authorized_keys /root/.ssh/authorized_keys
/tmp/keys/authorized_keys
+ chmod 600 /root/.ssh/authorized_keys
/tmp/keys/authorized_keys
+ read IGNOREME
+ echo /tmp/keys/authorized_keys
+ cp /tmp/keys/authorized_keys /root/.ssh/authorized_keys
+ chmod 600 /root/.ssh/authorized_keys
+ read IGNOREME