It should be possible to update a kubelet version, and the kubelet's dependencies, without terminating the pods on a machine.

Upgrade reasons:

• add new features to running cluster without disruptive restart of pods.
• a security issue found in kubelet, which runs as root

Issues:

• Kubelet restart should work fine.
• Kubelet version may depend on docker version. It appears docker doesn't restart reliably without losing containers (Docker restart on upgrade left processes behind moby/moby#5685, problems restarting the docker daemon for upgrades  moby/moby#7175)
• Managing kubelet version on VMs

About managing kubelet version on VMs:

• Users with large clusters may wish to do a rolling upgrade of the kubelet version.
  • kubernetes will already has a plan to address this for pods by using replication controllers and templates.
• Users may want clusters that grow automatically: VMs are added as knodes as needed.
  • VMs will be created from some template. Either that contains a kubelet binary, in which case, it needs to be the right version, or there needs to be something, of the same general nature as Salt, which configures the machine before it joins the cluster.
    Self-hosting could address these issues, though not without its own complexities.