See issue #6079 as the roll-up for node upgrades.

tl;dr:
Update MIG template, then call a MIG rollingupdate.

More details:
When upgrading nodes, an in-place upgrade must worry about leftover state on the boot disk, VM configuration like IP tables, and the idempotency of salt. Getting a fresh VM instead would remove these concerns and provide a clean state for a new node.

Furthermore, a versioned node spec (#4855) includes the tuple of (kubelet, kube-proxy, docker, kernel/OS). In the general case, changing docker or the kernel/OS means we have to stop the running pods. Once know we have to stop the pods, the simplest upgrade mechanism is to totally down the node and provision a new one.

Provisioning a new node is also a great backup to have for node repairs. To be clear, though, this issue is entirely separate from node repairs; once we have the capability to grow the cluster by adding new nodes, this mechanism, and other repair mechanisms, will build on top of that capability.

This was blocked by:

• Allow nodes to dynamically join a running cluster #6087 nodes joining a running cluster
• Node upgrades: GCE instances start with only common metadata #7912 (template-only metadata)

References:

• The most relevant discussion regarding the worries of complications with state while doing in-place upgrades is upgrade: Make node components upgradeable (at all possible) #3333.