This is the roll-up issue for node upgrades. It contains the following issues:

• a minimum viable node upgrade mechanism for 1.0
• improvements upon the basic mechanism
• previous ones that discuss cluster upgrades and provisioning tools

Here are the details of our initial requirements for upgrade mechanisms:

• mechanism: out-of-band instead of on the master
• configuration: salt instead of another mechanism
• provider: G*E instead of a provider-agnostic solution
• node mgmt: managed instance groups instead of individual nodes
• must save storage: best effort instead of fully-supported
• language: Bash instead of Go

1.0 requirements

For 1.0, we are working on the tear down / recreate path:

• Tear down / recreate: Upgrading all components of a node together: kubelet, kube-proxy, Docker, and the underlying node image (the userspace and the kernel). This will be done by tearing down and creating nodes in the cluster with a newer supported version tuple. This is blocked by true static clustering—allowing nodes to dynamically join the cluster (Allow nodes to dynamically join a running cluster #6087). This will be the "hammer" we can use for node upgrades and repairs: it is reliable as it starts with a fresh state, it covers all components of a node, but it is disruptive.
• Node upgrades: provision new nodes #6088 build path 2: provision new nodes to upgrade all components
• test: ensure node upgrades between k8s versions work #7914 test path 2
• Node upgrades: e2e test: ensure persistent volumes survive when pods die #6084 Save storage: ensure volumes are retained

Post-1.0

Post-1.0, we want to work on an in-place upgrade, and improve our existing mechanism to make it provider-agnostic, possibly run in a pod that manages the cluster, and other improvements listed below.

• Binary upgrade: Upgrading kubernetes binaries (kubelet, kube-proxy) independently of Docker and the node image. This is currently doable using kube-push.sh for some cloud providers. We need to ensure the cluster stays running in a good state after the upgrade, and support upgrading to official releases. This will cover most common upgrades, and future extensions to this mechanism (for example, to upgrade Docker) will allow for more thorough upgrades to be less disruptive, and help cover more use cases such as node repair.

The following are ordered by priority.

• Node upgrades: in-place upgrades #6099 build path 1: kubelet/kube-proxy in-place binary upgrade for 1.0.
• Node upgrades: tests for path 1 (in-place binary upgrades) #7256 test path 1
• Node upgrades: graceful node evacuation #6080 Node evacuation
• Node upgrades: version-aware upgrades #6086 Only upgrade nodes whose versions can and should be upgraded
• Node upgrades: write in Go and hook into pkg/cloudprovider #6095 Write in Go and hook into pkg/cloudprovider
• Node upgrades: provider agnosticism #6096 Make node upgrades work for all supported providers
• Node upgrades: investigate putting the mechanism on the master #6106 Put the node upgrade mechanism on the master

Related work

• Cluster Upgrade #2524 Full cluster upgrade discussion
• WIP: Modify kube-push for GCE to bring down the existing master VM and completely replace it with a new one #3174 Looked at requirements for master upgrades (GCE)
• kubectl doesn't fail silently with no k8s master #4175 Put master data on a PD and reserved the master's IP (GCE)
• upgrade: Make node components upgradeable (at all possible) #3333 Node component upgrades
• upgrade: Make node components upgradeable (via online don't-kill-containers) #1573 Online kubelet upgrades
• upgrade: Rolling system updates via delete/re-create nodes #5044 Node rolling upgrades
• Cluster versioning #4855 Full cluster upgrade discussion for 1.0; clarifies versioning
• Rolling node update #5511 Suggests kubectl command to do node rolling upgrades
• Proposal to rework Kubernetes deployment CLI #5472 Proposal for kubedeploy to replace kube-{up,down,push}.sh
• Master upgrades #6075 Master upgrades roll-up