This issue is to plan (and track) tests for path 1: in-place upgrades of the kubelet and kube-proxy.

See

• Node upgrades #6079 as the roll-up for node upgrades
• Node upgrades: in-place upgrades #6099 for info about path 1: in-place upgrades of the kubelet and kube-proxy

Once these are implemented, we can see about generalizing or reusing them to also test path 2, the "kill nodes, provision new ones" "heavy-hammer" approach.

1. e2e tests

Unit and integration tests for upgrades seem impossible (at least the node config part; when writing a controller, yes it should be unit-tested), so an e2e test is the most developer-visible way of ensuring upgrades don’t get broken.

1.a. Salt idempotentcy test

1. Create a bunch of stuff
   • RC + pods
   • Services
   • Mounted volumes with data
2. Ensure all the stuff (and basic functionality) works
3. Do an upgrade to this same version (run salt again)
4. Ensure all the stuff works
5. Tear the stuff down

2. Separate upgrade tests

These would take longer than be possible for an e2e test because they would involve running the complete e2e suite, and would involve multiple versions of clusters. The current paradigm of e2e tests is that you have an “e2e cluster;” this notion doesn’t make sense if you’re creating an earlier cluster and upgrading it (unless you always do this first, which seems like it might conflate problems). This would have to run in a separate Jenkins job.

A prerequisite of these tests is that we need a way to determine, statically, which versions of Kubernetes can be upgrade to which other versions. This will likely have to start out as a hand-edited list, but should be possible if we include or find metadata somewhere in a release about the OS and Docker versions it requires.

2.a. "Can you upgrade to me?" test

This test will help determine if HEAD is good from an upgrade perspective.

1. Determine all versions of Kuberenets that can be in-place upgradable to the current version (allowed versions).
2. For each allowed version:
   1. Create a cluster of that version
   2. Optionally: run e2e tests as a sanity check
   3. Upgrade it directly to the current version.
   4. Run the current e2e tests

2.b. Sequential upgrades test

This test will help determine our overall upgrade health between many versions.

1. Determine all sequences of allowed version upgrades. For example, say we have Kubernetes versions a through j. If there is a Docker version change between versions d and e, and there is a required OS version change between versions g and h, then the three sequences would be [a, b, c, d], [e, f, g], and [h, i, j].
2. For each sequence of allowed version upgrades:
   1. Create a cluster of the first version in the sequence
   2. Optionally: run e2e tests as a sanity check
   3. For each remaining allowed version in the sequence:
      1. Upgrade to that version
      2. Run its e2e tests