Capturing conversation with @bgrant0607 just now.

New feature for kubectl:
when you run kubectl rollingnodeupdate --exec myprogram --success_label=k3=v3 --fail_label=k4-v4 --selector k1=v1,k2=v2, then it would do this:

for each node matching the selector and not matching success_label or fail_label:
   cleanly remove the node
   exec myprogram with env vars set like KUBE_NODE_NAME=nodename
   depending on the return code of myprogram, do one of the following:
     set success_label on the node and return the node to service
     set failed_label and don't retry the node and do return it to service.
     set failed_label and don't retry the node, and do not return it to service.
     set no label and retry the node and do return it to service.
     set no label and retry the node, and do not return it to service.

This cleanly separates the problem of taking a node in and out (which kubectl should know about) from what you might do to a node once it is removed. Things you might do when it is removed include making salt/chef/puppet update the node, reboot the node, delete the node, power it down and wait for a human to touch it, etc.

The clean removal steps are documented in docs/cluster_management.md under Maintenance on a node.