Dedicated nodes, taints, and tolerations design proposal by davidopp · Pull Request #18263 · kubernetes/kubernetes

davidopp · 2015-12-06T22:58:33Z

ref/ #17190

It's helpful to read #18261 before this one.

@bgrant0607 @mml @kevin-wangzefeng @alfred-huangjian @mikedanese @erimatnor @timothysc

k8s-github-robot · 2015-12-06T23:00:32Z

Labelling this PR as size/L

k8s-bot · 2015-12-06T23:30:26Z

GCE e2e test build/test passed for commit bd131c17dc115f350f369e5b3a739a846c301e25.

k8s-bot · 2015-12-07T00:19:19Z

GCE e2e test build/test passed for commit 11d11e14a914be2f20022e1071a778a82b5c4c0c.

kevin-wangzefeng · 2015-12-07T10:56:10Z

docs/proposals/dedicated-nodes.md

Will probably implement this mechanism in Rescheduler, or anywhere else?

You could probably do it in the rescheduler, but I think it's probably better to do it in Kubelet.

mml · 2015-12-07T20:44:08Z

Can you cover use cases? If security is one motivation, we'll need to address things like the race I mentioned in #17190 .

mml · 2015-12-08T00:40:33Z

Is "keep dedicated users off the shared machines" a use case we care about?

davidopp · 2015-12-08T05:22:58Z

Comments from meeting with @bgrant0607 today follow. I will update the proposal to reflect them shortly:

Make it clear that Kubelet should enforce taints/tolerations (not just scheduler); goes into shared predicate library between kubelet and scheduler
String constants in API should start with capital letter (e.g. should be EvictAndNoStart, not evictAndNoStart)
Instead of two behaviors (evict, and no start) should have three behaviors: scheduler should not schedule onto the node, kubelet should not accept a pod onto the node, and evict. Kubelet only cares about last two. See bottom of Mark node to be decommissioned and act accordingly #3885 (comment) for more details
Consider relationship between tolerations and forgiveness, and correspondingly use cases that are about machine drains/failures/etc. Proposal: Forgiveness #1574. It's not clear that this requires a change to the design, except long-term we might want to add a timeframe to tolerations so you can say for example "I am willing to stay bound to a machine that is down for maintenance for up to 4 hours, but not longer." Taint auto-applied when problem happens, and has a default toleration.
The current proposal special-cases * as the Value of a Toleration to me wildcard rather than string asterisk. To be more consistent with rest of Kubernetes API, there should be no special-case for interpreting Value, instead add an "Operation" field to Toleration that can take on the values "Equal" and "Exists" which defaults to "Equal."
Should tell users not to use these features until they have been in Kubelet and master for enough binary versions that we feel comfortable we will not need to roll back either to a version that does not support them; longer-term would be good to have a way to enforce this (Cluster versioning #4855).

davidopp · 2015-12-08T11:15:01Z

Can you cover use cases? If security is one motivation, we'll need to address things like the race I mentioned in #17190 .

Yeah, it's definitely a good point. I think this falls under "In the future one can imagine an admission controller that applies taints to nodes and tolerations to pods based on a policy specifying dedicated node groups." Actually I realized the "taints to nodes" part doesn't make sense (admission controllers have nothing to do with nodes). But we could create some kind of API object that describes the dedicated node policy, and NodeController could attach the necessary taints when a machine registers based on the policy stored in that object (we can have the node register in an unschedulable state until NodeController marks it ready, after adding the taints). The "tolerations to pods" policy could use the mechanism described in #18262.

Is "keep dedicated users off the shared machines" a use case we care about?

Probably in at least some cases yes. Putting a taint on all of the shared machines is easy; I guess the annoying part is that you'd need to put a toleration on all of the non-dedicated users' pods (unless there's something I'm not thinking of).

davidopp · 2015-12-08T11:16:07Z

(continuing from end of last comment)

I guess you could say any pod that has no tolerations is assumed to tolerate the "shared machine" taint, or something hacky like that...

davidopp · 2015-12-08T23:06:18Z

@erictune

ncdc · 2015-12-09T16:44:33Z

cc @kubernetes/rh-cluster-infra

pmorie · 2015-12-09T18:26:04Z

docs/proposals/dedicated-nodes.md

Is the cordon/uncordon functionality in #16698 expressable in terms of taints?

@pmorie Yes. Discussed on #1574

mml · 2015-12-09T22:47:09Z

Do we have strong enough use cases that are not adequately addressed by using clusters to implement dedicated node pools to justify this added cognitive and implementation complexity?

Suppose the dedicated nodes share power infrastructure with the shared nodes. Managing drains across all clusters gets tricky and error-prone.

erictune · 2015-12-09T23:01:46Z

docs/proposals/dedicated-nodes.md

If all nodes have a dedicated label, then all pods can have a simple equality node constraint.

For example, suppose we have dedicated machine set "gpus".
Label those machines "dedicated=gpus", and label the rest of the machines "dedicated=public".
Then all pods just need a NodeConstraint: dedicated=gpus or NodeConstraint: dedicated=public.

The main benefit of taints over regular constraints is that with taints, pods that don't need to access the "special" nodes don't need to know anything about the special nodes. So for example, if you have a cluster with no GPUs and then you add some nodes to it with GPUs and you want only pods that request GPUs to be able to schedule onto those nodes, you just add a taint to the new nodes and a toleration to the pods that request GPUs. "Regular" pods (and nodes) can be oblivious. This property becomes especially valuable once you have lots of different types of special nodes (multiple dedicated groups, multiple special hardware types, different kinds of machine problems that exclude pods, etc.). With the constraint approach, you need "regular" pods to explicitly constrain away from every different type of special node.

The one place where this becomes a little less beneficial is when there are no "regular" nodes, as in the case where you not only want to keep pods off of dedicated nodes they aren't entitled to access, but also want to keep some pods (presumably the pods of dedicated users) off of non-dedicated nodes. In that case the world is essentially as you described in your comment, where every node is effectively dedicated, and taints/tolerations are not as valuable because the non-dedicated nodes must be tainted and the non-dedicated-user pods must have tolerations. But taints/tolerations are still better than constraints even in that case because as you add dedicated node groups, you don't need to change how you handle the non-dedicated pods, whereas with the constraint approach, you would need to start adding a "dedicated != " constraint to all future and currently-pending pods.

erictune · 2015-12-09T23:13:37Z

Is there a way to use taints to keep users off nodes that I want to be restricted?

ghost · 2015-12-09T23:15:46Z

docs/proposals/dedicated-nodes.md

The non-commutative nature of this one makes me uneasy (because it compromises the declarative semantics that we strive for across Kubernetes). As a trivial example, if a request to Taint:noStart all nodes arrives at roughly the same time as a request to launch a pod on every node, it's not clear what the correct/desirable steady state of the cluster should be afterwards. With Taint:evictAndNoStart that's clear, irrespective of what order the requests arrive in.

Could you list some real-world use cases for noStart. I realise that there are some obvious ones (e.g. stop scheduling stuff on this node/cluster, so that I can slowly retire it), but I'm hoping that we can shoot them down as unnecessary or not useful in practise (as I believe the aforementioed to be). In which case we can remove the entire TaintEffect concept (at least for now). And then a Taint just becomes a key-value pair like a label, and life becomes a lot simpler.

I don't think that the problem you're describing is a big deal. Someone who adds a noStart taint by definition doesn't care about what is running on the machine, they just don't want new stuff to start running there. So I don't think they care what the final state is in the scenario you described.

The main use case for the noStart taint is exactly the one you mentioned -- to "cordon" (#16698) a machine in preparation for draining.

erictune · 2015-12-09T23:19:52Z

Sorry. Please disregard last comment. I see that you explain this in the doc.

erictune · 2015-12-09T23:31:21Z

I think a requirement for dedicated machines is that "user A is allowed to run pods on dedicated machine set X but user B is not able to.". I don't see a description of what keeps User B from putting an arbitrary Toleration on her pod. Presumably some sort of admission controller is needed to enforce which Tolerations user B is allowed to put on her pod?

bgrant0607 · 2016-01-07T05:10:16Z

docs/proposals/dedicated-nodes.md

How about dedicated=special-user:NoScheduleNoAdmitNoExecute. (Instead of a second =)

Sure. I didn't really like the syntax I was using but couldn't think of anything better.

bgrant0607 · 2016-01-07T05:14:43Z

Looks pretty good. Just some minor comments.

bgrant0607 · 2016-01-07T05:18:48Z

See also #17151

davidopp · 2016-01-13T01:24:41Z

Don't remember if this is mentioned by the proposal:
we should "deprecate" the existing NodeSpec.unschedulable field and represent that as a taint
we should automatically apply taints for node conditions that cause the node to become unschedulable

These are good points but I'm trying to avoid expanding the scope of this doc too much. The idea was to present taints and tolerations, and give one convincing use case. I think we can do followup docs to explain how to use them in node maintenance and other scenarios. The first paragraph alludes to the fact that there are other use cases.

davidopp · 2016-01-13T04:03:24Z

@bgrant0607 I addressed your feedback; PTAL.

k8s-bot · 2016-01-13T04:38:36Z

GCE e2e test build/test passed for commit 566d138164f9fbc2f7e6a8ed39189414469969df.

bgrant0607 · 2016-01-16T02:38:03Z

docs/proposals/dedicated-nodes.md

Perhaps we could elide the extra Nos?

NoScheduleAdmitExecute

I think some people might mistakenly think that the "No" binds only to "Schedule" rather than to "ScheduleAdmitExecute"

bgrant0607 · 2016-01-22T18:56:09Z

Note: I don't remember why we decided not to support a richer set of operations in Toleration. In could be achieved by multiple tolerations, but NotIn cannot. I can't think of a use case for that, but this will be hard to extend.

However, we probably will implement this as an alpha annotation initially, in which case we could change it if we discovered use cases for more operators. I don't think we need to update the proposal to reflect that detail.

LGTM.

k8s-github-robot · 2016-01-24T07:15:02Z

PR changed after LGTM, removing LGTM.

k8s-bot · 2016-01-24T07:45:07Z

GCE e2e test build/test passed for commit 14c2763.

davidopp · 2016-01-24T20:15:49Z

Tests pass, manually merging as this is a doc-only PR with just one doc. @bgrant0607 's comment #18263 (comment) about possibly allowing a richer set of operations for Toleration is noted as a point for further consideration. (I don't have an opinion on it really; I think we just used the narrowest semantics that we knew for sure there were use cases for.)

Dedicated nodes, taints, and tolerations design proposal

davidopp assigned bgrant0607 Dec 6, 2015

googlebot added the cla: yes label Dec 6, 2015

k8s-github-robot added kind/design Categorizes issue or PR as related to design. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Dec 6, 2015

davidopp mentioned this pull request Dec 6, 2015

Inter-pod topological affinity/anti-affinity proposal #18265

Merged

davidopp force-pushed the taints-tolerations branch from bd131c1 to 11d11e1 Compare December 6, 2015 23:44

davidopp changed the title ~~Dedicated nodes, taints, and tolerations design proposal initial draft~~ Dedicated nodes, taints, and tolerations design proposal Dec 7, 2015

kevin-wangzefeng reviewed Dec 7, 2015
View reviewed changes

bgrant0607 mentioned this pull request Dec 8, 2015

Proposal: Forgiveness #1574

Closed

davidopp mentioned this pull request Dec 8, 2015

[WIP]Dedicated machines #18152

Closed

kevin-wangzefeng mentioned this pull request Dec 9, 2015

Node affinity and NodeSelector design doc #18261

Merged

pmorie reviewed Dec 9, 2015
View reviewed changes

erictune reviewed Dec 9, 2015
View reviewed changes

ghost reviewed Dec 9, 2015
View reviewed changes

bgrant0607 reviewed Jan 7, 2016
View reviewed changes

bgrant0607 added the e2e-not-required label Jan 7, 2016

bgrant0607 reviewed Jan 16, 2016
View reviewed changes

This was referenced Jan 22, 2016

Static pod Lifecycle Formalization (was: Decide fate of mirror pods) #16627

Open

Add proposal for node maintenance. #17393

Closed

bgrant0607 added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 22, 2016

Dedicated nodes, taints, and tolerations design doc.

14c2763

davidopp force-pushed the taints-tolerations branch from 566d138 to 14c2763 Compare January 24, 2016 07:06

k8s-github-robot removed lgtm "Looks good to me", indicates that a PR is ready to be merged. kind/design Categorizes issue or PR as related to design. labels Jan 24, 2016

davidopp added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 24, 2016

davidopp pushed a commit that referenced this pull request Jan 24, 2016

Merge pull request #18263 from davidopp/taints-tolerations

5abfb43

Dedicated nodes, taints, and tolerations design proposal

davidopp merged commit 5abfb43 into kubernetes:master Jan 24, 2016

davidopp mentioned this pull request May 8, 2016

Requirements for taints, tolerations, dedicated nodes to graduate to Beta and then v1 #25320

Closed

12 tasks

lukaszo mentioned this pull request Jul 19, 2016

Make NodeSpec.unschedulable a Taint #29178

Closed

davidopp mentioned this pull request Oct 1, 2016

Taints/tolerations kubernetes/enhancements#108

Closed

23 tasks

bgrant0607 mentioned this pull request Nov 21, 2016

Avoid hard-coding list of Node Conditions #36210

Merged

xingzhou pushed a commit to xingzhou/kubernetes that referenced this pull request Dec 15, 2016

Merge pull request kubernetes#18263 from davidopp/taints-tolerations

f943803

Dedicated nodes, taints, and tolerations design proposal

bgrant0607 mentioned this pull request Mar 31, 2017

kubeadm 1.6.0 (only 1.6.0!!) is broken due to unconfigured CNI making kubelet NotReady #43815

Closed

Conversation

davidopp commented Dec 6, 2015

Uh oh!

k8s-github-robot commented Dec 6, 2015

Uh oh!

k8s-bot commented Dec 6, 2015

Uh oh!

k8s-bot commented Dec 7, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mml commented Dec 7, 2015

Uh oh!

mml commented Dec 8, 2015

Uh oh!

davidopp commented Dec 8, 2015

Uh oh!

davidopp commented Dec 8, 2015

Uh oh!

davidopp commented Dec 8, 2015

Uh oh!

davidopp commented Dec 8, 2015

Uh oh!

ncdc commented Dec 9, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mml commented Dec 9, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

erictune commented Dec 9, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

erictune commented Dec 9, 2015

Uh oh!

erictune commented Dec 9, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bgrant0607 commented Jan 7, 2016

Uh oh!

bgrant0607 commented Jan 7, 2016

Uh oh!

davidopp commented Jan 13, 2016

Uh oh!

davidopp commented Jan 13, 2016

Uh oh!

k8s-bot commented Jan 13, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bgrant0607 commented Jan 22, 2016

Uh oh!

k8s-github-robot commented Jan 24, 2016

Uh oh!

k8s-bot commented Jan 24, 2016

Uh oh!

davidopp commented Jan 24, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone