adding docs for node allocatable by vishh · Pull Request #2649 · kubernetes/website

vishh · 2017-03-01T00:12:24Z

User facing documentation for https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node-allocatable.md#node-allocatable-resources

cc @derekwaynecarr @dashpole @dchen1107

This change is

Signed-off-by: Vishnu kannan <vishnuk@google.com>

derekwaynecarr · 2017-03-03T21:31:50Z

+
+### Kube Reserved
+
+**Kubelet Flag**: `--kube-reserved=[cpu=100mi][,][memory=100Mi]`


derekwaynecarr · 2017-03-03T21:32:20Z

+### Kube Reserved
+
+**Kubelet Flag**: `--kube-reserved=[cpu=100mi][,][memory=100Mi]`
+**Kubelet Flag**: `--kube-reserved-cgroup=`/runtime.slice`


is this the name you are using in your images?

we should make clear that /runtime.slice is not the kubelet default value.

Good point on the defaults. I hope I have made the defaults clear this time around. PTAL

derekwaynecarr · 2017-03-03T21:34:45Z

+[This performance dashboard](http://node-perf-dash.k8s.io/#/builds) exposes `cpu` and `memory` usage profiles of `kubelet` and `docker engine` at multiple levels of pod density.
+[This blog post](http://blog.kubernetes.io/2016/11/visualize-kubelet-performance-with-node-dashboard.html) explains how the dashboard can be interpreted to come up with a suitable `kube-reserved` reservation.
+
+It is recommended that the kubernetes system daemons are placed under a top level control group (`system.slice` on systemd machines for example).


i think this text should be in system reserved section.

you should have text specific to kube daemons here..

derekwaynecarr · 2017-03-03T21:35:43Z

+### System Reserved
+
+**Kubelet Flag**: `--system-reserved=[cpu=100mi][,][memory=100Mi]`
+**Kubelet Flag**: `--system-reserved-cgroup=`/system.slice`


make clear this flag has no default.

we need to make clear that the kubelet doesnt create either of these two cgroups.

derekwaynecarr · 2017-03-03T21:36:41Z

+Evictions are supported for `memory` and `storage` only.
+By reserving some memory via `--eviction-hard` flag, the `kubelet` attempts to `evict` pods whenever memory availability on the node drops below the reserved value.
+Hypothetically, if system daemons did not exist on a node, pods cannot use more than `capacity - eviction-hard`.
+For this reason, resources reserved for evictions will not be available for pods.


to schedule against?

Scheduling is meant to be implicit since pods can be placed directly on nodes bypassing the scheduler

derekwaynecarr · 2017-03-03T21:37:27Z

+
+**Kubelet Flag**: `--enforce-node-allocatable=[pods][,][system-reserved][,][kube-reserved]`
+
+The scheduler will treat `Allocatable` as the available `capacity` for pods.


i would remove the use of will style phrasing in the document as we are describing the present in this doc.

The scheduler treats 'Allocatable'...

derekwaynecarr

comments throughout.

derekwaynecarr · 2017-03-03T21:47:03Z

+The scheduler will treat `Allocatable` as the available `capacity` for pods.
+
+`kubelet` will enforce `Allocatable` across pods by default.
+This enforcement is controlled by specifying `pods` value to the kubelet flag `--enforce-node-allocatable`.


note that this is the default value.

derekwaynecarr · 2017-03-03T21:47:56Z

+### System Reserved
+
+**Kubelet Flag**: `--system-reserved=[cpu=100mi][,][memory=100Mi]`
+**Kubelet Flag**: `--system-reserved-cgroup=`/system.slice`


we need to make clear that the kubelet doesnt create either of these two cgroups.

derekwaynecarr · 2017-03-03T21:49:25Z

+However, Kubelet cannot burst and use up all available Node resources if `kube-reserved` is enforced.
+
+Be extra careful while enforcing `system-reserved` reservation since it can lead to critical system services being CPU starved or OOM killed on the node.
+The recommendation is to enforce `system-reserved` only if a user has profiled their nodes exhaustively to come up with precise estimates.


and is confident in their ability to recover if any item in that group is oom_killed.

derekwaynecarr · 2017-03-03T21:49:45Z

+
+* To begin with enforce `Allocatable` on `pods`.
+* Once adequate monitoring and alerting is in place to track kube system daemons, attempt to enforce `kube-reserved` based on usage heuristics.
+* If aboslutely necessary, enforce `system-reserved` over time.


typo on absolutely

vishh

@derekwaynecarr PTAL

Signed-off-by: Vishnu kannan <vishnuk@google.com>

vishh · 2017-03-09T19:45:52Z

@kubernetes/sig-docs-maintainers this PR is meant for v1.6 Can I get a docs review?

derekwaynecarr

one typo, and maybe some more clarifying text.

while not needed on this pr, i feel like this should be linked to from somewhere centrally on how to administer a kubernetes node. maybe a doc team member can assist there.

derekwaynecarr · 2017-03-10T19:14:33Z

+
+Memory pressure at the node level leads to System OOMs which affects the entire node and all pods running on it.
+Nodes can go offline temporarily until memory has been reclaimed.
+To avoid (or reduce the probabilty) system OOMs kubelet provides [`Out of Resource`](./out-of-resource.md) management.


typo: probability

derekwaynecarr · 2017-03-10T19:16:04Z

+The scheduler treats `Allocatable` as the available `capacity` for pods.
+
+`kubelet` enforce `Allocatable` across pods by default.
+This enforcement is controlled by specifying `pods` value to the kubelet flag `--enforce-node-allocatable`.


maybe explain what enforcement means? for example, by enforcing at this level, we ensure pods cannot consume more memory and cpu time than allocated?

Signed-off-by: Vishnu kannan <vishnuk@google.com>

vishh · 2017-03-14T22:00:07Z

@derekwaynecarr PTAL

derekwaynecarr · 2017-03-14T22:27:10Z

/lgtm

adding docs for node allocatable

316cce4

Signed-off-by: Vishnu kannan <vishnuk@google.com>

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Mar 1, 2017

vishh added this to the 1.6 milestone Mar 1, 2017

k8s-github-robot assigned derekwaynecarr Mar 1, 2017

vishh mentioned this pull request Mar 2, 2017

enable cgroups tiers and node allocatable enforcement on pods by default. kubernetes/kubernetes#42350

Merged

chenopis changed the base branch from master to release-1.6 March 2, 2017 17:50

chenopis added the Needs Tech Review label Mar 2, 2017

chenopis requested a review from derekwaynecarr March 3, 2017 08:23

derekwaynecarr reviewed Mar 3, 2017

View reviewed changes

derekwaynecarr mentioned this pull request Mar 3, 2017

Add support for pod and qos level cgroups kubernetes/enhancements#126

Closed

23 tasks

derekwaynecarr suggested changes Mar 3, 2017

View reviewed changes

chenopis added Tech Review: Open Issues and removed Needs Tech Review labels Mar 3, 2017

vishh commented Mar 9, 2017

View reviewed changes

vishh force-pushed the node-allocatable branch 2 times, most recently from 94c3ef3 to a617a42 Compare March 9, 2017 19:44

fix defaults and formatting on node-allocatable admin doc

bcd5e12

Signed-off-by: Vishnu kannan <vishnuk@google.com>

vishh force-pushed the node-allocatable branch from a617a42 to bcd5e12 Compare March 9, 2017 19:45

derekwaynecarr suggested changes Mar 10, 2017

View reviewed changes

define enforcement in node-allocatable doc

6e7b06d

Signed-off-by: Vishnu kannan <vishnuk@google.com>

k8s-ci-robot added the lgtm label Mar 14, 2017

derekwaynecarr added the Tech Review LGTM label Mar 14, 2017

derekwaynecarr removed the Tech Review: Open Issues label Mar 14, 2017

chenopis merged commit d4383a4 into kubernetes:release-1.6 Mar 15, 2017


		### Kube Reserved

		Kubelet Flag: `--kube-reserved=[cpu=100mi][,][memory=100Mi]`


		Kubelet Flag: `--enforce-node-allocatable=[pods][,][system-reserved][,][kube-reserved]`

		The scheduler will treat `Allocatable` as the available `capacity` for pods.

Conversation

vishh commented Mar 1, 2017 • edited by spxtr Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

derekwaynecarr left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vishh left a comment

Choose a reason for hiding this comment

Uh oh!

vishh commented Mar 9, 2017

Uh oh!

derekwaynecarr left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vishh commented Mar 14, 2017

Uh oh!

derekwaynecarr commented Mar 14, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

vishh commented Mar 1, 2017 •

edited by spxtr

Loading