Implementing support for --cpu-rt-period and --cpu-rt-runtime #23430

erikstmartin · 2016-06-10T13:07:38Z

- What I did
Added support for two new flags --cpu-rt-period and --cpu-rt-runtime combined with either the --privileged flag or --cap-add=sys_nice this allows processes within a container to prioritize themselves as real-time when CONFIG_RT_GROUP_SCHED is enabled in the kernel. See #22380. When this option is enabled each docker container's cgroup has to have cpu.rt_runtime_us set in order for the container to be able to prioritize its threads.

- How I did it

Support for changing the cgroup value already existed in runc, I updated all the necessary container configs to pass these values down to runc. I also implemented some sanity checks in the daemon.

- How to verify it
Ensure CONFIG_RT_GROUP_SCHED is enabled in the kernel (standard CentOS 7 install has this enabled). You can verify it's enabled by running
zcat /proc/config.gz | grep CONFIG_RT_GROUP_SCHED or by checking for the existence of /sys/fs/cgroup/cpu.rt_runtime_us

You will also need to ensure that the system and all parent cgroups have values set for the period and runtime as this limits what the children can be set to. If you're familiar with these values you can set them appropriately, sane defaults would be rt_period_us = 1000000 and rt_runtime_us = 950000, they only need to be set once on the system.

sysctl kernel.sched_rt_period_us
sysctl kernel.sched_rt_runtime_us

cat /sys/fs/cgroup/cpu.rt_period_us
cat /sys/fs/cgroup/cpu.rt_runtime_us

cat /sys/fs/cgroup/docker/cpu.rt_period_us
cat /sys/fs/cgroup/docker/cpu.rt_runtime_us

Update: You can now set the values for the parent cgroups with a new set of daemon flags. The system value will still need to be set using sysctl.
docker daemon --cpu-rt-runtime=950000

Now you have a system you can test on. To validate this works start a container with the new flags (we'll add a ulimit just to test that works as expected too)

docker run --it --cpu-rt-runtime=95000 --ulimit rtprio=99 --cap-add=sys_nice debian:jessie

If you receive an error like this, it's because your parent cgroups dont have rt_runtime_us set, or it's a lower value than you're trying to set for your container. Also ensure they are set for your container /sys/fs/cgroup/docker/<container id>/cpu.rt_runtime_us (after the container has been started). if you're testing this inside a container.

docker: Error response from daemon: oci runtime error: process_linux.go:286: setting cgroup config for ready process caused "failed to write 950000 to cpu.rt_runtime_us: write /sys/fs/cgroup/cpu,cpuacct/docker/5024d2730a83e86efd9042ea99972d582c66830f4a3aa21c747d1c25c824eb75/cpu.rt_runtime_us: invalid argument".

Once inside the container you can attempt to change the priority of your shell
chrt -r -p 99 1

Then validate by calling
chrt -p 1

root@c1d2f4b7d579:/# chrt -r -p 99 1
root@c1d2f4b7d579:/# chrt -p 1
pid 1's current scheduling policy: SCHED_RR
pid 1's current scheduling priority: 99

You can also test the following error conditions.

Passing either flag without having the sys_nice capability or being privileged results in error (no point setting your runtime if you dont have permission to change your priority)
Passing either flag on a host with the kernel option disabled results in error
Passing a runtime greater than the period results in an error

- Description for the changelog
Support for --cpu-rt-period and --cpu-rt-runtime flags, allowing containers to run real-time threads when CONFIG_RT_GROUP_SCHED is enabled in the kernel.

- A picture of a cute animal (not mandatory but encouraged)

justincormack · 2016-06-10T14:50:34Z

Thanks for working on this!

justincormack · 2016-06-10T14:57:17Z

runconfig/hostconfig_unix.go

I don't think I would check this, there are potentially other ways in which we could get capabilities, so I would just check the return codes when you actually try to change stuff and try to give a helpful message if it fails.

Remove just the capabilities check? or all the param sanity checking?

Also you wont see any errors until you make the syscall to change your priority, which would likely just be a permissions error and not overly helpful.

The capabilities check, it is a bit ugly and encodes a lot of presumed information about how capabilities are assigned which could change in future, and we don't do it anywhere else. EPERM in the syscall is pretty clear and you can provide a more helpful message there.

I got overly ambition trying to save people from themselves. I'll get this removed.

erikstmartin · 2016-06-10T15:00:18Z

No problem, sorry for the crazy delay, been insanely caught up on other tasks.

Let me know if you have any questions/problems trying to test this. I admit it's a lot of steps to get setup to test if you don't already have this kernel param enabled, then the Docker inception thing, having to make sure the container you're testing inside of has rt_runtime. You almost need this feature to test this feature :)

erikstmartin · 2016-06-10T17:14:47Z

I've removed the capabilities check and associated unit tests, PTAL.

erikstmartin · 2016-06-13T13:48:38Z

CI Failures are from the manual changes within vendor/ I was advised to do that for now here so that everything could be reviewed in one place.

Once docker-archive-public/docker.engine-api#262 is merged, this PR can be updated to pull in upstream and that failure will disappear.

thaJeztah · 2016-06-13T20:26:50Z

@erikstmartin yes, don't worry about the "failing-ci" label for now, it's just that we know it's failing (in this case due to the "vendor" CI). Should not be an issue while in design-review 👍

erikstmartin · 2016-07-20T18:52:42Z

I've updated this PR to include a --cpu-rt-runtime flag to the daemon. The daemon needs to be able to set the cgroup value otherwise a sysadmin will need to set the value after the first container is started (after every reboot)

jalaziz · 2016-07-21T09:15:28Z

I haven't had a chance to test this, but I believe the daemon flag won't actually work correctly with newer systemd versions. Until opencontainers/runc#931 is fixed, the cpu cgroup path ends up being /sys/fs/cgroup/cpu/init.scope/system.slice. In order for everything to work correctly, rt_runtime and rt_period have to be recursively set down the path.

erikstmartin · 2016-07-22T15:33:06Z

@jalaziz I'll double check, but i'm pretty sure I was developing this against 230.

jalaziz · 2016-07-25T00:29:30Z

@erikstmartin For reference, we're running CoreOS with systemd v229. However, I believe the problem exists on other OSes with newer systemd versions if you use --exec-opt native.cgroupdriver=systemd (I'm sure you already knew this).

I'm not sure if the cgroupParent is correct for the CPU subsystem when you call initCgroupsPath. Even if it is, though, the testing I've done seems to indicate you'd have to recursively set the runtime and period settings (e.g. for init.scope, then for init.scope/system.slice).

erikstmartin · 2016-08-08T15:55:46Z

@jalaziz You're completely correct. I dont think I tried it using the systemd cgroup driver on that machine.

erikstmartin · 2016-08-17T01:25:31Z

Latest push should handle climbing the cgroup tree and also account for init.scope in newer versions of systemd

cpuguy83 · 2016-10-26T23:35:54Z

🎉
Thanks @erikstmartin!

Pull request moby#27745 added support for the client to talk to older versions of the daemon. Various flags were added to docker 1.13 that are not compatible with older daemons. This PR adds annotations to those flags, so that they are automatically hidden if the daemon is older than docker 1.13 (API 1.25). Not all new flags affect the API (some are client-side only). The following PR's added new flags to docker 1.13 that affect the API; - moby#23430 added `--cpu-rt-period`and `--cpu-rt-runtime` - moby#27800 / moby#25317 added `--group` / `--group-add` / `--group-rm` - moby#27702 added `--network` to `docker build` - moby#25962 added `--attachable` to `docker network create` - moby#27998 added `--compose-file` to `docker stack deploy` - moby#22566 added `--stop-timeout` to `docker run` and `docker create` - moby#26061 added `--init` to `docker run` and `docker create` - moby#26941 added `--init-path` to `docker run` and `docker create` - moby#27958 added `--cpus` on `docker run` / `docker create` - moby#27567 added `--dns`, `--dns-opt`, and `--dns-search` to `docker service create` - moby#27596 added `--force` to `docker service update` - moby#27857 added `--hostname` to `docker service create` - moby#28031 added `--hosts`, `--host-add` / `--host-rm` to `docker service create` and `docker service update` - moby#28076 added `--tty` on `docker service create` / `docker service update` - moby#26421 added `--update-max-failure-ratio`, `--update-monitor` and `--rollback` on `docker service update` - moby#27369 added `--health-cmd`, `--health-interval`, `--health-retries`, `--health-timeout` and `--no-healthcheck` options to `docker service create` and `docker service update` Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

huster-hh · 2017-03-01T03:50:51Z

@erikstmartin thank you for reading my question.
i meet the same error you said.i submit my question in #31411
you said 'If you receive an error like this, it's because your parent cgroups dont have rt_runtime_us set, or it's a lower value than you're trying to set for your container.'
however i can not change docker cgroup 's rt_runtime_us to more than 0.
can you tell me how you solve this error??
Thank you very much !!!Thank you !

erikstmartin · 2017-03-14T20:09:11Z

@IT-huang This is going to be dependent on what the parent cgroup of docker is, is it user.slice, etc. You need to make sure that the parents also have rt_runtime allocated. This could also be at the system level of the host itself. Check sysctl kernel.sched_rt_runtime_us and sysctl kernel.sched_rt_period_us

cpu.rt_runtime PR moby#23430 introduced a couple more flags including `--cpu-rt-runtime` to the docker daemon. It appears recent changes or merge issues may have broken this. It currently does not take the cgroup mount point into account when determining the cgroup files to write values to. This breaks docker setting its own `cpu.rt_runtime` for the daemon. This also means containers aren't able to set theirs. Also, the cgroups.FindCgroupMountpointAndRoot returns back a mount point that includes the cgroup of the currently running container when docker is run inside a docker container. this breaks the `--cpu-rt-runtime` flag when running docker in docker. A fix has been placed here, but potentially could be pulled up into libcontainer if this is a better place for it. Signed-off-by: Erik St. Martin <alakriti@gmail.com>

Pull request moby/moby#27745 added support for the client to talk to older versions of the daemon. Various flags were added to docker 1.13 that are not compatible with older daemons. This PR adds annotations to those flags, so that they are automatically hidden if the daemon is older than docker 1.13 (API 1.25). Not all new flags affect the API (some are client-side only). The following PR's added new flags to docker 1.13 that affect the API; - moby/moby#23430 added `--cpu-rt-period`and `--cpu-rt-runtime` - moby/moby#27800 / moby/moby#25317 added `--group` / `--group-add` / `--group-rm` - moby/moby#27702 added `--network` to `docker build` - moby/moby#25962 added `--attachable` to `docker network create` - moby/moby#27998 added `--compose-file` to `docker stack deploy` - moby/moby#22566 added `--stop-timeout` to `docker run` and `docker create` - moby/moby#26061 added `--init` to `docker run` and `docker create` - moby/moby#26941 added `--init-path` to `docker run` and `docker create` - moby/moby#27958 added `--cpus` on `docker run` / `docker create` - moby/moby#27567 added `--dns`, `--dns-opt`, and `--dns-search` to `docker service create` - moby/moby#27596 added `--force` to `docker service update` - moby/moby#27857 added `--hostname` to `docker service create` - moby/moby#28031 added `--hosts`, `--host-add` / `--host-rm` to `docker service create` and `docker service update` - moby/moby#28076 added `--tty` on `docker service create` / `docker service update` - moby/moby#26421 added `--update-max-failure-ratio`, `--update-monitor` and `--rollback` on `docker service update` - moby/moby#27369 added `--health-cmd`, `--health-interval`, `--health-retries`, `--health-timeout` and `--no-healthcheck` options to `docker service create` and `docker service update` Signed-off-by: Sebastiaan van Stijn <github@gone.nl> Upstream-commit: 5d2722f83db9e301c6dcbe1c562c2051a52905db Component: cli

This adds bash completion for moby/moby#23430. Signed-off-by: Harald Albers <github@albersweb.de>

This adds bash completion for moby/moby#23430. Signed-off-by: Harald Albers <github@albersweb.de> Upstream-commit: 74a5d1af86a3d0898dda79bb13282f1b039b015c Component: cli

This adds bash completion for moby/moby#23430. Signed-off-by: Harald Albers <github@albersweb.de>

These options configure the parent cgroup, not the default for containers, nor the daemon itself, so adding that information to the flag description to make this slightly more clear. relates to 56f77d5 (moby#23430) which implemented these flags. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

These options configure the parent cgroup, not the default for containers, nor the daemon itself, so adding that information to the flag description to make this slightly more clear. relates to 56f77d5 (moby#23430) which implemented these flags. Signed-off-by: Sebastiaan van Stijn <github@gone.nl> Signed-off-by: zach <Zachary.Joyner@linux.com>

kolyshkin · 2020-05-22T22:50:12Z

pkg/sysinfo/sysinfo_linux.go

+	}
+
+	cpuRealtimeRuntime := cgroupEnabled(mountPoint, "cpu.rt_runtime_us")
+	if !quiet && !cpuRealtimeRuntime {


It does not make sense to check these two files separately -- this is a single kernel feature, so these two are either both present or not (and if one present and the other is not this is some kind of a gross kernel error).

I am trying to fix this in #41014

kolyshkin · 2020-05-22T22:51:28Z

daemon/oci_linux.go

 	s.Linux.Resources.OOMScoreAdj = &c.HostConfig.OomScoreAdj
 	s.Linux.Sysctl = c.HostConfig.Sysctls
+
+	p := *s.Linux.CgroupsPath


there is no need to do all of the below in case CPU RT parameters are not set, or the kernel does not support CPU RT sched.

kolyshkin · 2020-05-22T22:52:18Z

daemon/daemon_unix.go

+
+	daemon.initCgroupsPath(filepath.Dir(path))
+
+	_, root, err := cgroups.FindCgroupMountpointAndRoot("cpu")


this call is very expensive and ideally should only be done once.

kolyshkin · 2020-05-22T22:58:14Z

daemon/daemon_unix.go

+	}
+
+	path = filepath.Join(root, path)
+	sysinfo := sysinfo.New(false)


This is very expensive way to check whether CPU RT scheduler is supported, and it's also performed way too late in the code, and also multiple times :(

kolyshkin · 2020-05-22T22:59:08Z

Trying to address some of the issues in #41014 and #41016

GordonTheTurtle added the status/0-triage label Jun 10, 2016

erikstmartin force-pushed the realtime-threads branch from bbbc519 to 8612a12 Compare June 10, 2016 14:17

justincormack added status/1-design-review and removed status/0-triage labels Jun 10, 2016

justincormack reviewed Jun 10, 2016
View reviewed changes

erikstmartin force-pushed the realtime-threads branch from 8612a12 to 8b0b462 Compare June 10, 2016 16:28

justincormack mentioned this pull request Jun 10, 2016

Expose the CPU cgroup realtime options to the run CLI #13983

Closed

erikstmartin mentioned this pull request Jun 10, 2016

Setting sched_setscheduler for SCHED_FIFO in a container. Getting EPERM error. #22380

Closed

thaJeztah added the status/failing-ci Indicates that the PR in its current state fails the test suite label Jun 13, 2016

GordonTheTurtle added the dco/no Automatically set by a bot when one of the commits lacks proper signature label Jun 13, 2016

erikstmartin force-pushed the realtime-threads branch from e936503 to 4aa30da Compare June 13, 2016 20:05

GordonTheTurtle removed the dco/no Automatically set by a bot when one of the commits lacks proper signature label Jun 13, 2016

erikstmartin mentioned this pull request Jul 19, 2016

Accounting and feasibility for opaque integer resources in the API server and scheduler. kubernetes/kubernetes#28312

Closed

erikstmartin force-pushed the realtime-threads branch from 4aa30da to 7f24a14 Compare July 20, 2016 18:09

erikstmartin force-pushed the realtime-threads branch 2 times, most recently from 3a39af7 to fe2f46e Compare July 20, 2016 19:58

erikstmartin force-pushed the realtime-threads branch 3 times, most recently from 269ee59 to 1b749b4 Compare August 16, 2016 19:51

This was referenced Nov 15, 2016

Document support for --cpu-rt-period and --cpu-rt-runtime docker/docs#583

Closed

Document new CPU scheduling flags docker/docs#586

Merged

thaJeztah mentioned this pull request Jan 3, 2017

do not create init-dir if not needed #29846

Merged

cpuguy83 mentioned this pull request Jan 14, 2017

Not able to set priority inside docker container #30122

Closed

thaJeztah mentioned this pull request Jan 16, 2017

Add version annotation to various flags added in 1.13 #30186

Merged

erikstmartin mentioned this pull request Mar 14, 2017

Add mount point to cgroup root when initializing cgroup paths for cpu.rt_runtime #31827

Merged

albers added a commit to albers/docker-cli that referenced this pull request Jul 3, 2017

Add bash completion for dockerd --cpu-rt-(period|runtime)

74a5d1a

This adds bash completion for moby/moby#23430. Signed-off-by: Harald Albers <github@albersweb.de>

alshabib pushed a commit to alshabib/cli that referenced this pull request Aug 1, 2017

Add bash completion for dockerd --cpu-rt-(period|runtime)

9f1fc71

This adds bash completion for moby/moby#23430. Signed-off-by: Harald Albers <github@albersweb.de>

thaJeztah mentioned this pull request Jun 21, 2019

Improve description for cpu-rt-period,runtime daemon flags #39394

Merged

thaJeztah mentioned this pull request Aug 9, 2019

daemon.initCgroupsPath fix unhandled error in recursion #39704

Closed

kolyshkin reviewed May 22, 2020

View reviewed changes

thaJeztah mentioned this pull request May 25, 2020

refactor some CPU RT and CFS code #41016

Merged

Dalarialus mentioned this pull request Jun 29, 2021

Possible to add dependency to clear console warning in Valheim? MitchTalmadge/AMP-dockerized#75

Closed

kolyshkin mentioned this pull request Oct 23, 2024

daemon: use OwnCgroupPath in withCgroups #48730

Merged


		daemon.initCgroupsPath(filepath.Dir(path))

		_, root, err := cgroups.FindCgroupMountpointAndRoot("cpu")

Implementing support for --cpu-rt-period and --cpu-rt-runtime #23430

Implementing support for --cpu-rt-period and --cpu-rt-runtime #23430

Uh oh!

Conversation

erikstmartin commented Jun 10, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

justincormack commented Jun 10, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

erikstmartin commented Jun 10, 2016

Uh oh!

erikstmartin commented Jun 10, 2016

Uh oh!

erikstmartin commented Jun 13, 2016

Uh oh!

thaJeztah commented Jun 13, 2016

Uh oh!

erikstmartin commented Jul 20, 2016

Uh oh!

jalaziz commented Jul 21, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

erikstmartin commented Jul 22, 2016

Uh oh!

jalaziz commented Jul 25, 2016

Uh oh!

erikstmartin commented Aug 8, 2016

Uh oh!

erikstmartin commented Aug 17, 2016

Uh oh!

cpuguy83 commented Oct 26, 2016

Uh oh!

huster-hh commented Mar 1, 2017

Uh oh!

erikstmartin commented Mar 14, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kolyshkin commented May 22, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

12 participants

erikstmartin commented Jun 10, 2016 •

edited

Loading

jalaziz commented Jul 21, 2016 •

edited

Loading