Skip to content

Conversation

@kolyshkin
Copy link
Contributor

This is currently based on and includes #2458 and will be rebased once that one is merged. Please only review the last commit.

For some reason, runc systemd drivers (both v1 and v2) never set
systemd unit property named CPUQuotaPeriod (known as
CPUQuotaPeriodUSec on dbus and in systemctl show output).

Set it, and add a check to all the integration tests. The check is less
than trivial because, when not set, the value is shown as "infinity" but
when set to the same (default) value, shown as "100ms", so in case we
expect 100ms (period = 100000 us), we have to also check for
"infinity".

Fixes #2465

@kolyshkin
Copy link
Contributor Author

On Ubuntu 18.04 we have this:

Cannot set property CPUQuotaPeriodUSec, or unknown property.

Apparently systemd in ubuntu-18.04 did not have support for this parameter.

Looking at the systemd git, this was added in systemd v242 (systemd/systemd#9594).

I think we can check systemd version during controller initialization and set this conditionally.

@AkihiroSuda
Copy link
Member

#2458 is merged, please rebase

@kolyshkin kolyshkin force-pushed the systemd-cpu-quota-period branch from 671a84a to 7b5efe9 Compare June 15, 2020 19:44
@kolyshkin
Copy link
Contributor Author

Added systemd version detection and conditional setting.

I'm not quite sure why this one fails. @AkihiroSuda can you please take a look?

From https://travis-ci.org/github/opencontainers/runc/jobs/698672952#L4689:

make localrootlessintegration RUNC_USE_SYSTEMD=yes
...
[01] run rootless tests ... ()
...
not ok 79 update cgroup v1/v2 common limits
(in test file /vagrant/tests/integration/update.bats, line 41)
`[ "$status" -eq 0 ]' failed
...
time="2020-06-15T19:58:00Z" level=error msg="container_linux.go:353: starting container process caused: process_linux.go:326: applying cgroup configuration for process caused: error while starting unit "runc-cgroups-integration-test.scope" with properties [{Name:Description Value:"libcontainer container integration-test"} {Name:Slice Value:"machine.slice"} {Name:PIDs Value:@au [56928]} {Name:Delegate Value:true} {Name:MemoryAccounting Value:true} {Name:CPUAccounting Value:true} {Name:IOAccounting Value:true} {Name:DefaultDependencies Value:false} {Name:DevicePolicy Value:"strict"} {Name:DeviceAllow Value:@A(ss) []} {Name:DeviceAllow Value:["INVALID", "INVALID", "INVALID", "INVALID", "INVALID", "INVALID", "INVALID", "INVALID", "INVALID", "INVALID", "INVALID"]} {Name:MemoryMax Value:@t 33554432} {Name:MemoryLow Value:@t 25165824} {Name:CPUWeight Value:@t 4} {Name:CPUQuotaPeriodUSec Value:@t 1000000} {Name:CPUQuotaPerSecUSec Value:@t 500000} {Name:TasksAccounting Value:true} {Name:TasksMax Value:@t 20}]: Permission denied"

@kolyshkin
Copy link
Contributor Author

Added a commit to temporarily disable setting cpu quota period to check that the error above is caused by it.

@kolyshkin
Copy link
Contributor Author

Still have the same error after commenting out setting CPUQuotaPeriod. Hmm, is the master broken or I am not seeing something?

not ok 79 update cgroup v1/v2 common limits
(in test file /vagrant/tests/integration/update.bats, line 41)
`[ "$status" -eq 0 ]' failed
time="2020-06-15T21:54:35Z" level=error msg="container_linux.go:353: starting container process caused: process_linux.go:326: applying cgroup configuration for process caused: error while starting unit "runc-cgroups-integration-test.scope" with properties [{Name:Description Value:"libcontainer container integration-test"} {Name:Slice Value:"machine.slice"} {Name:PIDs Value:@au [56608]} {Name:Delegate Value:true} {Name:MemoryAccounting Value:true} {Name:CPUAccounting Value:true} {Name:IOAccounting Value:true} {Name:DefaultDependencies Value:false} {Name:DevicePolicy Value:"strict"} {Name:DeviceAllow Value:@A(ss) []} {Name:DeviceAllow Value:["INVALID", "INVALID", "INVALID", "INVALID", "INVALID", "INVALID", "INVALID", "INVALID", "INVALID", "INVALID", "INVALID"]} {Name:MemoryMax Value:@t 33554432} {Name:MemoryLow Value:@t 25165824} {Name:CPUWeight Value:@t 4} {Name:CPUQuotaPerSecUSec Value:@t 500000} {Name:TasksAccounting Value:true} {Name:TasksMax Value:@t 20}]: Permission denied"

@kolyshkin kolyshkin force-pushed the systemd-cpu-quota-period branch from 21f1c1f to a235fa9 Compare June 16, 2020 00:58
return nil
}

func systemdVersion() (int, error) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems heavy to call per runc invocation. Should we think about using build tags instead, so users can compile it for their version of systemd as needed? Another option may be a command-line flag. This applies to any other systemd feature detection as well.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's just always assume systemd >= 242, and fall back to 241-compatible mode on error

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems heavy to call per runc invocation

Does not look like heavy to me, and it's not per runc invocation, but only upon container creation or update (and in this case we talk to dbus already to create/update a unit, so there's no such thing as creating and new connection just to get the version).

Or am I missing something here?

The alternative is to exec systemctl and parse its output. I haven't checked which way is faster, but can look into it.

Let's just always assume systemd >= 242, and fall back to 241-compatible mode on error

I was thinking about it, but the logic would look really ugly (got EPERM? remove cpu quota period and retry). I would consider it if there is no other way to know if systemd supports the feature or not, but it seems

For some reason, runc systemd drivers (both v1 and v2) never set
systemd unit property named `CPUQuotaPeriod` (known as
`CPUQuotaPeriodUSec` on dbus and in `systemctl show` output).

Set it, and add a check to all the integration tests. The check is less
than trivial because, when not set, the value is shown as "infinity" but
when set to the same (default) value, shown as "100ms", so in case we
expect 100ms (period = 100000 us), we have to _also_ check for
"infinity".

[v2: add systemd version checks since CPUQuotaPeriod requires v242+]

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
@kolyshkin kolyshkin force-pushed the systemd-cpu-quota-period branch from a235fa9 to e751a16 Compare June 16, 2020 22:48
@kolyshkin
Copy link
Contributor Author

Modified the patch to explicitly reuse dbus connection (which was implicitly reused before). This also fixed the rootless failure.

@mrunalp mrunalp merged commit 406298f into opencontainers:master Jun 17, 2020
connDbus *systemdDbus.Conn
connErr error

versionOnce sync.Once
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that all this sync.Once stuff is currently redundant, as we only call systemdVersion() once. It was added in anticipation there will be other systemd parameters like this that need a version guard, and since we call systemdVersion() when and where needed (right from addCpuQuota) rather than in advance, the future calls might come from a similar "when and where needed" places.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

systemd drivers do not set CPUQuotaPeriodUSec

3 participants