[core] (cgroups 3/n) Creating CgroupManager to setup Ray's cgroup hierarchy and clean it up by israbbani · Pull Request #56186 · ray-project/ray

israbbani · 2025-09-03T01:37:44Z

This is the first PR introduces the CgroupManagerInterface. This will be used by the Raylet to manage the cgroup hierarchy. The implementation will be completed in subsequent PRs.

This PR stacks on #55063.

For more details about the resource isolation project see #54703.

The cgroup hierarchy for Ray will be:

    base_cgroup_path (e.g. /sys/fs/cgroup)
            |
    ray_node_<node_id>
      |           |
    system     application

The current implementation will only support

the cpu and memory controllers
cpu.weights and memory.min constraints.

I've signposted design decisions with comments in the code. Here's a summary:

CgroupManager::Create is a factory function. It is required because it can return a Status before creating an instance of CgroupManager if cgroupv2 is not setup correctly.
CgroupManager takes ownership of a CgroupDriverInterface for dependency injection.
CgroupManager uses ScopedCgroupOperations to register all side-effects to the cgroup hierarchy so graceful cleanup can apply them in reverse.

There are placeholders in the code for future work:

CgroupDriverInterface needs to implement a few more methods (DeleteCgroup, DisableController) to allow for full cleanup in CgroupManager.
CgroupManager needs additional tests for cleanup.
CgroupManagerInterface needs to expose APIs for moving processes into a cgroup.

to perform cgroup operations. Signed-off-by: irabbani <irabbani@anyscale.com>

Signed-off-by: irabbani <irabbani@anyscale.com>

instead of clone for older kernel headers < 5.7 (which is what we have in CI) Signed-off-by: irabbani <irabbani@anyscale.com>

Signed-off-by: irabbani <irabbani@anyscale.com>

Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com> Signed-off-by: Ibrahim Rabbani <israbbani@gmail.com>

Signed-off-by: irabbani <irabbani@anyscale.com>

…irabbani/cgroups-1

Signed-off-by: irabbani <irabbani@anyscale.com>

Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com> Signed-off-by: Ibrahim Rabbani <israbbani@gmail.com>

Signed-off-by: irabbani <irabbani@anyscale.com>

…irabbani/cgroups-1

Signed-off-by: irabbani <irabbani@anyscale.com>

fix CI. Signed-off-by: irabbani <irabbani@anyscale.com>

Signed-off-by: irabbani <irabbani@anyscale.com>

Signed-off-by: Ibrahim Rabbani <irabbani@anyscale.com>

israbbani · 2025-09-04T20:30:31Z

Test failures look unrelated (from serve)

edoakes · 2025-09-04T21:08:48Z

src/ray/common/cgroup2/BUILD.bazel

+    hdrs = [
+        "cgroup_manager_interface.h",
+    ],
+    target_compatible_with = [


src/ray/common/cgroup2/cgroup_manager.cc

edoakes · 2025-09-04T21:11:46Z

src/ray/common/cgroup2/cgroup_manager.cc

+  int64_t max_cpu_weight = supported_constraints_.at(kCPUWeightConstraint).Max();
+  int64_t application_cgroup_cpu_weight = max_cpu_weight - system_reserved_cpu_weight;
+
+  RAY_LOG(INFO) << absl::StrFormat(


seems a little noisy for an info level, but only once per raylet startup so should be ok

As an SRE, I found log lines like this one very useful when debugging issues. As a rule of thumb I think we should log the configuration each component starts up with (especially if it's only created once in the lifecycle of the application).

Sounds good. This one is nice that all of the info is logged in one place. We have some other startup logs that are noisy because we log each bit in a separate log line from different components.

edoakes · 2025-09-04T21:12:21Z

src/ray/common/cgroup2/cgroup_manager.cc

+#include "ray/common/cgroup2/scoped_cgroup_operation.h"
+#include "ray/common/status_or.h"
+
+namespace ray {


maybe should have a namespace cgroup ?

I'm open to trying it. I haven't really wrapped my head around what best practices should be around namespaces. I've added an item to #54703. I'll play around with it at the end.

edoakes · 2025-09-04T21:12:42Z

src/ray/common/cgroup2/cgroup_manager.h

+      3. move all processes from the system cgroup into the base cgroup.
+      4. delete the node, system, and application cgroups respectively.
+
+    Cleanup is best-effort. If any step fails, it will log a warning.


why's this? to avoid crashing before other cleanup can happen at a higher level?

Yep. I figured we'd want to attempt the rest of the graceful shutdown process of the raylet even if cgroup cleanup didn't succeed fully.

edoakes · 2025-09-04T21:14:33Z

src/ray/common/cgroup2/fake_cgroup_driver.h

+           enabled_controllers_ == other.enabled_controllers_;
+  }
+};
+class FakeCgroupDriver : public CgroupDriverInterface {


edoakes · 2025-09-04T21:16:55Z

minor comments; ping for merge

Signed-off-by: irabbani <irabbani@anyscale.com>

israbbani · 2025-09-05T16:47:22Z

@edoakes ready for merge. Thanks!

…anager. (#56246) This PR continues to implement the CgroupManager. CgroupManager will be used by the Raylet to manage the cgroup hierarchy. The implementation will be completed in subsequent PRs. This PR stacks on #56186. For more details about the resource isolation project see #54703. In this PR: * CgroupManager now bound checks constraints (e.g. cpu.weight is within [1,10000]. * CgroupDriver no longer bound checks constraints. --------- Signed-off-by: Ibrahim Rabbani <irabbani@anyscale.com> Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>

…rarchy and clean it up (ray-project#56186) Signed-off-by: irabbani <irabbani@anyscale.com> Signed-off-by: Ibrahim Rabbani <israbbani@gmail.com> Signed-off-by: Ibrahim Rabbani <irabbani@anyscale.com> Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com> Signed-off-by: sampan <sampan@anyscale.com>

…anager. (ray-project#56246) This PR continues to implement the CgroupManager. CgroupManager will be used by the Raylet to manage the cgroup hierarchy. The implementation will be completed in subsequent PRs. This PR stacks on ray-project#56186. For more details about the resource isolation project see ray-project#54703. In this PR: * CgroupManager now bound checks constraints (e.g. cpu.weight is within [1,10000]. * CgroupDriver no longer bound checks constraints. --------- Signed-off-by: Ibrahim Rabbani <irabbani@anyscale.com> Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com> Signed-off-by: sampan <sampan@anyscale.com>

…rarchy and clean it up (ray-project#56186) Signed-off-by: irabbani <irabbani@anyscale.com> Signed-off-by: Ibrahim Rabbani <israbbani@gmail.com> Signed-off-by: Ibrahim Rabbani <irabbani@anyscale.com> Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com> Signed-off-by: jugalshah291 <shah.jugal291@gmail.com>

…anager. (ray-project#56246) This PR continues to implement the CgroupManager. CgroupManager will be used by the Raylet to manage the cgroup hierarchy. The implementation will be completed in subsequent PRs. This PR stacks on ray-project#56186. For more details about the resource isolation project see ray-project#54703. In this PR: * CgroupManager now bound checks constraints (e.g. cpu.weight is within [1,10000]. * CgroupDriver no longer bound checks constraints. --------- Signed-off-by: Ibrahim Rabbani <irabbani@anyscale.com> Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com> Signed-off-by: jugalshah291 <shah.jugal291@gmail.com>

…rarchy and clean it up (ray-project#56186) Signed-off-by: irabbani <irabbani@anyscale.com> Signed-off-by: Ibrahim Rabbani <israbbani@gmail.com> Signed-off-by: Ibrahim Rabbani <irabbani@anyscale.com> Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com> Signed-off-by: yenhong.wong <yenhong.wong@grabtaxi.com>

…anager. (ray-project#56246) This PR continues to implement the CgroupManager. CgroupManager will be used by the Raylet to manage the cgroup hierarchy. The implementation will be completed in subsequent PRs. This PR stacks on ray-project#56186. For more details about the resource isolation project see ray-project#54703. In this PR: * CgroupManager now bound checks constraints (e.g. cpu.weight is within [1,10000]. * CgroupDriver no longer bound checks constraints. --------- Signed-off-by: Ibrahim Rabbani <irabbani@anyscale.com> Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com> Signed-off-by: yenhong.wong <yenhong.wong@grabtaxi.com>

…rarchy and clean it up (ray-project#56186) Signed-off-by: irabbani <irabbani@anyscale.com> Signed-off-by: Ibrahim Rabbani <israbbani@gmail.com> Signed-off-by: Ibrahim Rabbani <irabbani@anyscale.com> Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com> Signed-off-by: zac <zac@anyscale.com>

…anager. (ray-project#56246) This PR continues to implement the CgroupManager. CgroupManager will be used by the Raylet to manage the cgroup hierarchy. The implementation will be completed in subsequent PRs. This PR stacks on ray-project#56186. For more details about the resource isolation project see ray-project#54703. In this PR: * CgroupManager now bound checks constraints (e.g. cpu.weight is within [1,10000]. * CgroupDriver no longer bound checks constraints. --------- Signed-off-by: Ibrahim Rabbani <irabbani@anyscale.com> Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com> Signed-off-by: zac <zac@anyscale.com>

…rarchy and clean it up (#56186) Signed-off-by: irabbani <irabbani@anyscale.com> Signed-off-by: Ibrahim Rabbani <israbbani@gmail.com> Signed-off-by: Ibrahim Rabbani <irabbani@anyscale.com> Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com> Signed-off-by: Douglas Strodtman <douglas@anyscale.com>

…anager. (ray-project#56246) This PR continues to implement the CgroupManager. CgroupManager will be used by the Raylet to manage the cgroup hierarchy. The implementation will be completed in subsequent PRs. This PR stacks on ray-project#56186. For more details about the resource isolation project see ray-project#54703. In this PR: * CgroupManager now bound checks constraints (e.g. cpu.weight is within [1,10000]. * CgroupDriver no longer bound checks constraints. --------- Signed-off-by: Ibrahim Rabbani <irabbani@anyscale.com> Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com> Signed-off-by: Douglas Strodtman <douglas@anyscale.com>

…rarchy and clean it up (ray-project#56186) Signed-off-by: irabbani <irabbani@anyscale.com> Signed-off-by: Ibrahim Rabbani <israbbani@gmail.com> Signed-off-by: Ibrahim Rabbani <irabbani@anyscale.com> Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>

…anager. (ray-project#56246) This PR continues to implement the CgroupManager. CgroupManager will be used by the Raylet to manage the cgroup hierarchy. The implementation will be completed in subsequent PRs. This PR stacks on ray-project#56186. For more details about the resource isolation project see ray-project#54703. In this PR: * CgroupManager now bound checks constraints (e.g. cpu.weight is within [1,10000]. * CgroupDriver no longer bound checks constraints. --------- Signed-off-by: Ibrahim Rabbani <irabbani@anyscale.com> Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>

irabbani and others added 30 commits July 24, 2025 20:39

[core] (cgroups 1/n) Adding a sys/fs filesystem driver

05c4dbc

to perform cgroup operations. Signed-off-by: irabbani <irabbani@anyscale.com>

adding the copyright

645f9a0

Signed-off-by: irabbani <irabbani@anyscale.com>

Adding a fallback for creating processes inside cgroups with fork/exec

2bb2c5b

instead of clone for older kernel headers < 5.7 (which is what we have in CI) Signed-off-by: irabbani <irabbani@anyscale.com>

adding a pause in the tests to see what's up with the container

4793094

Signed-off-by: irabbani <irabbani@anyscale.com>

Update src/ray/common/cgroup2/cgroup_driver_interface.h

85d0ebf

Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com> Signed-off-by: Ibrahim Rabbani <israbbani@gmail.com>

Comments

3a5a020

Signed-off-by: irabbani <irabbani@anyscale.com>

Merge branch 'irabbani/cgroups-1' of github.com:ray-project/ray into …

68b0c93

…irabbani/cgroups-1

Putting the cgroupv2 tests into a separate target

f52354b

Signed-off-by: irabbani <irabbani@anyscale.com>

removing test sleep

148d04d

Signed-off-by: irabbani <irabbani@anyscale.com>

Removing a docstring

d3f8b79

Signed-off-by: irabbani <irabbani@anyscale.com>

enabling CI tests

d76ff15

Signed-off-by: irabbani <irabbani@anyscale.com>

fixing absl imports

2798ea5

Signed-off-by: irabbani <irabbani@anyscale.com>

commenting local

3fda505

Signed-off-by: irabbani <irabbani@anyscale.com>

doxygen formatting

9e1e931

Signed-off-by: irabbani <irabbani@anyscale.com>

Merge branch 'master' into irabbani/cgroups-1

f066f34

removing integration tests

e6b4926

Signed-off-by: irabbani <irabbani@anyscale.com>

final cleanup

f4e0cb2

Signed-off-by: irabbani <irabbani@anyscale.com>

iwyu

544ba83

Signed-off-by: irabbani <irabbani@anyscale.com>

Merge branch 'master' into irabbani/cgroups-1

669ba99

we cpplintin!

2e341d6

Signed-off-by: irabbani <irabbani@anyscale.com>

Update src/ray/common/cgroup2/sysfs_cgroup_driver.cc

9e46ce6

Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com> Signed-off-by: Ibrahim Rabbani <israbbani@gmail.com>

Apply suggestions from code review

7c745c5

Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com> Signed-off-by: Ibrahim Rabbani <israbbani@gmail.com>

bug

d7eb863

Signed-off-by: irabbani <irabbani@anyscale.com>

Merge branch 'irabbani/cgroups-1' of github.com:ray-project/ray into …

ff64534

…irabbani/cgroups-1

[core] Integration tests for SysFsCgroupDriver.

da4b475

Signed-off-by: irabbani <irabbani@anyscale.com>

Cleaning up cgroup_test_utils and attempting to

37e205f

fix CI. Signed-off-by: irabbani <irabbani@anyscale.com>

broken

7b83932

Signed-off-by: irabbani <irabbani@anyscale.com>

up

b911d25

Signed-off-by: irabbani <irabbani@anyscale.com>

upup

63506dc

Signed-off-by: irabbani <irabbani@anyscale.com>

Merge branch 'master' into irabbani/cgroups-2

e6f1ae9

Signed-off-by: Ibrahim Rabbani <irabbani@anyscale.com>

israbbani requested a review from a team as a code owner September 4, 2025 00:04

Merge branch 'irabbani/cgroups-2' into irabbani/cgroups-3

ca63baa

israbbani added the core Issues that should be addressed in Ray Core label Sep 4, 2025

Base automatically changed from irabbani/cgroups-2 to master September 4, 2025 04:54

Ibrahim Rabbani added 2 commits September 4, 2025 09:28

Merge branch 'master' into irabbani/cgroups-3

0fe9113

Signed-off-by: Ibrahim Rabbani <irabbani@anyscale.com>

Merge branch 'master' into irabbani/cgroups-3

ca83426

israbbani mentioned this pull request Sep 4, 2025

[core] (cgroups 4/n) adding constraint bounds checking to the CgroupManager. #56246

Merged

edoakes reviewed Sep 4, 2025

View reviewed changes

src/ray/common/cgroup2/BUILD.bazel

hdrs = [

"cgroup_manager_interface.h",

],

target_compatible_with = [

Copy link
Copy Markdown

Collaborator

edoakes Sep 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🫶

edoakes approved these changes Sep 4, 2025

View reviewed changes

Ibrahim Rabbani and others added 3 commits September 4, 2025 15:09

Merge branch 'master' into irabbani/cgroups-3

2457558

Addressing feedback.

b5f6c5e

Signed-off-by: irabbani <irabbani@anyscale.com>

Merge branch 'master' into irabbani/cgroups-3

03c731e

jjyao merged commit 5e85227 into master Sep 5, 2025
5 checks passed

jjyao deleted the irabbani/cgroups-3 branch September 5, 2025 17:57

Conversation

israbbani commented Sep 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

israbbani commented Sep 4, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

edoakes commented Sep 4, 2025

Uh oh!

israbbani commented Sep 5, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

israbbani commented Sep 3, 2025 •

edited

Loading