[core] (cgroups 19/n) Allow fractions when getting the number of CPUs to calculate weights by israbbani · Pull Request #57800 · ray-project/ray

israbbani · 2025-10-16T17:37:11Z

This PR stacks on #57776.

For more details about the resource isolation project see #54703.

When Ray calculates the number of cpus available on the machine, it checks to see if it's running in a container. However, it truncates the number of cpus.

In this PR,

If the number of CPUs on the machine is <= DEFAULT_MIN_SYSTEM_RESERVED_CPU_CORES, then raise a ValueError. Previously, this was < DEFAULT_MIN_SYSTEM_RESERVED_CPU_CORES.
Return fractional CPUs from ray._private.utils.get_num_cpus if an optional parameter is set to True.

cpus available on the machine. This will prevent us from rounding down when running in a container that has cpu.max set. Signed-off-by: irabbani <israbbani@gmail.com>

israbbani · 2025-10-16T20:05:48Z

Tested on Anyscale w/ a 2 core machine. Works with default parameters now.

lscpu | grep "CPU(s)"
CPU(s):                                  2

cat /sys/fs/cgroup/ray-node_e06784dc2316943d0918f0257d1d2cb24d24605ea9022d745cb23fe4/user/cpu.weight
4445

cat /sys/fs/cgroup/ray-node_e06784dc2316943d0918f0257d1d2cb24d24605ea9022d745cb23fe4/system/cpu.weight
5555

From the logs

(base) ray@ip-10-0-251-150:~/default$ grep "CgroupManager" /tmp/ray/session_latest/logs/raylet.out
{"asctime":"2025-10-16 12:49:53,974","levelname":"I","message":"Initializing CgroupManager at base cgroup at '/sys/fs/cgroup'. Ray's cgroup hierarchy will under the node cgroup at '/sys/fs/cgroup/ray-node_e06784dc2316943d0918f0257d1d2cb24d24605ea9022d745cb23fe4' with [memory, cpu] controllers enabled. The system cgroup at '/sys/fs/cgroup/ray-node_e06784dc2316943d0918f0257d1d2cb24d24605ea9022d745cb23fe4/system' will have [memory] controllers enabled with [cpu.weight=5555, memory.min=5946149682] constraints. The user cgroup '/sys/fs/cgroup/ray-node_e06784dc2316943d0918f0257d1d2cb24d24605ea9022d745cb23fe4/user' will have no controllers enabled with [cpu.weight=4445] constraints. The user cgroup will contain the [/sys/fs/cgroup/ray-node_e06784dc2316943d0918f0257d1d2cb24d24605ea9022d745cb23fe4/user/workers, /sys/fs/cgroup/ray-node_e06784dc2316943d0918f0257d1d2cb24d24605ea9022d745cb23fe4/user/non-ray] cgroups.","component":"raylet","filename":"cgroup_manager.cc","lineno":212}

edoakes

add comment in followup

edoakes · 2025-10-16T20:07:44Z

python/ray/_private/resource_isolation_config.py


        """
-        available_system_cpus = utils.get_num_cpus()
+        available_system_cpus = utils.get_num_cpus(truncate=False)


should leave a comment for why we don't truncate

) For more details about the resource isolation project see #54703. This PR moves the driver into the workers cgroup when it registers with the NodeManager. Also updates the tests to reflect this. This now includes changes from #57800. --------- Signed-off-by: irabbani <israbbani@gmail.com> Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>

…-project#57776) For more details about the resource isolation project see ray-project#54703. This PR moves the driver into the workers cgroup when it registers with the NodeManager. Also updates the tests to reflect this. This now includes changes from ray-project#57800. --------- Signed-off-by: irabbani <israbbani@gmail.com> Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>

…-project#57776) For more details about the resource isolation project see ray-project#54703. This PR moves the driver into the workers cgroup when it registers with the NodeManager. Also updates the tests to reflect this. This now includes changes from ray-project#57800. --------- Signed-off-by: irabbani <israbbani@gmail.com> Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com> Signed-off-by: xgui <xgui@anyscale.com>

) For more details about the resource isolation project see #54703. This PR moves the driver into the workers cgroup when it registers with the NodeManager. Also updates the tests to reflect this. This now includes changes from #57800. --------- Signed-off-by: irabbani <israbbani@gmail.com> Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

…-project#57776) For more details about the resource isolation project see ray-project#54703. This PR moves the driver into the workers cgroup when it registers with the NodeManager. Also updates the tests to reflect this. This now includes changes from ray-project#57800. --------- Signed-off-by: irabbani <israbbani@gmail.com> Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>

…-project#57776) For more details about the resource isolation project see ray-project#54703. This PR moves the driver into the workers cgroup when it registers with the NodeManager. Also updates the tests to reflect this. This now includes changes from ray-project#57800. --------- Signed-off-by: irabbani <israbbani@gmail.com> Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com> Signed-off-by: Aydin Abiar <aydin@anyscale.com>

…-project#57776) For more details about the resource isolation project see ray-project#54703. This PR moves the driver into the workers cgroup when it registers with the NodeManager. Also updates the tests to reflect this. This now includes changes from ray-project#57800. --------- Signed-off-by: irabbani <israbbani@gmail.com> Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com> Signed-off-by: Future-Outlier <eric901201@gmail.com>

[core] (cgroups 19/n) Use fractional values when getting the number of

0b8ed24

cpus available on the machine. This will prevent us from rounding down when running in a container that has cpu.max set. Signed-off-by: irabbani <israbbani@gmail.com>

israbbani changed the base branch from master to irabbani/cgroups-18 October 16, 2025 17:37

israbbani added core Issues that should be addressed in Ray Core go add ONLY when ready to merge, run all tests labels Oct 16, 2025

Ibrahim Rabbani added 2 commits October 16, 2025 10:38

Merge branch 'irabbani/cgroups-18' into irabbani/cgroups-19

b34b095

Merge branch 'irabbani/cgroups-18' into irabbani/cgroups-19

a9abf64

israbbani marked this pull request as ready for review October 16, 2025 18:02

israbbani requested a review from a team as a code owner October 16, 2025 18:02

edoakes approved these changes Oct 16, 2025

View reviewed changes

edoakes merged commit 40a027e into irabbani/cgroups-18 Oct 16, 2025
5 checks passed

edoakes deleted the irabbani/cgroups-19 branch October 16, 2025 20:07

israbbani mentioned this pull request Oct 16, 2025

[core] (cgroups 18/n) Moving the driver into the workers cgroup. #57776

Merged

israbbani mentioned this pull request Oct 17, 2025

[Core] Resource Isolation Milestone 1 Implementation Tracker #54703

Open

15 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[core] (cgroups 19/n) Allow fractions when getting the number of CPUs to calculate weights#57800

[core] (cgroups 19/n) Allow fractions when getting the number of CPUs to calculate weights#57800
edoakes merged 3 commits intoirabbani/cgroups-18from
irabbani/cgroups-19

israbbani commented Oct 16, 2025 •

edited

Loading

Uh oh!

israbbani commented Oct 16, 2025

Uh oh!

edoakes left a comment

Uh oh!

edoakes Oct 16, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

israbbani commented Oct 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

israbbani commented Oct 16, 2025

Uh oh!

edoakes left a comment

Choose a reason for hiding this comment

Uh oh!

edoakes Oct 16, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

israbbani commented Oct 16, 2025 •

edited

Loading