…r to move processes into system cgroup (#56446)

This PR stacks on #56352 .

For more details about the resource isolation project see
#54703.

This PR the following functions to move a process into the system
cgroup:
* CgroupManagerInterface::AddProcessToSystemCgroup
* CgroupDriverInterface::AddProcessToCgroup

I've also added integration tests for SysFsCgroupDriver and unit tests
for CgroupManager.

Let me explain how these APIs will be used. In the next PR, the raylet
will
* be passed a list of pids of system processes that are started before
the raylet starts and need to be moved into the system cgroup (e.g.
gcs_server)
* call CgroupManagerInterface::AddProcessToSystemCgroup for each of
these pids to move them into the system cgroup.

---------

Signed-off-by: Ibrahim Rabbani <irabbani@anyscale.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>

ray-project#56352)

This PR stacks on ray-project#56297.

For more details about the resource isolation project see
ray-project#54703.

This PR wires the resource isolation config (introduced here ray-project#51865)
from ray cli (ray start) and the ray sdk (ray.init) into the raylet.
Notable changes include:
1. A separate python test target for running test_resource_isolation
related unit and integration tests. This unifies all cgroup related
tests under one buildkite target and removes the need for `--except-tags
cgroup` everywhere else.
2. Modification to the cgroup hierarchy. This was an oversight on my
part. The "no internal processes" constraint says that a non-root cgroup
can either have controllers enabled or have processes. Therefore, the
new hierarchy looks like:
```
  //      base_cgroup_path (e.g. /sys/fs/cgroup)
  //             |
  //     ray_node_<node_id>
  //       |           |
  //     system     application
  //       |           |
  //      leaf        leaf
  //
```
where the leaf nodes contain all processes and the system/application
cgroups apply cpu.weight and memory.min constraints.
3. CgroupManager now has a move ctor/assignment operator that allows
ownership and lifecycle to be managed by the NodeManager.
4. CgroupManager is now owned by NodeManager.
5. An end-to-end integration test in
`python/ray/tests/resource_isolation/test_resource_isolation_integration.py`.
6. Moved all previous integration tests from test_ray_init and test_cli
into test_resource_isolation_integration.py. These tests have TODOs to
finish them up once the rest of cgroup features are implemented.

---------

Signed-off-by: irabbani <israbbani@gmail.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: Zhiqiang Ma <zhiqiang.ma@intel.com>

…r to move processes into system cgroup (ray-project#56446)

This PR stacks on ray-project#56352 .

For more details about the resource isolation project see
ray-project#54703.

This PR the following functions to move a process into the system
cgroup:
* CgroupManagerInterface::AddProcessToSystemCgroup
* CgroupDriverInterface::AddProcessToCgroup

I've also added integration tests for SysFsCgroupDriver and unit tests
for CgroupManager.

Let me explain how these APIs will be used. In the next PR, the raylet
will
* be passed a list of pids of system processes that are started before
the raylet starts and need to be moved into the system cgroup (e.g.
gcs_server)
* call CgroupManagerInterface::AddProcessToSystemCgroup for each of
these pids to move them into the system cgroup.

---------

Signed-off-by: Ibrahim Rabbani <irabbani@anyscale.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: Zhiqiang Ma <zhiqiang.ma@intel.com>

…n startup (#56522)

This PR stacks on #56352 .

For more details about the resource isolation project see
#54703.

This PR the makes the raylet move the system processes into the system
cgroup on startup if resource isolation is enabled.

It introduces the following
* A new raylet cli arg `--system-pids` which is a comma-separated string
of pids of system processes that are started before the raylet. As of
today, it contains
* On the head node: gcs_server, dashboard_api_server, ray client server,
monitor (autoscaler)
  * On every node (including head): process subreaper, log monitor.
* End-to-end integration tests for resource isolation with the Ray SDK
(`ray.init`) and the Ray CLI (`ray --start`)

There are a few rough edges (I've added a comment on the PR where
relevant):
1. The construction of ResourceIsolationConfig is spread across multiple
call-sites (create the object, add the object store memory, add the
system pids). The big positive of doing it this way was to fail fast on
invalid user input (in scripts.py and worker.py). I think it needs to
have at least two components: the user input (cgroup_path,
system_reserved_memory, ...) and the derived input (system_pids,
total_system_reserved_memory).
2. How to determine which processes should be moved? Right now I'm using
`self.all_processes` in `node.py`. It _should_ contain all processes
started so far, but there's no guarantee.
3. How intrusive should the integration test be? Should we count the
number of pids inside the system cgroup? (This was answered in #56549)
4. How should a user setup multiple nodes on the same VM? I haven't
written an integration test for it yet because there are multiple
options for how to set this up.

---------

Signed-off-by: irabbani <israbbani@gmail.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>

ray-project#56352)

This PR stacks on ray-project#56297.

For more details about the resource isolation project see
ray-project#54703.

This PR wires the resource isolation config (introduced here ray-project#51865)
from ray cli (ray start) and the ray sdk (ray.init) into the raylet.
Notable changes include:
1. A separate python test target for running test_resource_isolation
related unit and integration tests. This unifies all cgroup related
tests under one buildkite target and removes the need for `--except-tags
cgroup` everywhere else.
2. Modification to the cgroup hierarchy. This was an oversight on my
part. The "no internal processes" constraint says that a non-root cgroup
can either have controllers enabled or have processes. Therefore, the
new hierarchy looks like:
```
  //      base_cgroup_path (e.g. /sys/fs/cgroup)
  //             |
  //     ray_node_<node_id>
  //       |           |
  //     system     application
  //       |           |
  //      leaf        leaf
  //
```
where the leaf nodes contain all processes and the system/application
cgroups apply cpu.weight and memory.min constraints.
3. CgroupManager now has a move ctor/assignment operator that allows
ownership and lifecycle to be managed by the NodeManager.
4. CgroupManager is now owned by NodeManager.
5. An end-to-end integration test in
`python/ray/tests/resource_isolation/test_resource_isolation_integration.py`.
6. Moved all previous integration tests from test_ray_init and test_cli
into test_resource_isolation_integration.py. These tests have TODOs to
finish them up once the rest of cgroup features are implemented.

---------

Signed-off-by: irabbani <israbbani@gmail.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: zac <zac@anyscale.com>

…r to move processes into system cgroup (ray-project#56446)

This PR stacks on ray-project#56352 .

For more details about the resource isolation project see
ray-project#54703.

This PR the following functions to move a process into the system
cgroup:
* CgroupManagerInterface::AddProcessToSystemCgroup
* CgroupDriverInterface::AddProcessToCgroup

I've also added integration tests for SysFsCgroupDriver and unit tests
for CgroupManager.

Let me explain how these APIs will be used. In the next PR, the raylet
will
* be passed a list of pids of system processes that are started before
the raylet starts and need to be moved into the system cgroup (e.g.
gcs_server)
* call CgroupManagerInterface::AddProcessToSystemCgroup for each of
these pids to move them into the system cgroup.

---------

Signed-off-by: Ibrahim Rabbani <irabbani@anyscale.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: zac <zac@anyscale.com>

…n startup (ray-project#56522)

This PR stacks on ray-project#56352 .

For more details about the resource isolation project see
ray-project#54703.

This PR the makes the raylet move the system processes into the system
cgroup on startup if resource isolation is enabled.

It introduces the following
* A new raylet cli arg `--system-pids` which is a comma-separated string
of pids of system processes that are started before the raylet. As of
today, it contains
* On the head node: gcs_server, dashboard_api_server, ray client server,
monitor (autoscaler)
  * On every node (including head): process subreaper, log monitor.
* End-to-end integration tests for resource isolation with the Ray SDK
(`ray.init`) and the Ray CLI (`ray --start`)

There are a few rough edges (I've added a comment on the PR where
relevant):
1. The construction of ResourceIsolationConfig is spread across multiple
call-sites (create the object, add the object store memory, add the
system pids). The big positive of doing it this way was to fail fast on
invalid user input (in scripts.py and worker.py). I think it needs to
have at least two components: the user input (cgroup_path,
system_reserved_memory, ...) and the derived input (system_pids,
total_system_reserved_memory).
2. How to determine which processes should be moved? Right now I'm using
`self.all_processes` in `node.py`. It _should_ contain all processes
started so far, but there's no guarantee.
3. How intrusive should the integration test be? Should we count the
number of pids inside the system cgroup? (This was answered in ray-project#56549)
4. How should a user setup multiple nodes on the same VM? I haven't
written an integration test for it yet because there are multiple
options for how to set this up.

---------

Signed-off-by: irabbani <israbbani@gmail.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: zac <zac@anyscale.com>

#56352)

This PR stacks on #56297.

For more details about the resource isolation project see
#54703.

This PR wires the resource isolation config (introduced here #51865)
from ray cli (ray start) and the ray sdk (ray.init) into the raylet.
Notable changes include:
1. A separate python test target for running test_resource_isolation
related unit and integration tests. This unifies all cgroup related
tests under one buildkite target and removes the need for `--except-tags
cgroup` everywhere else.
2. Modification to the cgroup hierarchy. This was an oversight on my
part. The "no internal processes" constraint says that a non-root cgroup
can either have controllers enabled or have processes. Therefore, the
new hierarchy looks like:
```
  //      base_cgroup_path (e.g. /sys/fs/cgroup)
  //             |
  //     ray_node_<node_id>
  //       |           |
  //     system     application
  //       |           |
  //      leaf        leaf
  //
```
where the leaf nodes contain all processes and the system/application
cgroups apply cpu.weight and memory.min constraints.
3. CgroupManager now has a move ctor/assignment operator that allows
ownership and lifecycle to be managed by the NodeManager.
4. CgroupManager is now owned by NodeManager.
5. An end-to-end integration test in
`python/ray/tests/resource_isolation/test_resource_isolation_integration.py`.
6. Moved all previous integration tests from test_ray_init and test_cli
into test_resource_isolation_integration.py. These tests have TODOs to
finish them up once the rest of cgroup features are implemented.

---------

Signed-off-by: irabbani <israbbani@gmail.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

…r to move processes into system cgroup (#56446)

This PR stacks on #56352 .

For more details about the resource isolation project see
#54703.

This PR the following functions to move a process into the system
cgroup:
* CgroupManagerInterface::AddProcessToSystemCgroup
* CgroupDriverInterface::AddProcessToCgroup

I've also added integration tests for SysFsCgroupDriver and unit tests
for CgroupManager.

Let me explain how these APIs will be used. In the next PR, the raylet
will
* be passed a list of pids of system processes that are started before
the raylet starts and need to be moved into the system cgroup (e.g.
gcs_server)
* call CgroupManagerInterface::AddProcessToSystemCgroup for each of
these pids to move them into the system cgroup.

---------

Signed-off-by: Ibrahim Rabbani <irabbani@anyscale.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

…n startup (#56522)

This PR stacks on #56352 .

For more details about the resource isolation project see
#54703.

This PR the makes the raylet move the system processes into the system
cgroup on startup if resource isolation is enabled.

It introduces the following
* A new raylet cli arg `--system-pids` which is a comma-separated string
of pids of system processes that are started before the raylet. As of
today, it contains
* On the head node: gcs_server, dashboard_api_server, ray client server,
monitor (autoscaler)
  * On every node (including head): process subreaper, log monitor.
* End-to-end integration tests for resource isolation with the Ray SDK
(`ray.init`) and the Ray CLI (`ray --start`)

There are a few rough edges (I've added a comment on the PR where
relevant):
1. The construction of ResourceIsolationConfig is spread across multiple
call-sites (create the object, add the object store memory, add the
system pids). The big positive of doing it this way was to fail fast on
invalid user input (in scripts.py and worker.py). I think it needs to
have at least two components: the user input (cgroup_path,
system_reserved_memory, ...) and the derived input (system_pids,
total_system_reserved_memory).
2. How to determine which processes should be moved? Right now I'm using
`self.all_processes` in `node.py`. It _should_ contain all processes
started so far, but there's no guarantee.
3. How intrusive should the integration test be? Should we count the
number of pids inside the system cgroup? (This was answered in #56549)
4. How should a user setup multiple nodes on the same VM? I haven't
written an integration test for it yet because there are multiple
options for how to set this up.

---------

Signed-off-by: irabbani <israbbani@gmail.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

ray-project#56352)

This PR stacks on ray-project#56297.

For more details about the resource isolation project see
ray-project#54703.

This PR wires the resource isolation config (introduced here ray-project#51865)
from ray cli (ray start) and the ray sdk (ray.init) into the raylet.
Notable changes include:
1. A separate python test target for running test_resource_isolation
related unit and integration tests. This unifies all cgroup related
tests under one buildkite target and removes the need for `--except-tags
cgroup` everywhere else.
2. Modification to the cgroup hierarchy. This was an oversight on my
part. The "no internal processes" constraint says that a non-root cgroup
can either have controllers enabled or have processes. Therefore, the
new hierarchy looks like:
```
  //      base_cgroup_path (e.g. /sys/fs/cgroup)
  //             |
  //     ray_node_<node_id>
  //       |           |
  //     system     application
  //       |           |
  //      leaf        leaf
  //
```
where the leaf nodes contain all processes and the system/application
cgroups apply cpu.weight and memory.min constraints.
3. CgroupManager now has a move ctor/assignment operator that allows
ownership and lifecycle to be managed by the NodeManager.
4. CgroupManager is now owned by NodeManager.
5. An end-to-end integration test in
`python/ray/tests/resource_isolation/test_resource_isolation_integration.py`.
6. Moved all previous integration tests from test_ray_init and test_cli
into test_resource_isolation_integration.py. These tests have TODOs to
finish them up once the rest of cgroup features are implemented.

---------

Signed-off-by: irabbani <israbbani@gmail.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: Marco Stephan <marco@magic.dev>

…r to move processes into system cgroup (ray-project#56446)

This PR stacks on ray-project#56352 .

For more details about the resource isolation project see
ray-project#54703.

This PR the following functions to move a process into the system
cgroup:
* CgroupManagerInterface::AddProcessToSystemCgroup
* CgroupDriverInterface::AddProcessToCgroup

I've also added integration tests for SysFsCgroupDriver and unit tests
for CgroupManager.

Let me explain how these APIs will be used. In the next PR, the raylet
will
* be passed a list of pids of system processes that are started before
the raylet starts and need to be moved into the system cgroup (e.g.
gcs_server)
* call CgroupManagerInterface::AddProcessToSystemCgroup for each of
these pids to move them into the system cgroup.

---------

Signed-off-by: Ibrahim Rabbani <irabbani@anyscale.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: Marco Stephan <marco@magic.dev>

…n startup (ray-project#56522)

This PR stacks on ray-project#56352 .

For more details about the resource isolation project see
ray-project#54703.

This PR the makes the raylet move the system processes into the system
cgroup on startup if resource isolation is enabled.

It introduces the following
* A new raylet cli arg `--system-pids` which is a comma-separated string
of pids of system processes that are started before the raylet. As of
today, it contains
* On the head node: gcs_server, dashboard_api_server, ray client server,
monitor (autoscaler)
  * On every node (including head): process subreaper, log monitor.
* End-to-end integration tests for resource isolation with the Ray SDK
(`ray.init`) and the Ray CLI (`ray --start`)

There are a few rough edges (I've added a comment on the PR where
relevant):
1. The construction of ResourceIsolationConfig is spread across multiple
call-sites (create the object, add the object store memory, add the
system pids). The big positive of doing it this way was to fail fast on
invalid user input (in scripts.py and worker.py). I think it needs to
have at least two components: the user input (cgroup_path,
system_reserved_memory, ...) and the derived input (system_pids,
total_system_reserved_memory).
2. How to determine which processes should be moved? Right now I'm using
`self.all_processes` in `node.py`. It _should_ contain all processes
started so far, but there's no guarantee.
3. How intrusive should the integration test be? Should we count the
number of pids inside the system cgroup? (This was answered in ray-project#56549)
4. How should a user setup multiple nodes on the same VM? I haven't
written an integration test for it yet because there are multiple
options for how to set this up.

---------

Signed-off-by: irabbani <israbbani@gmail.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: Marco Stephan <marco@magic.dev>

#56352)

This PR stacks on #56297.

For more details about the resource isolation project see
#54703.

This PR wires the resource isolation config (introduced here #51865)
from ray cli (ray start) and the ray sdk (ray.init) into the raylet.
Notable changes include:
1. A separate python test target for running test_resource_isolation
related unit and integration tests. This unifies all cgroup related
tests under one buildkite target and removes the need for `--except-tags
cgroup` everywhere else.
2. Modification to the cgroup hierarchy. This was an oversight on my
part. The "no internal processes" constraint says that a non-root cgroup
can either have controllers enabled or have processes. Therefore, the
new hierarchy looks like:
```
  //      base_cgroup_path (e.g. /sys/fs/cgroup)
  //             |
  //     ray_node_<node_id>
  //       |           |
  //     system     application
  //       |           |
  //      leaf        leaf
  //
```
where the leaf nodes contain all processes and the system/application
cgroups apply cpu.weight and memory.min constraints.
3. CgroupManager now has a move ctor/assignment operator that allows
ownership and lifecycle to be managed by the NodeManager.
4. CgroupManager is now owned by NodeManager.
5. An end-to-end integration test in
`python/ray/tests/resource_isolation/test_resource_isolation_integration.py`.
6. Moved all previous integration tests from test_ray_init and test_cli
into test_resource_isolation_integration.py. These tests have TODOs to
finish them up once the rest of cgroup features are implemented.

---------

Signed-off-by: irabbani <israbbani@gmail.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

…r to move processes into system cgroup (#56446)

This PR stacks on #56352 .

For more details about the resource isolation project see
#54703.

This PR the following functions to move a process into the system
cgroup:
* CgroupManagerInterface::AddProcessToSystemCgroup
* CgroupDriverInterface::AddProcessToCgroup

I've also added integration tests for SysFsCgroupDriver and unit tests
for CgroupManager.

Let me explain how these APIs will be used. In the next PR, the raylet
will
* be passed a list of pids of system processes that are started before
the raylet starts and need to be moved into the system cgroup (e.g.
gcs_server)
* call CgroupManagerInterface::AddProcessToSystemCgroup for each of
these pids to move them into the system cgroup.

---------

Signed-off-by: Ibrahim Rabbani <irabbani@anyscale.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

…n startup (#56522)

This PR stacks on #56352 .

For more details about the resource isolation project see
#54703.

This PR the makes the raylet move the system processes into the system
cgroup on startup if resource isolation is enabled.

It introduces the following
* A new raylet cli arg `--system-pids` which is a comma-separated string
of pids of system processes that are started before the raylet. As of
today, it contains
* On the head node: gcs_server, dashboard_api_server, ray client server,
monitor (autoscaler)
  * On every node (including head): process subreaper, log monitor.
* End-to-end integration tests for resource isolation with the Ray SDK
(`ray.init`) and the Ray CLI (`ray --start`)

There are a few rough edges (I've added a comment on the PR where
relevant):
1. The construction of ResourceIsolationConfig is spread across multiple
call-sites (create the object, add the object store memory, add the
system pids). The big positive of doing it this way was to fail fast on
invalid user input (in scripts.py and worker.py). I think it needs to
have at least two components: the user input (cgroup_path,
system_reserved_memory, ...) and the derived input (system_pids,
total_system_reserved_memory).
2. How to determine which processes should be moved? Right now I'm using
`self.all_processes` in `node.py`. It _should_ contain all processes
started so far, but there's no guarantee.
3. How intrusive should the integration test be? Should we count the
number of pids inside the system cgroup? (This was answered in #56549)
4. How should a user setup multiple nodes on the same VM? I haven't
written an integration test for it yet because there are multiple
options for how to set this up.

---------

Signed-off-by: irabbani <israbbani@gmail.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

ray-project#56352)

This PR stacks on ray-project#56297.

For more details about the resource isolation project see
ray-project#54703.

This PR wires the resource isolation config (introduced here ray-project#51865)
from ray cli (ray start) and the ray sdk (ray.init) into the raylet.
Notable changes include:
1. A separate python test target for running test_resource_isolation
related unit and integration tests. This unifies all cgroup related
tests under one buildkite target and removes the need for `--except-tags
cgroup` everywhere else.
2. Modification to the cgroup hierarchy. This was an oversight on my
part. The "no internal processes" constraint says that a non-root cgroup
can either have controllers enabled or have processes. Therefore, the
new hierarchy looks like:
```
  //      base_cgroup_path (e.g. /sys/fs/cgroup)
  //             |
  //     ray_node_<node_id>
  //       |           |
  //     system     application
  //       |           |
  //      leaf        leaf
  //
```
where the leaf nodes contain all processes and the system/application
cgroups apply cpu.weight and memory.min constraints.
3. CgroupManager now has a move ctor/assignment operator that allows
ownership and lifecycle to be managed by the NodeManager.
4. CgroupManager is now owned by NodeManager.
5. An end-to-end integration test in
`python/ray/tests/resource_isolation/test_resource_isolation_integration.py`.
6. Moved all previous integration tests from test_ray_init and test_cli
into test_resource_isolation_integration.py. These tests have TODOs to
finish them up once the rest of cgroup features are implemented.

---------

Signed-off-by: irabbani <israbbani@gmail.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: Douglas Strodtman <douglas@anyscale.com>

…r to move processes into system cgroup (ray-project#56446)

This PR stacks on ray-project#56352 .

For more details about the resource isolation project see
ray-project#54703.

This PR the following functions to move a process into the system
cgroup:
* CgroupManagerInterface::AddProcessToSystemCgroup
* CgroupDriverInterface::AddProcessToCgroup

I've also added integration tests for SysFsCgroupDriver and unit tests
for CgroupManager.

Let me explain how these APIs will be used. In the next PR, the raylet
will
* be passed a list of pids of system processes that are started before
the raylet starts and need to be moved into the system cgroup (e.g.
gcs_server)
* call CgroupManagerInterface::AddProcessToSystemCgroup for each of
these pids to move them into the system cgroup.

---------

Signed-off-by: Ibrahim Rabbani <irabbani@anyscale.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: Douglas Strodtman <douglas@anyscale.com>

…n startup (ray-project#56522)

This PR stacks on ray-project#56352 .

For more details about the resource isolation project see
ray-project#54703.

This PR the makes the raylet move the system processes into the system
cgroup on startup if resource isolation is enabled.

It introduces the following
* A new raylet cli arg `--system-pids` which is a comma-separated string
of pids of system processes that are started before the raylet. As of
today, it contains
* On the head node: gcs_server, dashboard_api_server, ray client server,
monitor (autoscaler)
  * On every node (including head): process subreaper, log monitor.
* End-to-end integration tests for resource isolation with the Ray SDK
(`ray.init`) and the Ray CLI (`ray --start`)

There are a few rough edges (I've added a comment on the PR where
relevant):
1. The construction of ResourceIsolationConfig is spread across multiple
call-sites (create the object, add the object store memory, add the
system pids). The big positive of doing it this way was to fail fast on
invalid user input (in scripts.py and worker.py). I think it needs to
have at least two components: the user input (cgroup_path,
system_reserved_memory, ...) and the derived input (system_pids,
total_system_reserved_memory).
2. How to determine which processes should be moved? Right now I'm using
`self.all_processes` in `node.py`. It _should_ contain all processes
started so far, but there's no guarantee.
3. How intrusive should the integration test be? Should we count the
number of pids inside the system cgroup? (This was answered in ray-project#56549)
4. How should a user setup multiple nodes on the same VM? I haven't
written an integration test for it yet because there are multiple
options for how to set this up.

---------

Signed-off-by: irabbani <israbbani@gmail.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: Douglas Strodtman <douglas@anyscale.com>

ray-project#56352)

This PR stacks on ray-project#56297.

For more details about the resource isolation project see
ray-project#54703.

This PR wires the resource isolation config (introduced here ray-project#51865)
from ray cli (ray start) and the ray sdk (ray.init) into the raylet.
Notable changes include:
1. A separate python test target for running test_resource_isolation
related unit and integration tests. This unifies all cgroup related
tests under one buildkite target and removes the need for `--except-tags
cgroup` everywhere else.
2. Modification to the cgroup hierarchy. This was an oversight on my
part. The "no internal processes" constraint says that a non-root cgroup
can either have controllers enabled or have processes. Therefore, the
new hierarchy looks like:
```
  //      base_cgroup_path (e.g. /sys/fs/cgroup)
  //             |
  //     ray_node_<node_id>
  //       |           |
  //     system     application
  //       |           |
  //      leaf        leaf
  //
```
where the leaf nodes contain all processes and the system/application
cgroups apply cpu.weight and memory.min constraints.
3. CgroupManager now has a move ctor/assignment operator that allows
ownership and lifecycle to be managed by the NodeManager.
4. CgroupManager is now owned by NodeManager.
5. An end-to-end integration test in
`python/ray/tests/resource_isolation/test_resource_isolation_integration.py`.
6. Moved all previous integration tests from test_ray_init and test_cli
into test_resource_isolation_integration.py. These tests have TODOs to
finish them up once the rest of cgroup features are implemented.

---------

Signed-off-by: irabbani <israbbani@gmail.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>

…r to move processes into system cgroup (ray-project#56446)

This PR stacks on ray-project#56352 .

For more details about the resource isolation project see
ray-project#54703.

This PR the following functions to move a process into the system
cgroup:
* CgroupManagerInterface::AddProcessToSystemCgroup
* CgroupDriverInterface::AddProcessToCgroup

I've also added integration tests for SysFsCgroupDriver and unit tests
for CgroupManager.

Let me explain how these APIs will be used. In the next PR, the raylet
will
* be passed a list of pids of system processes that are started before
the raylet starts and need to be moved into the system cgroup (e.g.
gcs_server)
* call CgroupManagerInterface::AddProcessToSystemCgroup for each of
these pids to move them into the system cgroup.

---------

Signed-off-by: Ibrahim Rabbani <irabbani@anyscale.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>

…n startup (ray-project#56522)

This PR stacks on ray-project#56352 .

For more details about the resource isolation project see
ray-project#54703.

This PR the makes the raylet move the system processes into the system
cgroup on startup if resource isolation is enabled.

It introduces the following
* A new raylet cli arg `--system-pids` which is a comma-separated string
of pids of system processes that are started before the raylet. As of
today, it contains
* On the head node: gcs_server, dashboard_api_server, ray client server,
monitor (autoscaler)
  * On every node (including head): process subreaper, log monitor.
* End-to-end integration tests for resource isolation with the Ray SDK
(`ray.init`) and the Ray CLI (`ray --start`)

There are a few rough edges (I've added a comment on the PR where
relevant):
1. The construction of ResourceIsolationConfig is spread across multiple
call-sites (create the object, add the object store memory, add the
system pids). The big positive of doing it this way was to fail fast on
invalid user input (in scripts.py and worker.py). I think it needs to
have at least two components: the user input (cgroup_path,
system_reserved_memory, ...) and the derived input (system_pids,
total_system_reserved_memory).
2. How to determine which processes should be moved? Right now I'm using
`self.all_processes` in `node.py`. It _should_ contain all processes
started so far, but there's no guarantee.
3. How intrusive should the integration test be? Should we count the
number of pids inside the system cgroup? (This was answered in ray-project#56549)
4. How should a user setup multiple nodes on the same VM? I haven't
written an integration test for it yet because there are multiple
options for how to set this up.

---------

Signed-off-by: irabbani <israbbani@gmail.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>

ray-project#56352)

This PR stacks on ray-project#56297.

For more details about the resource isolation project see
ray-project#54703.

This PR wires the resource isolation config (introduced here ray-project#51865)
from ray cli (ray start) and the ray sdk (ray.init) into the raylet.
Notable changes include:
1. A separate python test target for running test_resource_isolation
related unit and integration tests. This unifies all cgroup related
tests under one buildkite target and removes the need for `--except-tags
cgroup` everywhere else.
2. Modification to the cgroup hierarchy. This was an oversight on my
part. The "no internal processes" constraint says that a non-root cgroup
can either have controllers enabled or have processes. Therefore, the
new hierarchy looks like:
```
  //      base_cgroup_path (e.g. /sys/fs/cgroup)
  //             |
  //     ray_node_<node_id>
  //       |           |
  //     system     application
  //       |           |
  //      leaf        leaf
  //
```
where the leaf nodes contain all processes and the system/application
cgroups apply cpu.weight and memory.min constraints.
3. CgroupManager now has a move ctor/assignment operator that allows
ownership and lifecycle to be managed by the NodeManager.
4. CgroupManager is now owned by NodeManager.
5. An end-to-end integration test in
`python/ray/tests/resource_isolation/test_resource_isolation_integration.py`.
6. Moved all previous integration tests from test_ray_init and test_cli
into test_resource_isolation_integration.py. These tests have TODOs to
finish them up once the rest of cgroup features are implemented.

---------

Signed-off-by: irabbani <israbbani@gmail.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>

…r to move processes into system cgroup (ray-project#56446)

This PR stacks on ray-project#56352 .

For more details about the resource isolation project see
ray-project#54703.

This PR the following functions to move a process into the system
cgroup:
* CgroupManagerInterface::AddProcessToSystemCgroup
* CgroupDriverInterface::AddProcessToCgroup

I've also added integration tests for SysFsCgroupDriver and unit tests
for CgroupManager.

Let me explain how these APIs will be used. In the next PR, the raylet
will
* be passed a list of pids of system processes that are started before
the raylet starts and need to be moved into the system cgroup (e.g.
gcs_server)
* call CgroupManagerInterface::AddProcessToSystemCgroup for each of
these pids to move them into the system cgroup.

---------

Signed-off-by: Ibrahim Rabbani <irabbani@anyscale.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>

…n startup (ray-project#56522)

This PR stacks on ray-project#56352 .

For more details about the resource isolation project see
ray-project#54703.

This PR the makes the raylet move the system processes into the system
cgroup on startup if resource isolation is enabled.

It introduces the following
* A new raylet cli arg `--system-pids` which is a comma-separated string
of pids of system processes that are started before the raylet. As of
today, it contains
* On the head node: gcs_server, dashboard_api_server, ray client server,
monitor (autoscaler)
  * On every node (including head): process subreaper, log monitor.
* End-to-end integration tests for resource isolation with the Ray SDK
(`ray.init`) and the Ray CLI (`ray --start`)

There are a few rough edges (I've added a comment on the PR where
relevant):
1. The construction of ResourceIsolationConfig is spread across multiple
call-sites (create the object, add the object store memory, add the
system pids). The big positive of doing it this way was to fail fast on
invalid user input (in scripts.py and worker.py). I think it needs to
have at least two components: the user input (cgroup_path,
system_reserved_memory, ...) and the derived input (system_pids,
total_system_reserved_memory).
2. How to determine which processes should be moved? Right now I'm using
`self.all_processes` in `node.py`. It _should_ contain all processes
started so far, but there's no guarantee.
3. How intrusive should the integration test be? Should we count the
number of pids inside the system cgroup? (This was answered in ray-project#56549)
4. How should a user setup multiple nodes on the same VM? I haven't
written an integration test for it yet because there are multiple
options for how to set this up.

---------

Signed-off-by: irabbani <israbbani@gmail.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>

ray-project#56352)

This PR stacks on ray-project#56297.

For more details about the resource isolation project see
ray-project#54703.

This PR wires the resource isolation config (introduced here ray-project#51865)
from ray cli (ray start) and the ray sdk (ray.init) into the raylet.
Notable changes include:
1. A separate python test target for running test_resource_isolation
related unit and integration tests. This unifies all cgroup related
tests under one buildkite target and removes the need for `--except-tags
cgroup` everywhere else.
2. Modification to the cgroup hierarchy. This was an oversight on my
part. The "no internal processes" constraint says that a non-root cgroup
can either have controllers enabled or have processes. Therefore, the
new hierarchy looks like:
```
  //      base_cgroup_path (e.g. /sys/fs/cgroup)
  //             |
  //     ray_node_<node_id>
  //       |           |
  //     system     application
  //       |           |
  //      leaf        leaf
  //
```
where the leaf nodes contain all processes and the system/application
cgroups apply cpu.weight and memory.min constraints.
3. CgroupManager now has a move ctor/assignment operator that allows
ownership and lifecycle to be managed by the NodeManager.
4. CgroupManager is now owned by NodeManager.
5. An end-to-end integration test in
`python/ray/tests/resource_isolation/test_resource_isolation_integration.py`.
6. Moved all previous integration tests from test_ray_init and test_cli
into test_resource_isolation_integration.py. These tests have TODOs to
finish them up once the rest of cgroup features are implemented.

---------

Signed-off-by: irabbani <israbbani@gmail.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: Future-Outlier <eric901201@gmail.com>

…r to move processes into system cgroup (ray-project#56446)

This PR stacks on ray-project#56352 .

For more details about the resource isolation project see
ray-project#54703.

This PR the following functions to move a process into the system
cgroup:
* CgroupManagerInterface::AddProcessToSystemCgroup
* CgroupDriverInterface::AddProcessToCgroup

I've also added integration tests for SysFsCgroupDriver and unit tests
for CgroupManager.

Let me explain how these APIs will be used. In the next PR, the raylet
will
* be passed a list of pids of system processes that are started before
the raylet starts and need to be moved into the system cgroup (e.g.
gcs_server)
* call CgroupManagerInterface::AddProcessToSystemCgroup for each of
these pids to move them into the system cgroup.

---------

Signed-off-by: Ibrahim Rabbani <irabbani@anyscale.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: Future-Outlier <eric901201@gmail.com>

…n startup (ray-project#56522)

This PR stacks on ray-project#56352 .

For more details about the resource isolation project see
ray-project#54703.

This PR the makes the raylet move the system processes into the system
cgroup on startup if resource isolation is enabled.

It introduces the following
* A new raylet cli arg `--system-pids` which is a comma-separated string
of pids of system processes that are started before the raylet. As of
today, it contains
* On the head node: gcs_server, dashboard_api_server, ray client server,
monitor (autoscaler)
  * On every node (including head): process subreaper, log monitor.
* End-to-end integration tests for resource isolation with the Ray SDK
(`ray.init`) and the Ray CLI (`ray --start`)

There are a few rough edges (I've added a comment on the PR where
relevant):
1. The construction of ResourceIsolationConfig is spread across multiple
call-sites (create the object, add the object store memory, add the
system pids). The big positive of doing it this way was to fail fast on
invalid user input (in scripts.py and worker.py). I think it needs to
have at least two components: the user input (cgroup_path,
system_reserved_memory, ...) and the derived input (system_pids,
total_system_reserved_memory).
2. How to determine which processes should be moved? Right now I'm using
`self.all_processes` in `node.py`. It _should_ contain all processes
started so far, but there's no guarantee.
3. How intrusive should the integration test be? Should we count the
number of pids inside the system cgroup? (This was answered in ray-project#56549)
4. How should a user setup multiple nodes on the same VM? I haven't
written an integration test for it yet because there are multiple
options for how to set this up.

---------

Signed-off-by: irabbani <israbbani@gmail.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: Future-Outlier <eric901201@gmail.com>