…n startup (ray-project#56522)

This PR stacks on ray-project#56352 .

For more details about the resource isolation project see
ray-project#54703.

This PR the makes the raylet move the system processes into the system
cgroup on startup if resource isolation is enabled.

It introduces the following
* A new raylet cli arg `--system-pids` which is a comma-separated string
of pids of system processes that are started before the raylet. As of
today, it contains
* On the head node: gcs_server, dashboard_api_server, ray client server,
monitor (autoscaler)
  * On every node (including head): process subreaper, log monitor.
* End-to-end integration tests for resource isolation with the Ray SDK
(`ray.init`) and the Ray CLI (`ray --start`)

There are a few rough edges (I've added a comment on the PR where
relevant):
1. The construction of ResourceIsolationConfig is spread across multiple
call-sites (create the object, add the object store memory, add the
system pids). The big positive of doing it this way was to fail fast on
invalid user input (in scripts.py and worker.py). I think it needs to
have at least two components: the user input (cgroup_path,
system_reserved_memory, ...) and the derived input (system_pids,
total_system_reserved_memory).
2. How to determine which processes should be moved? Right now I'm using
`self.all_processes` in `node.py`. It _should_ contain all processes
started so far, but there's no guarantee.
3. How intrusive should the integration test be? Should we count the
number of pids inside the system cgroup? (This was answered in ray-project#56549)
4. How should a user setup multiple nodes on the same VM? I haven't
written an integration test for it yet because there are multiple
options for how to set this up.

---------

Signed-off-by: irabbani <israbbani@gmail.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: Marco Stephan <marco@magic.dev>

…cation cgroup (ray-project#56549)

This PR stacks on ray-project#56522 .

For more details about the resource isolation project see
ray-project#54703.

This PR the makes the raylet move runtime_env and dashboard agents into
the system cgroup. Workers are now spawned inside the application
cgroup.

It introduces the following:
* I've added a new target `raylet_cgroup_types` which defines the type
used all functions that need to add a process to a cgroup.
* A new parameter is added to `NodeManager`, `WorkerPool`,
`AgentManager`, and `Process` constructors. The parameter is a callback
that will use the CgroupManager to add a process to the respective
cgroup.
* The callback is created in `main.cc`.
* `main.cc` owns CgroupManager because it needs to outlive the
`WorkerPool`.
* `process.c` calls the callback after fork() in the child process so
nothing else can happen in the forked process before it's moved into the
correct cgroup.
* Integration tests in python for end-to-end testing of cgroups with
system and application processes moved into their respective cgroups.
The tests are inside
`python/ray/tests/resource_isolation/test_resource_isolation_integration.py`
and have similar setup/teardown to the C++ integration tests introduced
in ray-project#55063.

---------

Signed-off-by: Ibrahim Rabbani <irabbani@anyscale.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: Marco Stephan <marco@magic.dev>

This PR stacks on #56549.

For more details about the resource isolation project see
#54703.

This PR deletes the old cgroup code inside `/src/ray/common/cgroup`
along with all associated comments.

---------

Signed-off-by: irabbani <israbbani@gmail.com>

…he application cgroup (ray-project#56549)"

This reverts commit 6044577.

…n startup (#56522)

This PR stacks on #56352 .

For more details about the resource isolation project see
#54703.

This PR the makes the raylet move the system processes into the system
cgroup on startup if resource isolation is enabled.

It introduces the following
* A new raylet cli arg `--system-pids` which is a comma-separated string
of pids of system processes that are started before the raylet. As of
today, it contains
* On the head node: gcs_server, dashboard_api_server, ray client server,
monitor (autoscaler)
  * On every node (including head): process subreaper, log monitor.
* End-to-end integration tests for resource isolation with the Ray SDK
(`ray.init`) and the Ray CLI (`ray --start`)

There are a few rough edges (I've added a comment on the PR where
relevant):
1. The construction of ResourceIsolationConfig is spread across multiple
call-sites (create the object, add the object store memory, add the
system pids). The big positive of doing it this way was to fail fast on
invalid user input (in scripts.py and worker.py). I think it needs to
have at least two components: the user input (cgroup_path,
system_reserved_memory, ...) and the derived input (system_pids,
total_system_reserved_memory).
2. How to determine which processes should be moved? Right now I'm using
`self.all_processes` in `node.py`. It _should_ contain all processes
started so far, but there's no guarantee.
3. How intrusive should the integration test be? Should we count the
number of pids inside the system cgroup? (This was answered in #56549)
4. How should a user setup multiple nodes on the same VM? I haven't
written an integration test for it yet because there are multiple
options for how to set this up.

---------

Signed-off-by: irabbani <israbbani@gmail.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

…cation cgroup (#56549)

This PR stacks on #56522 .

For more details about the resource isolation project see
#54703.

This PR the makes the raylet move runtime_env and dashboard agents into
the system cgroup. Workers are now spawned inside the application
cgroup.

It introduces the following:
* I've added a new target `raylet_cgroup_types` which defines the type
used all functions that need to add a process to a cgroup.
* A new parameter is added to `NodeManager`, `WorkerPool`,
`AgentManager`, and `Process` constructors. The parameter is a callback
that will use the CgroupManager to add a process to the respective
cgroup.
* The callback is created in `main.cc`.
* `main.cc` owns CgroupManager because it needs to outlive the
`WorkerPool`.
* `process.c` calls the callback after fork() in the child process so
nothing else can happen in the forked process before it's moved into the
correct cgroup.
* Integration tests in python for end-to-end testing of cgroups with
system and application processes moved into their respective cgroups.
The tests are inside
`python/ray/tests/resource_isolation/test_resource_isolation_integration.py`
and have similar setup/teardown to the C++ integration tests introduced
in #55063.

---------

Signed-off-by: Ibrahim Rabbani <irabbani@anyscale.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

This PR stacks on #56549.

For more details about the resource isolation project see
#54703.

This PR deletes the old cgroup code inside `/src/ray/common/cgroup`
along with all associated comments.

---------

Signed-off-by: irabbani <israbbani@gmail.com>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

…n startup (ray-project#56522)

This PR stacks on ray-project#56352 .

For more details about the resource isolation project see
ray-project#54703.

This PR the makes the raylet move the system processes into the system
cgroup on startup if resource isolation is enabled.

It introduces the following
* A new raylet cli arg `--system-pids` which is a comma-separated string
of pids of system processes that are started before the raylet. As of
today, it contains
* On the head node: gcs_server, dashboard_api_server, ray client server,
monitor (autoscaler)
  * On every node (including head): process subreaper, log monitor.
* End-to-end integration tests for resource isolation with the Ray SDK
(`ray.init`) and the Ray CLI (`ray --start`)

There are a few rough edges (I've added a comment on the PR where
relevant):
1. The construction of ResourceIsolationConfig is spread across multiple
call-sites (create the object, add the object store memory, add the
system pids). The big positive of doing it this way was to fail fast on
invalid user input (in scripts.py and worker.py). I think it needs to
have at least two components: the user input (cgroup_path,
system_reserved_memory, ...) and the derived input (system_pids,
total_system_reserved_memory).
2. How to determine which processes should be moved? Right now I'm using
`self.all_processes` in `node.py`. It _should_ contain all processes
started so far, but there's no guarantee.
3. How intrusive should the integration test be? Should we count the
number of pids inside the system cgroup? (This was answered in ray-project#56549)
4. How should a user setup multiple nodes on the same VM? I haven't
written an integration test for it yet because there are multiple
options for how to set this up.

---------

Signed-off-by: irabbani <israbbani@gmail.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: Douglas Strodtman <douglas@anyscale.com>

…cation cgroup (#56549)

This PR stacks on #56522 .

For more details about the resource isolation project see
#54703.

This PR the makes the raylet move runtime_env and dashboard agents into
the system cgroup. Workers are now spawned inside the application
cgroup.

It introduces the following:
* I've added a new target `raylet_cgroup_types` which defines the type
used all functions that need to add a process to a cgroup.
* A new parameter is added to `NodeManager`, `WorkerPool`,
`AgentManager`, and `Process` constructors. The parameter is a callback
that will use the CgroupManager to add a process to the respective
cgroup.
* The callback is created in `main.cc`.
* `main.cc` owns CgroupManager because it needs to outlive the
`WorkerPool`.
* `process.c` calls the callback after fork() in the child process so
nothing else can happen in the forked process before it's moved into the
correct cgroup.
* Integration tests in python for end-to-end testing of cgroups with
system and application processes moved into their respective cgroups.
The tests are inside
`python/ray/tests/resource_isolation/test_resource_isolation_integration.py`
and have similar setup/teardown to the C++ integration tests introduced
in #55063.

---------

Signed-off-by: Ibrahim Rabbani <irabbani@anyscale.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: Douglas Strodtman <douglas@anyscale.com>

This PR stacks on #56549.

For more details about the resource isolation project see
#54703.

This PR deletes the old cgroup code inside `/src/ray/common/cgroup`
along with all associated comments.

---------

Signed-off-by: irabbani <israbbani@gmail.com>
Signed-off-by: Douglas Strodtman <douglas@anyscale.com>

…n startup (ray-project#56522)

This PR stacks on ray-project#56352 .

For more details about the resource isolation project see
ray-project#54703.

This PR the makes the raylet move the system processes into the system
cgroup on startup if resource isolation is enabled.

It introduces the following
* A new raylet cli arg `--system-pids` which is a comma-separated string
of pids of system processes that are started before the raylet. As of
today, it contains
* On the head node: gcs_server, dashboard_api_server, ray client server,
monitor (autoscaler)
  * On every node (including head): process subreaper, log monitor.
* End-to-end integration tests for resource isolation with the Ray SDK
(`ray.init`) and the Ray CLI (`ray --start`)

There are a few rough edges (I've added a comment on the PR where
relevant):
1. The construction of ResourceIsolationConfig is spread across multiple
call-sites (create the object, add the object store memory, add the
system pids). The big positive of doing it this way was to fail fast on
invalid user input (in scripts.py and worker.py). I think it needs to
have at least two components: the user input (cgroup_path,
system_reserved_memory, ...) and the derived input (system_pids,
total_system_reserved_memory).
2. How to determine which processes should be moved? Right now I'm using
`self.all_processes` in `node.py`. It _should_ contain all processes
started so far, but there's no guarantee.
3. How intrusive should the integration test be? Should we count the
number of pids inside the system cgroup? (This was answered in ray-project#56549)
4. How should a user setup multiple nodes on the same VM? I haven't
written an integration test for it yet because there are multiple
options for how to set this up.

---------

Signed-off-by: irabbani <israbbani@gmail.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>

…cation cgroup (ray-project#56549)

This PR stacks on ray-project#56522 .

For more details about the resource isolation project see
ray-project#54703.

This PR the makes the raylet move runtime_env and dashboard agents into
the system cgroup. Workers are now spawned inside the application
cgroup.

It introduces the following:
* I've added a new target `raylet_cgroup_types` which defines the type
used all functions that need to add a process to a cgroup.
* A new parameter is added to `NodeManager`, `WorkerPool`,
`AgentManager`, and `Process` constructors. The parameter is a callback
that will use the CgroupManager to add a process to the respective
cgroup.
* The callback is created in `main.cc`.
* `main.cc` owns CgroupManager because it needs to outlive the
`WorkerPool`.
* `process.c` calls the callback after fork() in the child process so
nothing else can happen in the forked process before it's moved into the
correct cgroup.
* Integration tests in python for end-to-end testing of cgroups with
system and application processes moved into their respective cgroups.
The tests are inside
`python/ray/tests/resource_isolation/test_resource_isolation_integration.py`
and have similar setup/teardown to the C++ integration tests introduced
in ray-project#55063.

---------

Signed-off-by: Ibrahim Rabbani <irabbani@anyscale.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>

…t#56909)

This PR stacks on ray-project#56549.

For more details about the resource isolation project see
ray-project#54703.

This PR deletes the old cgroup code inside `/src/ray/common/cgroup`
along with all associated comments.

---------

Signed-off-by: irabbani <israbbani@gmail.com>

…n startup (ray-project#56522)

This PR stacks on ray-project#56352 .

For more details about the resource isolation project see
ray-project#54703.

This PR the makes the raylet move the system processes into the system
cgroup on startup if resource isolation is enabled.

It introduces the following
* A new raylet cli arg `--system-pids` which is a comma-separated string
of pids of system processes that are started before the raylet. As of
today, it contains
* On the head node: gcs_server, dashboard_api_server, ray client server,
monitor (autoscaler)
  * On every node (including head): process subreaper, log monitor.
* End-to-end integration tests for resource isolation with the Ray SDK
(`ray.init`) and the Ray CLI (`ray --start`)

There are a few rough edges (I've added a comment on the PR where
relevant):
1. The construction of ResourceIsolationConfig is spread across multiple
call-sites (create the object, add the object store memory, add the
system pids). The big positive of doing it this way was to fail fast on
invalid user input (in scripts.py and worker.py). I think it needs to
have at least two components: the user input (cgroup_path,
system_reserved_memory, ...) and the derived input (system_pids,
total_system_reserved_memory).
2. How to determine which processes should be moved? Right now I'm using
`self.all_processes` in `node.py`. It _should_ contain all processes
started so far, but there's no guarantee.
3. How intrusive should the integration test be? Should we count the
number of pids inside the system cgroup? (This was answered in ray-project#56549)
4. How should a user setup multiple nodes on the same VM? I haven't
written an integration test for it yet because there are multiple
options for how to set this up.

---------

Signed-off-by: irabbani <israbbani@gmail.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>

…cation cgroup (ray-project#56549)

This PR stacks on ray-project#56522 .

For more details about the resource isolation project see
ray-project#54703.

This PR the makes the raylet move runtime_env and dashboard agents into
the system cgroup. Workers are now spawned inside the application
cgroup.

It introduces the following:
* I've added a new target `raylet_cgroup_types` which defines the type
used all functions that need to add a process to a cgroup.
* A new parameter is added to `NodeManager`, `WorkerPool`,
`AgentManager`, and `Process` constructors. The parameter is a callback
that will use the CgroupManager to add a process to the respective
cgroup.
* The callback is created in `main.cc`.
* `main.cc` owns CgroupManager because it needs to outlive the
`WorkerPool`.
* `process.c` calls the callback after fork() in the child process so
nothing else can happen in the forked process before it's moved into the
correct cgroup.
* Integration tests in python for end-to-end testing of cgroups with
system and application processes moved into their respective cgroups.
The tests are inside
`python/ray/tests/resource_isolation/test_resource_isolation_integration.py`
and have similar setup/teardown to the C++ integration tests introduced
in ray-project#55063.

---------

Signed-off-by: Ibrahim Rabbani <irabbani@anyscale.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>

…t#56909)

This PR stacks on ray-project#56549.

For more details about the resource isolation project see
ray-project#54703.

This PR deletes the old cgroup code inside `/src/ray/common/cgroup`
along with all associated comments.

---------

Signed-off-by: irabbani <israbbani@gmail.com>

…n startup (ray-project#56522)

This PR stacks on ray-project#56352 .

For more details about the resource isolation project see
ray-project#54703.

This PR the makes the raylet move the system processes into the system
cgroup on startup if resource isolation is enabled.

It introduces the following
* A new raylet cli arg `--system-pids` which is a comma-separated string
of pids of system processes that are started before the raylet. As of
today, it contains
* On the head node: gcs_server, dashboard_api_server, ray client server,
monitor (autoscaler)
  * On every node (including head): process subreaper, log monitor.
* End-to-end integration tests for resource isolation with the Ray SDK
(`ray.init`) and the Ray CLI (`ray --start`)

There are a few rough edges (I've added a comment on the PR where
relevant):
1. The construction of ResourceIsolationConfig is spread across multiple
call-sites (create the object, add the object store memory, add the
system pids). The big positive of doing it this way was to fail fast on
invalid user input (in scripts.py and worker.py). I think it needs to
have at least two components: the user input (cgroup_path,
system_reserved_memory, ...) and the derived input (system_pids,
total_system_reserved_memory).
2. How to determine which processes should be moved? Right now I'm using
`self.all_processes` in `node.py`. It _should_ contain all processes
started so far, but there's no guarantee.
3. How intrusive should the integration test be? Should we count the
number of pids inside the system cgroup? (This was answered in ray-project#56549)
4. How should a user setup multiple nodes on the same VM? I haven't
written an integration test for it yet because there are multiple
options for how to set this up.

---------

Signed-off-by: irabbani <israbbani@gmail.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: Future-Outlier <eric901201@gmail.com>

…cation cgroup (ray-project#56549)

This PR stacks on ray-project#56522 .

For more details about the resource isolation project see
ray-project#54703.

This PR the makes the raylet move runtime_env and dashboard agents into
the system cgroup. Workers are now spawned inside the application
cgroup.

It introduces the following:
* I've added a new target `raylet_cgroup_types` which defines the type
used all functions that need to add a process to a cgroup.
* A new parameter is added to `NodeManager`, `WorkerPool`,
`AgentManager`, and `Process` constructors. The parameter is a callback
that will use the CgroupManager to add a process to the respective
cgroup.
* The callback is created in `main.cc`.
* `main.cc` owns CgroupManager because it needs to outlive the
`WorkerPool`.
* `process.c` calls the callback after fork() in the child process so
nothing else can happen in the forked process before it's moved into the
correct cgroup.
* Integration tests in python for end-to-end testing of cgroups with
system and application processes moved into their respective cgroups.
The tests are inside
`python/ray/tests/resource_isolation/test_resource_isolation_integration.py`
and have similar setup/teardown to the C++ integration tests introduced
in ray-project#55063.

---------

Signed-off-by: Ibrahim Rabbani <irabbani@anyscale.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: Future-Outlier <eric901201@gmail.com>

…t#56909)

This PR stacks on ray-project#56549.

For more details about the resource isolation project see
ray-project#54703.

This PR deletes the old cgroup code inside `/src/ray/common/cgroup`
along with all associated comments.

---------

Signed-off-by: irabbani <israbbani@gmail.com>
Signed-off-by: Future-Outlier <eric901201@gmail.com>