Search before asking

• I searched the issues and found no similar issues.

Ray Component

Ray Clusters

What happened + What you expected to happen

I have a ray cluster in kubernetes environment, I use it for training lightgbm_ray models.
But after running this code multiple times, ray::IDLE processes keeps growing and eventually workers die by OOM.

Versions / Dependencies

custom build by cherry-picking #22007 commit upon 1.10.0 version

Reproduction script

import ray
import modin

ray.init(address="ray://our-cluster-head")
df = ray.data.read_parquet("hdfs://NAMESPACE/user/hive/warehouse/DB.db/TABLE").to_modin()

ray.shutdown()

Anything else

there was a same result even when we used dask instead of modin.

Are you willing to submit a PR?

• Yes I am willing to submit a PR!