Skip to content

[fork][tracking issue] grpc thread pool hanging on fork #31885

@rickyyx

Description

@rickyyx

What version of gRPC and what language are you using?

grpc 1.51.1

What operating system (Linux, Windows,...) and version?

macOS Catalina 10.15

What runtime / compiler are you using (e.g. python version or version of gcc)

python 3.8.15

What did you do?

Please provide either 1) A unit test for reproducing the bug or 2) Specific steps for us to follow to reproduce the bug. If there’s not enough information to debug the problem, gRPC team may close the issue at their discretion. You’re welcome to re-open the issue once you have a reproduction.

  1. With a ray installation 2.1.0
pip install ray==2.1.0
  1. Running a simple workload that starts a ray cluster and launches tasks to 2 threads.
import ray

def run(): 
    ray.init(num_cpus=2)

    @ray.remote
    def g():
        pass

    ray.get(g.remote())
    ray.get(g.remote())

    ray.shutdown()

for _ in range(10000):
    run()

What did you expect to see?

Script runs ok.

What did you see instead?

Script hanging with

E1213 01:47:43.538549000 4544900544 thread_pool.cc:253]                Waiting for thread pool to idle before forking
E1213 01:47:46.545305000 4544900544 thread_pool.cc:253]                Waiting for thread pool to idle before forking
E1213 01:47:49.551290000 4544900544 thread_pool.cc:253]                Waiting for thread pool to idle before forking
E1213 01:47:52.554189000 4544900544 thread_pool.cc:253]                Waiting for thread pool to idle before forking
E1213 01:47:55.556100000 4544900544 thread_pool.cc:253]                Waiting for thread pool to idle before forking
....

Make sure you include information that can help us debug (full error message, exception listing, stack trace, logs).

See TROUBLESHOOTING.md for how to diagnose problems better.

Anything else we should know about your project / environment?

Metadata

Metadata

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions