[Core] The sample code runs longer and freezes the system for about a minute on an M3 Pro after version 2.48. #59663
Description
What happened + What you expected to happen
On Ray version 2.48 and above, on m3 mac, the provided script runs and the system hangs almost immediately for about 40 seconds.
Script logs:
....
This cluster consists of
nodes: 1
resources: {'CPU': 11.0, 'node:127.0.0.1': 1.0, 'memory': 8610283520.0, 'object_store_memory': 2147483648.0, 'node:__internal_head__': 1.0}
Tasks executed
10000 tasks on 127.0.0.1
Time taken: 10.32 secondsWhile on 2.47 and below it runs faster with no system freeze whatsoever:
...
This cluster consists of
nodes: 1
resources: {'object_store_memory': 2147483648.0, 'memory': 8400879616.0, 'node:127.0.0.1': 1.0, 'CPU': 11.0, 'node:__internal_head__': 1.0}
Tasks executed
10000 tasks on 127.0.0.1
Time taken: 2.92 seconds
Versions / Dependencies
-
M3 macbook pro.
-
ray[default]==2.48
Reproduction script
from collections import Counter
import socket
import time
import ray
ray.init()
print(
"""This cluster consists of
nodes: {}
resources: {}
""".format(
len(ray.nodes()), ray.cluster_resources()
)
)
@ray.remote
def f():
time.sleep(0.001)
# Return IP address.
return socket.gethostbyname("localhost")
start = time.time()
object_ids = [f.remote() for _ in range(10_000)]
object_ids = ray.get(object_ids)
print("Tasks executed")
for ip_address, num_tasks in Counter(object_ids).items():
print(" {} tasks on {}".format(num_tasks, ip_address))
print("Time taken: {:.2f} seconds".format(time.time() - start))Issue Severity
High: It blocks me from completing my task.