-
Notifications
You must be signed in to change notification settings - Fork 7.4k
Closed
Labels
questionJust a question :)Just a question :)
Description
I am very new to ray and I recently updated some code from ray 0.8.0 to 0.8.6 and am running into an issue where my workers seem to be holding onto GPU resources after this update. Previously all submitted tasks would complete, but now after a few tasks run, the remaining tasks hang. I took a look at the dashboard and see some IDLE workers that have GPU resources assigned. I set "max_calls=1" for all remote functions, but I still see these IDLE workers holding onto portions of the GPU. It was my understanding that after completing a task, the worker should free the GPU resources it was holding if max_calls is set. Am I misunderstanding how this functions?
Ray 0.8.6
Ubuntu 16.4
Python 3.6
Pytorch 1.2.0
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
questionJust a question :)Just a question :)
