Skip to content

IDLE workers holding GPU resources #10739

@acdifran

Description

@acdifran

I am very new to ray and I recently updated some code from ray 0.8.0 to 0.8.6 and am running into an issue where my workers seem to be holding onto GPU resources after this update. Previously all submitted tasks would complete, but now after a few tasks run, the remaining tasks hang. I took a look at the dashboard and see some IDLE workers that have GPU resources assigned. I set "max_calls=1" for all remote functions, but I still see these IDLE workers holding onto portions of the GPU. It was my understanding that after completing a task, the worker should free the GPU resources it was holding if max_calls is set. Am I misunderstanding how this functions?

Ray 0.8.6
Ubuntu 16.4
Python 3.6
Pytorch 1.2.0

Screen Shot 2020-09-10 at 5 02 34 PM

Metadata

Metadata

Assignees

Labels

questionJust a question :)

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions