Permanently pin object IDs that are captured in functions or serialized out of band#7358
Permanently pin object IDs that are captured in functions or serialized out of band#7358edoakes merged 2 commits intoray-project:masterfrom
Conversation
|
Can one of the admins verify this patch? |
|
Test FAILed. |
| # then pin the object for the lifetime of this worker by adding | ||
| # a local reference that won't ever be removed. | ||
| ray.worker.get_global_worker( | ||
| ).core_worker.add_object_id_reference(obj) |
There was a problem hiding this comment.
Instead of modifying the serialization tracking, could we instead call get_and_clear_contained_object_ids() at the end of pickling a function definition, and add all of those to globals?
There was a problem hiding this comment.
I don't think that solution would be enough, since ObjectIDs can also get arbitrarily pickled by other parts of the user's code.
| # then pin the object for the lifetime of this worker by adding | ||
| # a local reference that won't ever be removed. | ||
| ray.worker.get_global_worker( | ||
| ).core_worker.add_object_id_reference(obj) |
There was a problem hiding this comment.
I don't think that solution would be enough, since ObjectIDs can also get arbitrarily pickled by other parts of the user's code.
Why are these changes needed?
ObjectIDs can't be properly reference counted when they're used in unsupported ways (e.g., serialized directly with cloudpickle or captured in a function/actor). This PR permanently increases their reference count when serialized, effectively pinning the object until the offending worker/driver disconnects.Related issue number
Checks
scripts/format.shto lint the changes in this PR.