[ray] Objects are being evicted improperly #5624
Description
System information
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): local machines
- Ray installed from (source or binary): pip install -U https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-0.8.0.dev4-cp37-cp37m-manylinux1_x86_64.whl
- Ray version: 0.8.0.dev4
- Python version: 3.7
- Exact command to reproduce:
import ray
import torch
ray.init(object_store_memory=int(100e6))
@ray.remote
def identity(vectors):
return [ray.put(ray.get(vec)) for vec in vectors]
obj_id = ray.put(torch.randn(int(1e5)))
vectors = [obj_id for _ in range(200)]
while True:
vectors = ray.get(identity.remote(vectors))
Describe the problem
This code throws the following error.
2019-09-03 13:46:22,598 WARNING worker.py:1797 -- The task with ID ffffffffffffffffffff01000000 is a driver task and so the object created by ray.put could not be reconstructed.
(pid=38222) 2019-09-03 13:46:23,308 INFO worker.py:432 -- The object with ID ObjectID(7d58f415c89effffffff0100000000c001000000) already exists in the object store.
2019-09-03 13:46:28,320 ERROR worker.py:1737 -- Possible unhandled error from worker: ray_worker (pid=38222, host=atlas)
ray.exceptions.UnreconstructableError: Object ffffffffffffffffffff01000000008002000000 is lost (either LRU evicted or deleted by user) and cannot be reconstructed. Try increasing the object store memory available with ray.init(object_store_memory=<bytes>) or setting object store limits with ray.remote(object_store_memory=<bytes>). See also: https://ray.readthe
docs.io/en/latest/memory-management.html
However, if you replace the definition of obj_id with
obj_id = ray.put(list(range(int(1e5)))) then we get the correct error which @ericl 's recent PR added, or if you replace the definition with obj_id = torch.randn(int(1e5)):
(pid=46751) 2019-09-03 13:56:21,919 INFO worker.py:2381 -- Put failed since the value was either too large or the store was full of pinned objects. If you are putting and holding references to a lot of object ids, consider ray.put(value, weakref=True) to allow object data to be evicted early.
However, neither error should be raised -we have only 80 MB of objects and the object store has 100 MB capacity.