I think there are some missing BUILD files (doesn't compile).

This is pretty cool. One thing I noticed is that

import json
import numpy as np
import ray

ray.init(
    object_store_memory=100 * 1024 * 1024,
    _system_config={
        "automatic_object_spilling_enabled": True,
        "object_spilling_config": json.dumps(
            {"type": "filesystem", "params": {"directory_path": "/tmp/spill"}},
            separators=(",", ":")
        )
    },
)

@ray.remote
def f():
    return np.zeros(10 * 1024 * 1024)

ids = []
for _ in range(10):
    x = f.remote()
    ids.append(x)

for x in ids:
    print(ray.get(x).shape)

This kind of workload will hang because the retries are waiting not enough time. I guess this will be addressed with the async RPC, but in the interim can we return a special return code to allow indefinite retries with low delay (e.g., every 10ms retry). This would allow spilling to be used for real workloads with pretty good performance.

This kind of workload will hang because the retries are waiting not enough time. I guess this will be addressed with the async RPC, but in the interim can we return a special return code to allow indefinite retries with low delay (e.g., every 10ms retry). This would allow spilling to be used for real workloads with pretty good performance.

Hmm so I tried this and it actually still has the same problem. I added a new error code for the object store being transiently out of memory, and the client will retry quickly after receiving this error code. This is to prevent indefinite retries in the case where two clients need to create an object and there are pending spills, but only one of the clients will be able to create its object after the spills complete. If we just did an infinite retry loop on the second client, we could hang and never throw OutOfMemory.

There are two issues in the script you posted, both pretty subtle:

1. Since there are multiple clients trying to create an object at the same time, only one of them will succeed after an object has been spilled. But then since all the clients retry at around the same time, the others retry while the one that succeeded is still creating its object (so its object can't be evicted or spilled). I managed to get this to work by resetting the normal OutOfMemory retries if the client ever receives the transient error, but I don't think that will work in all cases.
2. During the script, there is one client (the driver) trying to get objects. With only one IO worker, this will hang because the IO worker will try to restore the driver's object, but that requires the other object to be spilled, which also requires an IO worker. I confirmed that it works once max IO workers is set to 2.

I'll push what I have, but it seems like there's a lot more to do here.

@stephanie-wang I need more familiarity to the codebase, so it is probably not possible, but why don't we just queue up the creation requests and invoke them after object spilling requests are completed? Is it tricky to be implemented?

@stephanie-wang I need more familiarity to the codebase, so it is probably not possible, but why don't we just queue up the creation requests and invoke them after object spilling requests are completed? Is it tricky to be implemented?

It is possible, and I think we should do it, but it's a bit complicated to do right now without refactoring the plasma store. I thought it'd be better to leave it for a separate PR.