Currently, when spilling is triggered, clients retry indefinitely until space "shows up" in the object store.

As discovered in #11673, this simple policy doesn't work well when there are many concurrent clients spilling. Some clients might end up waiting forever for space to free up (e.g., if they are allocating larger objects, or get unlucky).

Solving this is a bit tricky given the single-threaded event loop of the plasma store. Ideally, we'd maintain a queue of client requests and process them in ~FIFO order for fairness. This might be possible to emulate without too large of a refactoring [requires design].

cc @stephanie-wang @rkooo567