💊 CircleCI build failures summary and remediations

As of commit 42e6f50 (more details on the Dr. CI page):

Commit 42e6f50 was recently pushed. Waiting for builds...

@ngimel @mruberry Would you mind help reviewing this PR?
(Please feel free to remove if it's not a good fit)

fyi @csarofeen and @ptrblck

cc @mcarilli. I'm concerned about how cupy is going to handle streams. Memory allocated by pytorch is safe to use only on the stream it is allocated on, but since cupy and pytorch don't share cuda stream pools (?) I don't see then how this memory can be used safely without additional synchronizations. Do we expect users to put in all the necessary synchronizations?

Taking memory from the Pytorch caching allocator and handing it to some other library seems hard, and certainly isn't safe as implemented here, unless the recipient (Cupy) happens to use that memory on the same stream that the Pytorch allocator associated with the memory.

@ngimel @mruberry stating stuff you know so it's in the open, there are two issues:

• syncing the recipient's intended usage stream(s) with the stream on which Pytorch most recently populated the memory
• telling Pytorch to recordStream(memory, usage streams) so that the caching allocator doesn't free and reuse that same memory until the recipient's work is done.

In principle such usage could be made safe, if the recipient either communicated back to Pytorch (in a way Pytorch could understand) on what stream(s) the recipient intended to use the memory (at which point Pytorch could call recordStream(memory, streams)) OR the recipient queried from Pytorch (in a way the recipient could understand) a list of streams on which it was safe (according to Pytorch) to use the memory.

Hello, and thanks for all the feedback.
I am not sure which would be the better way to achieve this, probably allowing to use the underlying PyTorch streams as cupy streams could be an easy solution?.
It would require changes in cupy and the user should be aware of it.
We would like to avoid having a dependency between both libraries by using generic interfaces.

Another possible and simpler solution could be to pass the current stream being used in CuPy to the memory allocator?

The discussion is also on-going at cupy/cupy#3126
Sorry for scattering it.

I just added a PR in CuPY to allow it using external streams. So that a PyTorch managed stream can be set as the CuPy default one and events, kernels, and memory managed through that one.
I will modify this PR to explicitly tell the stream when allocating memory. This should make the approach safe.

I added the option to specify a stream.
As I said above, streams from the PyTorch pool can be shared to CuPy using cupy/cupy#3141
It is the user responsibility to use the pytorch streams in CuPy.

The following gist creates and sets CuPy streams based on the torch ones and makes a custom allocator aware of the current stream.

https://gist.github.com/emcastillo/44a033399ad67ddbdb306bed0f5fa6e0

I am not sure if this proposal makes 100% sense for you guys so any feedback is more than welcome.

Removing myself from reviewer list, lmk if I need to look.

@ngimel merged this pull request in 31cc311.