Allow access to CUDA pointers for interoperability with other libraries by pwuertz · Pull Request #16513 · opencv/opencv

pwuertz · 2020-02-05T15:34:10Z

This is a proposal for adding CV_WRAP compatible cudaPtr() getter methods to GpuMat and Stream, required for enabling interoperability between OpenCV and other CUDA supporting python libraries like Numba, CuPy, PyTorch, etc.

Here is an example for sharing a GpuMat with CuPy:

import cv2 as cv
import cupy as cp

# Create GPU array with OpenCV
data_gpu_cv = cv.cuda_GpuMat()
data_gpu_cv.upload(np.eye(64, dtype=np.float32))

# Modify the same GPU array with CuPy
data_gpu_cp = cp.asarray(CudaArrayInterface(data_gpu_cv))
data_gpu_cp *= 42.0

# Download and verify
assert np.allclose(data_gpu_cp.get(), np.eye(64) * 42.0)

In this example CudaArrayInterface is a (incomplete) adapter class that implements the cuda array interface used by other frameworks:

class CudaArrayInterface:
    def __init__(self, gpu_mat):
        w, h = gpu_mat.size()
        type_map = {
            cv.CV_8U: "u1", cv.CV_8S: "i1",
            cv.CV_16U: "u2", cv.CV_16S: "i2",
            cv.CV_32S: "i4",
            cv.CV_32F: "f4", cv.CV_64F: "f8",
        }
        self.__cuda_array_interface__ = {
            "version": 2,
            "shape": (h, w),
            "data": (gpu_mat.cudaPtr(), False),
            "typestr": type_map[gpu_mat.type()],
            "strides": (gpu_mat.step, gpu_mat.elemSize()),
        }

If possible, I'd like to implement __cuda_array_interface__ within the GpuMat python binding in a future PR (not sure how to define a python property using the wrapper generator though).

force_builders=Custom
buildworker:Custom=linux-4
build_image:Custom=ubuntu-cuda:18.04

leofang · 2020-02-07T15:03:43Z

Hi @pwuertz, thanks for joining the discussion on Numba. I don't have comment for your effort here (yet), but I'd like to kindly ask that when this PR (and any subsequent ones) is merged, it'd be nice if you could follow numba/numba#5104 and add opencv to the list 🙂 Thank you.

alalek · 2020-02-10T13:07:15Z

CudaArrayInterface

I believe it should own upstream gpu_mat object in its fields (extend lifetime of data_gpu_cv in example).

pwuertz · 2020-02-10T22:44:47Z

@alalek Please note that class CudaArrayInterface is not part of this PR. The PR was meant to provide the minimum requirement for any kind of interoperability in python, which is the access to the CUDA pointers.

For out-of-the-box interoperability with other libraries like Numba and Cupy I was planning on implementing the CUDA array interface (CAI) in a follow-up PR. As described earlier, I'm having trouble figuring out some OpenCV python binding generator details though.

Also note that within the current CAI specification (version 2), the responsibility for lifetime and synchronization resides with the user (similar to using cv::Mat constructors with data pointers). If this changes in some future version I'd be willing to update the CAI version on the OpenCV side of course.

asmorkalov · 2020-02-14T07:04:01Z

modules/core/include/opencv2/core/cuda.hpp

    operator bool_type() const;

+    //! return Pointer to CUDA stream
+    CV_WRAP void* cudaPtr() const;


There is StreamAccessor to do it: https://docs.opencv.org/master/d6/df1/structcv_1_1cuda_1_1StreamAccessor.html. I think it's better to wrap accessor rather expose private fields.

StreamAccessor

Not sure that this can help to reach the final goal:

to implement __cuda_array_interface__ within the GpuMat python binding

asmorkalov · 2020-02-14T07:11:37Z

modules/core/include/opencv2/core/cuda.inl.hpp

+inline
+void* GpuMat::cudaPtr() const
+{
+    return data;


There is CV_PROP_RW macro that allows to expose object properties to Python and other languages. You do not need own method for it.

I believe current approach is fine.
We should not allow changing of this pointer through CV_PROP_RW.

pwuertz · 2020-02-14T17:59:14Z

@asmorkalov

I tried adding uchar* data as CV_PROP, but it doesn't compile. CV_PROP apparently doesn't support pointer types.
Like @alalek, I figured that it shouldn't be allowed to change the data pointer via public interface (neither from C++ nor from python), so having/promoting a getter method for it seems like a good thing.
The GpuMat::data pointer looks identical to Mat::data (name, uint8_t*, doc), yet it represents something completely different. I think void* cudaPtr() is a good name for what this pointer represents - an opaque handle to the CUDA array, not a data pointer for reading / writing uint8_t.

pwuertz · 2020-02-14T18:17:20Z

@alalek It's true that __cuda_array_interface__ currently does not specify stream handling, but stream interoperability really is a low hanging fruit. With access to the CUDA pointers, I did the following:

# allocated dataX_cpu, dataX_gpu_cv, stream_cv
# (proof of principle numba views dataX_gpu_nb, stream_nb)
t1 = time.time()
data1_gpu_cv.upload(arr=data1_cpu, stream=stream_cv)         # OpenCV upload
kernel_nb(data1_gpu_nb, out=data2_gpu_nb, stream=stream_nb)  # Numba operation
data2_gpu_cv.download(dst=data2_cpu, stream=stream_cv)       # OpenCV download
t2 = time.time()  # CPU time: 0.001 s
stream_nb.synchronize()
t3 = time.time()  # GPU time: 0.042 s
# verified data2_cpu

So you can freely mix OpenCV and Numba operations on a single, fully async stream.

@asmorkalov Oh sorry, haven't noticed the StreamAccessor class before. I'll try adding wrapper definitions to it. This means that cudaStream_t needs some kind of globally defined conversion rule too?

pwuertz · 2020-02-14T20:45:09Z

@asmorkalov StreamAccessor uses the cudaStream_t typedef, thus having a hard dependency on the CUDA SDK. I assume this is the reason for keeping it separate from Stream, which provides a public interface without CUDA dependency.

Even with HAVE_CUDA defined, neither misc/python headers nor cv2.cpp are able to include <cuda_runtime.h>.

How to proceed? Using void* for transporting CUDA stream pointers appears to be the least intrusive solution..

alalek

Looks good to me 👍

pwuertz mentioned this pull request Feb 6, 2020

Specify synchronization semantics of CUDA Array Interface numba/numba#5162

Merged

asmorkalov requested changes Feb 14, 2020

View reviewed changes

asmorkalov added the pr: Discussion Required label Feb 14, 2020

Allow access to CUDA memory and stream pointers for interoperability.

53c0189

alalek approved these changes Mar 4, 2020

View reviewed changes

asmorkalov approved these changes Mar 5, 2020

View reviewed changes

alalek assigned asmorkalov Mar 5, 2020

opencv-pushbot merged commit 90a4d67 into opencv:master Mar 5, 2020

gigony mentioned this pull request Jul 8, 2022

[QST] Is it possible to exchange data on GPU with OpenCV CUDA? rapidsai/cucim#329

Open

cudawarped mentioned this pull request Mar 17, 2023

cuda: Add bindings to allow GpuMat and Stream objects to be initialized from memory initialized in other libraries #23371

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Allow access to CUDA pointers for interoperability with other libraries#16513

Allow access to CUDA pointers for interoperability with other libraries#16513
opencv-pushbot merged 1 commit intoopencv:masterfrom
pwuertz:cuda_py_interop

pwuertz commented Feb 5, 2020 •

edited by alalek

Loading

Uh oh!

leofang commented Feb 7, 2020

Uh oh!

alalek commented Feb 10, 2020

Uh oh!

pwuertz commented Feb 10, 2020

Uh oh!

asmorkalov Feb 14, 2020

Uh oh!

alalek Feb 14, 2020

Uh oh!

asmorkalov Feb 14, 2020

Uh oh!

alalek Feb 14, 2020

Uh oh!

pwuertz commented Feb 14, 2020

Uh oh!

pwuertz commented Feb 14, 2020

Uh oh!

pwuertz commented Feb 14, 2020

Uh oh!

alalek left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

pwuertz commented Feb 5, 2020 • edited by alalek Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

leofang commented Feb 7, 2020

Uh oh!

alalek commented Feb 10, 2020

Uh oh!

pwuertz commented Feb 10, 2020

Uh oh!

asmorkalov Feb 14, 2020

Choose a reason for hiding this comment

Uh oh!

alalek Feb 14, 2020

Choose a reason for hiding this comment

Uh oh!

asmorkalov Feb 14, 2020

Choose a reason for hiding this comment

Uh oh!

alalek Feb 14, 2020

Choose a reason for hiding this comment

Uh oh!

pwuertz commented Feb 14, 2020

Uh oh!

pwuertz commented Feb 14, 2020

Uh oh!

pwuertz commented Feb 14, 2020

Uh oh!

alalek left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

pwuertz commented Feb 5, 2020 •

edited by alalek

Loading