Description:

Description as title.

• Particularly useful for the scenario where-in custom ops compiled into shared libraries need to achieve implicit synchronization with ORT's CUDA kernels

  • Also useful for the "regular" custom ops scenario where custom ops are not compiled into a shared library and directly registered used the APIs. But that had a work-around of creating a session with a user-created stream and just using that stream to launch custom kernels. This work-around is much harder to achieve when custom ops are compiled into shared libraries (See Update CUDA custom op unit tests to account for recent ORT change #6971)

• Currently, we only have one compute stream per-session. So, this could be a session level API. But it is kept as an API at the OrtKernelContext level to keep the design flexible enough for the case where-in (in future) sessions could have multiple streams (one per host thread). When the sessions starts maintaining one stream per host thread, the API will start returning the stream corresponding to that thread.

Motivation and Context
#7068 (comment)