G-API: oneVPL Implement asynchronous MFP demux data provider#21022
G-API: oneVPL Implement asynchronous MFP demux data provider#21022alalek merged 30 commits intoopencv:4.xfrom
Conversation
|
Please reuse CI instructions from #20926 to enable oneVPL builder configuration |
modules/gapi/include/opencv2/gapi/streaming/onevpl/data_provider_interface.hpp
Outdated
Show resolved
Hide resolved
| */ | ||
| virtual size_t fetch_data(size_t out_data_bytes_size, void* out_data) = 0; | ||
|
|
||
| #ifdef HAVE_ONEVPL |
There was a problem hiding this comment.
I don't think that using ifdefs in a public header works. Please, remove the guard and handle it in the sources
There was a problem hiding this comment.
Have you added #else with assert? I think other reviewers need to check on that
modules/gapi/src/streaming/onevpl/demux/async_mfp_demux_data_provider.cpp
Outdated
Show resolved
Hide resolved
modules/gapi/src/streaming/onevpl/demux/async_mfp_demux_data_provider.cpp
Outdated
Show resolved
Hide resolved
|
|
||
| // close reader | ||
| provider_state.store(State::Exhausted); | ||
| buffer_storage_non_empty_cond.notify_all(); |
There was a problem hiding this comment.
Will notify_one work here?
There was a problem hiding this comment.
i had idea to upgrade fetch_bitstream for multithreading version, if required. but not for now.
Is there any drawback for notify all instead of one?
There was a problem hiding this comment.
I don't think more than one thread should wake in your case, since others will be still blocked if I get it correctly. There is no major difference though. Feel free to resolve this comment
There was a problem hiding this comment.
i do not want to debug code to find when it get stucked when something changed, semantically close reader means broadcast
modules/gapi/src/streaming/onevpl/demux/async_mfp_demux_data_provider.cpp
Show resolved
Hide resolved
modules/gapi/src/streaming/onevpl/demux/async_mfp_demux_data_provider.cpp
Outdated
Show resolved
Hide resolved
modules/gapi/src/streaming/onevpl/demux/async_mfp_demux_data_provider.cpp
Outdated
Show resolved
Hide resolved
modules/gapi/src/streaming/onevpl/demux/async_mfp_demux_data_provider.hpp
Show resolved
Hide resolved
| GAPI_LOG_WARNING(nullptr, "[" << this << "] " << | ||
| "\"mfxImplDescription.mfxDecoderDescription.decoder.CodecID\" " | ||
| "is absent, total param count" << codec_params.size()); | ||
| return; |
There was a problem hiding this comment.
must throw unsupported exception instead of return
| try { | ||
| provider = std::make_shared<MFPAsyncDemuxDataProvider>(file_path); | ||
| GAPI_LOG_INFO(nullptr, "MFP data provider created"); | ||
| } catch (const DataProviderUnsupportedException& ex) { |
There was a problem hiding this comment.
it is supposed to throw Unsupported exception at least. Any other kind of exception must be considered as unpredictable error and must be not catched here but deliver to application level (missing file, access rights etc)
modules/gapi/src/streaming/onevpl/demux/async_mfp_demux_data_provider.cpp
Outdated
Show resolved
Hide resolved
| if (!is_stream_selected) { | ||
| GAPI_LOG_WARNING(nullptr, "[" << this << "] couldn't find video stream with supported params"); | ||
| // TODO print codec supported list | ||
| throw DataProviderUnsupportedException("couldn't find supported video stream"); |
There was a problem hiding this comment.
Need to print it ultimately
There was a problem hiding this comment.
stream is suppose :)
if only audio stream is present in container then it is unsupported stream, not codec
modules/gapi/src/streaming/onevpl/demux/async_mfp_demux_data_provider.hpp
Show resolved
Hide resolved
|
|
||
| TEST_P(OneVPL_Source_MFPDispatcherTest, open_and_decode_file) | ||
| { | ||
| if (!initTestDataPathSilent()) { |
There was a problem hiding this comment.
must be removed as agreement of opecv test implementation
| } | ||
| using namespace cv::gapi::wip::onevpl; | ||
|
|
||
| source_t path = findDataFile(std::get<0>(GetParam()), false); |
There was a problem hiding this comment.
must be always true
modules/gapi/src/streaming/onevpl/demux/async_mfp_demux_data_provider.cpp
Outdated
Show resolved
Hide resolved
smirnov-alexey
left a comment
There was a problem hiding this comment.
Overall looks good to me. Please, seek additional reviewers
AsyaPronina
left a comment
There was a problem hiding this comment.
Part of files are only reviewed, will proceed tomorrow
| * declare untagged struct mfxBitstream. | ||
| * | ||
| * IDataProvider makes sense only for HAVE_VPL is ON and to keep IDataProvider | ||
| * interface API/ABI compiant between core library and user application layer |
| * @return true for fetched data, false on EOF and throws exception on error | ||
| */ | ||
| virtual size_t fetch_data(size_t out_data_bytes_size, void* out_data) = 0; | ||
| virtual bool fetch_bitstream_data(std::shared_ptr<mfx_bitstream> &in_out_bitsream) = 0; |
There was a problem hiding this comment.
it is intended to be not copied, isn't it?
There was a problem hiding this comment.
it's filled by Microsoft Foundation Primitives implementation there is no clear point: copy or not - it depends.
Anyway, this huge work happens in different thread: MFP async call, but current thread which is caller of fetch_bitstream_data just pulls for cooked up bitstream
| #ifndef GAPI_STREAMING_ONEVPL_ONEVPL_DATA_PROVIDER_INTERFACE_HPP | ||
| #define GAPI_STREAMING_ONEVPL_ONEVPL_DATA_PROVIDER_INTERFACE_HPP | ||
| #include <exception> | ||
| #include <limits> |
There was a problem hiding this comment.
it was, but no anymore. will be deleted
| #else // HAVE_ONEVPL | ||
| struct IDataProvider::mfx_bitstream { | ||
| mfx_bitstream() { | ||
| GAPI_Assert(false && "Reject to create `mfxBitstream` till library compiled without VPL/MFX support"); |
There was a problem hiding this comment.
Might be to replace till with more common word - because?
| namespace onevpl { | ||
|
|
||
| IDataProvider::Ptr DataProviderDispatcher::create(const std::string& file_path, | ||
| const std::vector<CfgParam> cfg_params) { |
There was a problem hiding this comment.
thanks, i missed it
|
|
||
| ComPtrGuard<IMFStreamDescriptor> stream_descriptor = | ||
| createCOMPtrGuard(stream_descriptor_tmp); | ||
| is_stream_selected = false; // deselect until find supported stream found |
| " - " << is_codec_supported) | ||
| } else { | ||
| GAPI_LOG_WARNING(nullptr, "[" << this << "] " << | ||
| "Cannot media GUID subtype for stream by index: " << |
| if (!is_stream_selected) { | ||
| GAPI_LOG_WARNING(nullptr, "[" << this << "] couldn't find video stream with supported params"); | ||
| // TODO print codec supported list | ||
| throw DataProviderUnsupportedException("couldn't find supported video stream"); |
| hr = MFCreateSourceReaderFromMediaSource(source.get(), attributes.get(), | ||
| &source_reader_tmp); | ||
| if (FAILED(hr)) { | ||
| throw DataProviderSystemErrorException(HRESULT_CODE(hr), "Cannot create MFCreateSourceReaderFromMediaSource"); |
There was a problem hiding this comment.
"Cannot create MFCreateSourceReaderFromMediaSource" -> "Cannot create IMFSourceReader"
| // Ask for the first sample. | ||
| request_next(hr, 0, 0); | ||
| } else { | ||
| throw DataProviderSystemErrorException(HRESULT_CODE(hr), "Cannot ReadSample"); |
There was a problem hiding this comment.
"Cannot ReadSample"" or something which is already handled on the 457 line?
if (FAILED(hr)) {
throw DataProviderSystemErrorException(HRESULT_CODE(hr), "Cannot create MFCreateSourceReaderFromMediaSource");
This is confusing. How can we reach this else statement at all?
|
UTs failed on CI, waiting #20995 to be merged |
| std::string reason; | ||
| }; | ||
|
|
||
| struct GAPI_EXPORTS DataProviderUnsupportedException : public DataProviderException { |
There was a problem hiding this comment.
i don't know at the moment, maybe
| virtual const char* what() const noexcept override; | ||
|
|
||
| private: | ||
| std::string reason; |
There was a problem hiding this comment.
Should reason be part of DataProviderException?
There was a problem hiding this comment.
two commenters ask the same: then ok will move it to base
| GAPI_Assert(false && "Reject to create `mfxBitstream` because library compiled without VPL/MFX support"); | ||
| } | ||
| }; | ||
| #endif // HAVE_ONEVPL |
There was a problem hiding this comment.
should the whole file body be under one big #ifdef?
There was a problem hiding this comment.
I believe not, because mfxBitsream is VPL specific type and should not be exposed in public headers
so for NO_VPL configuration we have to have stub
Otherwise, it would have namespace-open-close too much
There was a problem hiding this comment.
Another way which I considered before - it move all IDataProvider interface declaration & definition into lib private scope. but want to preserve customization from public scope in data provider implementaiton
There was a problem hiding this comment.
In NO_VPL this whole streaming/onevpl directory is excluded isn't it?
There was a problem hiding this comment.
actually no - it activates some stub-like public classed which throws assert on using
here is just wrapper on VPL bitstream which figured as public interface item
|
|
||
| // Look-up CodecId from input params | ||
| // If set then raw data provider is preferred | ||
| GAPI_LOG_DEBUG(nullptr, "try find explicit cfg param\"mfxImplDescription.mfxDecoderDescription.decoder.CodecID\""); |
There was a problem hiding this comment.
should this sting be defined only once somewhere?
There was a problem hiding this comment.
will add in similar way as queue_capacity in future PRs
https://github.com/opencv/opencv/pull/21049/files#diff-3b97bff710e57efd4529da47be268daad0dddeeda82225e6354c18b93808b786R63
| if (FAILED(hr)) { | ||
| GAPI_LOG_WARNING(nullptr, "Cannot create MF_RESOLUTION_MEDIASOURCE from URI: " << | ||
| url << ". Abort"); | ||
| pSourceResolver->Release(); |
There was a problem hiding this comment.
yes, thanks it was preserved from prototype version
| if (!out_bitstream) { | ||
| out_bitstream = std::make_shared<mfx_bitstream>(); | ||
| out_bitstream->MaxLength = bitstream_data_size; | ||
| out_bitstream->Data = (mfxU8 *)calloc(out_bitstream->MaxLength, sizeof(mfxU8)); |
There was a problem hiding this comment.
if mxf_bitstream structure is yours, I'd avoid unnecessary allocations outside of it
If it is not, I'd find a way to associate an automatic buffer with a wrapper over this structure and set Data to that buffer pointer.
There was a problem hiding this comment.
second option:
it allocates at once for whole source, effort is more that profit
There was a problem hiding this comment.
allocating it once for the whole source changes the overall allocation model, what I ask for above is just making what's now automatic
There was a problem hiding this comment.
I agree, that it would be quite worth for regular usage, but otherwise it doesn't use any pre-allocation in case of async demux, because it uses mapping from MFP buffers, so byte-receiving constructor would be useless for this case.
From my perspective, implementing smart-class mxf_bitstream for both cases looks redundant at the moment and should be done by demand when it would be used more than in 1-2 places
| throw std::runtime_error("Cannot allocate bitstream.Data bytes: " + | ||
| std::to_string(out_bitstream->MaxLength * sizeof(mfxU8))); |
There was a problem hiding this comment.
..so using new could save you from extra handling like this
| GAPI_LOG_DEBUG(nullptr, "bitstream after fetch, DataOffset: " << out_bitstream->DataOffset << | ||
| ", DataLength: " << out_bitstream->DataLength); |
There was a problem hiding this comment.
do we need that extensive debug logging here?
There was a problem hiding this comment.
i used it to find no fully drained data problem and unconsumed bytes
In release is no Debug anyway isn't? moreover if(level==Debug) is nothing with syscall fread time
| source_handle(nullptr, &fclose), | ||
| codec(std::numeric_limits<mfx_codec_id_type>::max()), | ||
| bitstream_data_size(bitstream_data_size_value) { | ||
| GAPI_Assert(false && "Unsupported: G-API compiled without `WITH_GAPI_ONEVPL=ON`"); |
There was a problem hiding this comment.
in fact it is OpenCV compiled, maybe we should fix this message text elsewhere
There was a problem hiding this comment.
it make sense. Will rename it when WITH_GAPI_ONEVPL tnasfrormed into WITH_MFX due to custom builders turning on
| const char* mfx_codec_id_to_cstr(mfxU32 mfx_id) { | ||
| switch(mfx_id) { | ||
| case MFX_CODEC_AVC: | ||
| return "MFX_CODEC_AVC"; | ||
| case MFX_CODEC_HEVC: | ||
| return "MFX_CODEC_HEVC"; | ||
| case MFX_CODEC_MPEG2: | ||
| return "MFX_CODEC_MPEG2"; | ||
| case MFX_CODEC_VC1: | ||
| return "MFX_CODEC_VC1"; | ||
| case MFX_CODEC_VP9: | ||
| return "MFX_CODEC_VP9"; | ||
| case MFX_CODEC_AV1: | ||
| return "MFX_CODEC_AV1"; | ||
| case MFX_CODEC_JPEG: | ||
| return "MFX_CODEC_JPEG"; | ||
| default: | ||
| return "<unsupported>"; | ||
| } | ||
| } |
There was a problem hiding this comment.
A simple define cuts this code in half
There was a problem hiding this comment.
let's waiting for DX11 PR...
| worker_key_to_buffer_mapping_storage.clear(); | ||
| } | ||
|
|
||
| GAPI_LOG_INFO(nullptr, "Clean didn't used up storage, count: " << |
There was a problem hiding this comment.
"Clean didn't used up storage, count: " -> "Clean up didn't used storage, count: " ?
There was a problem hiding this comment.
use up -> phrasal vers -> drain to empty
Ok, let's make it quite simple:
Clean data storage, elapsed count:
| // Get the video frame buffer from the sample. | ||
| IMFMediaBuffer *buffer_ptr = nullptr; | ||
| hr = sample_ptr->ConvertToContiguousBuffer(&buffer_ptr); | ||
| if (FAILED(hr)) { |
| IMFMediaBuffer *buffer_ptr = nullptr; | ||
| hr = sample_ptr->ConvertToContiguousBuffer(&buffer_ptr); | ||
| if (FAILED(hr)) { | ||
| GAPI_Assert(SUCCEEDED(hr) && "MFPAsyncDemuxDataProvider::OnReadSample"); |
There was a problem hiding this comment.
Or you just may write GAPI_Assert(false && ...)
Message shouldn't contain a lot of details, am I right?
|
|
||
| hr = buffer_ptr->Lock(&staging_stream->Data, &max_buffer_size, &curr_size); | ||
| if (FAILED(hr)) { | ||
| GAPI_Assert(SUCCEEDED(hr) && "MFPAsyncDemuxDataProvider::OnReadSample"); |
There was a problem hiding this comment.
You used SUCCEEDED(hr) just for double-check?
Thing that the message is the same as in error above is intended?
There was a problem hiding this comment.
it seems like copy-paste. thanks
| "iteration: " << iterations); | ||
| } else { | ||
| GAPI_LOG_DEBUG(nullptr, "[" << this << "] is waiting for flush finishing, " | ||
| "iteration: " << iterations); |
| using file_ptr = std::unique_ptr<FILE, decltype(&fclose)>; | ||
| FileDataProvider(const std::string& file_path); | ||
| FileDataProvider(const std::string& file_path, | ||
| const std::vector<CfgParam> codec_params = {}, |
There was a problem hiding this comment.
const std::vector<CfgParam>&
|
|
||
| const char* mfx_codec_id_to_cstr(mfxU32 mfx_id); | ||
|
|
||
| const std::set<mfxU32> &get_supported_mfx_codec_id(); |
There was a problem hiding this comment.
I would write get_supported_mfx_codec_ids, but it is up to you
| } | ||
|
|
||
| EXPECT_TRUE(dd_result); | ||
| EXPECT_TRUE(std::dynamic_pointer_cast<MFPAsyncDemuxDataProvider>(provider_ptr)); |
| size_t total_produced_count = 0; | ||
| for (size_t i = 0; i < produce_buffer_count; i ++) { | ||
| std::shared_ptr<IDataProvider::mfx_bitstream> dummy_stream = | ||
| std::make_shared<IDataProvider::mfx_bitstream>(); |
| } | ||
|
|
||
| class OneVPL_Source_MFPAsyncDispatcherTest : public ::testing::TestWithParam<array_element_t> {}; | ||
| TEST_P(OneVPL_Source_MFPAsyncDispatcherTest, open_and_decode_file) |
There was a problem hiding this comment.
I don't understand differences with the first OneVPL_Source_MFPDispatcherTest :(
Could you please help?
There was a problem hiding this comment.
retained from prev version for sync demux
G-API: oneVPL Implement asynchronous MFP demux data provider * Add dummy dmux * Initial commit for draft versionn * Demux for low res file works * Add media source resolver to work over incorrect MIME * Add MFP Demux logger * stash changes * Extend IDataProvider with CodecId, Add troubleshooting info * Add IDapaProvider dispatcher * Add ComPtrGuard wrappers * Add new unit test scope for MFP demux & Add minor changes * Enhance UTs * Remove ATL header * Remove ATL another one * Fix build * Add static for some methods * Initial commit * Add async demuxing * Apply tdd idea * Intro IDataProvider changes: +fetch_bitstream, -fetch_data * Fix UTs * Remove IDataProvider::CodecId & Fix EOF hang * Remove sync demux * Remove mfp async dependencies * Remove VPL dependencies from IDataProvider declaration * Apply comments * Fix compilation * Suppress unused warning * Apply some comments * Apply some comments * Apply comments
This PR is descendant of sync version:
#20926
the
IDataProviderbase interface changed: fetch_data was improved from raw memory to mfx bitstream, CodecId is not requried in this casePull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
Patch to opencv_extra has the same branch name.
Build Configuration