Skip to content

G-API: Introduce runtime in-graph metadata#18793

Merged
alalek merged 6 commits intoopencv:masterfrom
dmatveev:dm/in_graph_metadata
Nov 17, 2020
Merged

G-API: Introduce runtime in-graph metadata#18793
alalek merged 6 commits intoopencv:masterfrom
dmatveev:dm/in_graph_metadata

Conversation

@dmatveev
Copy link
Copy Markdown
Contributor

@dmatveev dmatveev commented Nov 12, 2020

also resolves #18773

Intro

This PR brings support of generic in-graph metadata in the G-API.
The metadata comes into graph with its input arguments and propagates through the graph then.

The main idea behind this metadata today is to tag every object with extra streaming-related info (e.g., a frame number and a timestamp) to allow app-side data synchronization when streaming::desync() feature is used (see tests).

Overview of changes

  • cv::GRunArg is not an alias over variant<> anymore, now it is a distinct type;
  • cv::GRunArg is extended with a metadata dictionary;
  • cv::GCaptureSource is extended to put the meta information to the captured frames;
  • Introduced a generic streaming::meta<T>(obj, str) operation which accesses a metadata field str of object obj and of the given type T; its return value is GOpaque<T> which can be later used in the graph;
  • Introduced an internal backend (in backends/common) which implements this generic meta<>() operation;
  • Implemented a simple metadata propagation model at the GExecutor/GStreamingExecutor level. This is the most debatable part of the patch -- now asynchronous islands have to manage and post this metadata information for outputs on their own. However, there is a default implementation for the synchronous islands (which covers 100% of our public backends).

Sporadic failures in desync* tests

There was an issue with some tests on desync failing in different configurations. The failing check compared number of frames received through the "main" path against the number of frames received through a "desynchronized" path, what in theory shouldn't happen.

The issue occurred due to the main path actually running slower than a desync path on a particular machine or in a particular build configuration. Because of that, a desync path has got its "Stop" message earlier and broadcasted it down to the output queue while the "main" path was still busy. It led to terminating the streaming pipeline earlier that expected and that "counter mismatch" problem occurred because of that.

  • I agree to contribute to the project under Apache 2 License.
  • To the best of my knowledge, the proposed patch is not based on a code under GPL or other license that is incompatible with OpenCV
  • The PR is proposed to proper branch
  • There is reference to original bug report and related work
  • There is accuracy test, performance test and test data in opencv_extra repository, if applicable
    Patch to opencv_extra has the same branch name.
  • The feature is well documented and sample code can be built with the project CMake
force_builders=Custom,Custom Win,Custom Mac
build_gapi_standalone:Linux x64=ade-0.1.1f
build_gapi_standalone:Win64=ade-0.1.1f
build_gapi_standalone:Mac=ade-0.1.1f
build_gapi_standalone:Linux x64 Debug=ade-0.1.1f

Xbuild_image:Custom=centos:7
Xbuildworker:Custom=linux-1
build_gapi_standalone:Custom=ade-0.1.1f

build_image:Custom=ubuntu-openvino-2021.1.0:18.04
build_image:Custom Win=openvino-2021.1.0
build_image:Custom Mac=openvino-2021.1.0

test_modules:Custom=gapi
test_modules:Custom Win=gapi
test_modules:Custom Mac=gapi

buildworker:Custom=linux-1
# disabled due high memory usage: test_opencl:Custom=ON
test_opencl:Custom=OFF
test_bigdata:Custom=1
test_filter:Custom=*

@dmatveev dmatveev added this to the 4.5.1 milestone Nov 12, 2020
@dmatveev dmatveev self-assigned this Nov 12, 2020
@dmatveev
Copy link
Copy Markdown
Contributor Author

@smirnov-alexey @TolyaTalamanov @rgarnov can you please have a look one more time? Thanks

Copy link
Copy Markdown
Contributor

@smirnov-alexey smirnov-alexey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good

Copy link
Copy Markdown
Contributor

@rgarnov rgarnov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good

@dmatveev
Copy link
Copy Markdown
Contributor Author

@alalek done.

@alalek alalek merged commit b866d0d into opencv:master Nov 17, 2020
@dmatveev
Copy link
Copy Markdown
Contributor Author

Fantastic!

@alalek alalek mentioned this pull request Nov 27, 2020
a-sajjad72 pushed a commit to a-sajjad72/opencv that referenced this pull request Mar 30, 2023
G-API: Introduce runtime in-graph metadata

* G-API: In-graph metadata -- initial implementation

* G-API: Finish the in-graph metadata implementation for Streaming

* G-API: Fix standalone build & warnings for in-graph metadata

* G-API: In-graph meta -- fixed review comments

* G-API: Fix issues with desync causing failing tests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

G-API: sporadic failures in GAPI_Streaming_Desync.* tests

5 participants