Implement Camera Multiplexing API by mpashchenkov · Pull Request #15100 · opencv/opencv

mpashchenkov · 2019-07-19T15:28:20Z

Added static function waitAny( ..., ..., ... ) to work with multiple cameras in one thread. The method accepts a collection of VideoCapture pointers, state collection ( 1 - ready, -1 - error, 0 - not ready ), timeout. It is implemented to find the cameras states for V4L.
Method camerasPoll( IVideoCupture --cast>> CvCupture, ..., ... ) calls function deviceHandlePoll( deviceHandle, ..., ... ). Method uses poll( ..., ..., ... ) function to find camera's state.
User recevies a collection of states for each camera.

Proposal: camera multiplexing API.
Multiplex camera API demo for tests.

Device details: Intel® Core™ i7-8700 CPU @ 3.20GHz × 12.
Cameras: Logitech C922 Pro Stream Webcam.

FPS test:
Parameters:
Net: Face-detection-retail-0004;
Size: Reshape (1, 3, 1000, 1000);
Launch time: 25 ms.

Method read()

Requests	2	3	4	5	6	7
2 cameras	27.1322	26.9473	26.9839	26.8831	27.0233	-------
3 cameras	-------	27.3065	27.1696	27.2726	27.3191	27.3035
4 cameras	-------	-------	27.2271	27.4342	26.624	27.0994

Method WaitAny()

Requests	2	3	4	5	6	7
2 cameras	30.9216	30.7505	29.5278	29.876	30.8151	-------
3 cameras	-------	29.9616	30.1515	30.8857	29.981	29.9445
4 cameras	-------	-------	30.69	30.9046	31.1103	30.5051

Parameters:
Net: Face-detection-adas-0001;
Size: (1, 3, 384, 672);
Launch time: 6.16 ms.

Method read()

Requests	2	3	4	5	6	7
2 cameras	81.3524	84.9231	83.3405	85.006	85.1356	-------
3 cameras	-------	86.8687	84.2587	87.7585	84.6507	84.6306
4 cameras	-------	-------	88.7628	85.0765	89.5895	85.2633

Method WaitAny()

Requests	2	3	4	5	6	7
2 cameras	85.8336	89.1905	86.3272	85.7945	89.0111	-------
3 cameras	-------	86.8968	84.0484	88.4938	85.3782	88.2053
4 cameras	-------	-------	83.4528	87.8218	88.0895	87.1574

mshabunin

Do you have an example of function usage. Ideally as a test case here (disabled by default): https://github.com/opencv/opencv/blob/master/modules/videoio/test/test_camera.cpp

And I think this fix changes too large part of public API, while it can be enough to add static function waitAny and several methods to CvCaptureCAM_V4L class.

modules/videoio/src/cap_interface.hpp

modules/videoio/test/test_camera.cpp

mshabunin

Did you manage to run these tests? Do they pass for you?

mshabunin · 2019-07-25T11:53:27Z

modules/videoio/test/test_camera.cpp

+TEST_P(DISABLED_videoio_timeout, v4l_poll_timeout)
+{
+   //Two identical cameras
+   VideoCapture cap1(0);


Please add explicit backend parameter to all constructors: cap(0, CAP_V4L)

mshabunin · 2019-07-25T12:31:51Z

modules/videoio/perf/perf_input.cpp

+            {
+                if(state[i] == CAP_CAM_READY)
+                {
+                  EXPECT_TRUE(VCM[i].retrieve(forMAt[i]));


Perhaps performance tests do not make much sense in this case. Standalone benchmark application (in samples/cpp dir) would be more useful. Lets say it could be launched in two modes: sync and async, and will perform some processing on frames (e.g. Canny) and measure performance and efficiency metrics. Take a look at the videocapture_camera.cpp sample application, it can be used as a starting point.

function waitAny Add errors catcher Stub for Python added. Sifting warnings One test added Two tests for camera and Perf tests added

Hardcode84 · 2019-07-29T14:48:48Z

modules/videoio/src/cap.cpp

+        {
+            if(!(bcknd == v_captures[bcknd_num].icap->getCaptureDomain()))
+            {
+                CV_Error(Error::StsError, "Backend equality expected");


Something like "All captures must have same backend"

Hardcode84 · 2019-07-29T14:49:06Z

modules/videoio/src/cap.cpp

+            return true;
+    }
+    else
+        CV_Error(Error::StsError, "Vector is empty");


Something like "capture list is empty"

Hardcode84 · 2019-07-29T14:49:30Z

modules/videoio/src/cap.cpp

+    {
+        int bcknd = v_captures[0].icap->getCaptureDomain();
+
+        for (unsigned int bcknd_num = 0; bcknd_num < v_captures.size(); ++bcknd_num)


use size_t

from 1

Hardcode84 · 2019-07-29T14:51:20Z

modules/videoio/src/cap.cpp

+    return false;
+}
+
+bool VideoCapture::waitAnyInterior(std::vector<VideoCapture>& v_captures, std::vector<int>& state,  const int & timeout)


IMO better to remove this function and move this logic into waitAny

Hardcode84 · 2019-07-29T14:52:36Z

modules/videoio/src/cap_interface.hpp

    virtual bool setProperty(int, double) { return 0; }
    virtual bool grabFrame() { return true; }
+
+    virtual int getDeviceHandle() { return 0; }


Device handles are implementation detail, remove from common interface

Hardcode84 · 2019-07-29T14:53:20Z

modules/videoio/src/videoio_c.cpp


+CV_IMPL bool cvCamerasPoll(const std::vector<CvCapture*>& pointers, std::vector<int>& state, const int64_t & timeout)
+{
+    return pointers[0] ? pointers[0]->camerasPoll(pointers, state, timeout) : 0;


false instead of 0

Hardcode84 · 2019-07-29T14:55:47Z

modules/videoio/test/test_camera.cpp

+   Mat frame1;
+
+   //false start
+   cap1>>frame1;


Test for read success in all tests

Hardcode84 · 2019-07-29T14:57:15Z

modules/videoio/perf/perf_input.cpp

+    Mat frame1;
+    int ITERATION_COUNT = 500;
+
+    //false start


Why is this needed? Write some meaningful comment

Hardcode84 · 2019-07-29T14:58:21Z

Need test for "waitAny" with single VIdeoCapture.
Also nice to have tests for errors handling.

Hardcode84 · 2019-07-29T15:00:57Z

modules/videoio/src/cap.cpp

+            }
+        }
+
+        State.resize(v_captures.size());


Instead of resizing better just check size and bail with error on mismatch

…ted, getDeviceHandle() deleted

Hardcode84 · 2019-07-31T13:02:02Z

modules/videoio/include/opencv2/videoio.hpp

    CV_WRAP virtual bool grab();

+
+    CV_WRAP static bool waitAny(std::vector<VideoCapture>&, std::vector<int>&, const int &);


just int instead of const int &

add names

use int64

Hardcode84 · 2019-07-31T13:03:01Z

modules/videoio/perf/perf_input.cpp

+#else
+    char* datapath_dir = OPENCV_TEST_CAMERA_LIST;
+#endif
+     ASSERT_FALSE(datapath_dir == nullptr || datapath_dir[0] == '\0');


remove extra space

Hardcode84 · 2019-07-31T13:03:37Z

modules/videoio/perf/perf_input.cpp

+PERF_TEST(, DISABLED_GetReadFrame)
+{
+    int ITERATION_COUNT = 50; //number of expected frames from all cameras
+#ifndef WINRT


wrap this block into function

Hardcode84 · 2019-08-01T10:30:11Z

modules/videoio/src/cap_interface.hpp

+   {
+       std::vector< CvCapture * > capPtrs;
+
+       for (size_t capnum = 0; capnum < deviceHandles.size(); ++capnum)


Use range for

Hardcode84 · 2019-08-01T10:31:05Z

modules/videoio/src/cap_interface.hpp

+
+       for (size_t capnum = 0; capnum < deviceHandles.size(); ++capnum)
+       {
+           CV_Assert(dynamic_cast< LegacyCapture * >(deviceHandles[capnum]) != nullptr);


replace with CV_DbgAssert

Hardcode84 · 2019-08-01T10:31:51Z

modules/videoio/src/cap_v4l.cpp

    virtual bool grabFrame() CV_OVERRIDE;
    virtual IplImage* retrieveFrame(int) CV_OVERRIDE;

+    virtual bool deviceHandlePoll(const std::vector<int>&, std::vector<int>&, const int64_t &) CV_OVERRIDE;


Pass integers by value

Hardcode84 · 2019-08-01T10:32:56Z

modules/videoio/src/cap_v4l.cpp

+
+        std::vector<pollfd> fds;
+
+        for (size_t dhand_num = 0; dhand_num < deviceHandles.size(); ++dhand_num)


Hardcode84 · 2019-08-01T10:35:00Z

modules/videoio/test/test_camera.cpp

+TEST_P(DISABLED_videoio_fps, v4l_poll_fps)
+{
+    //Two identical cameras
+    VideoCapture cap1(0, CAP_V4L);


Unhardcode camera indices

Hardcode84 · 2019-08-01T11:23:45Z

modules/videoio/src/cap_v4l.cpp

+        {
+            if(pointers[ptr_num])
+            {
+                CV_Assert(dynamic_cast< CvCaptureCAM_V4L * >(pointers[ptr_num]) != nullptr);


CV_DbgAssert

Hardcode84 · 2019-08-01T11:26:32Z

modules/videoio/src/cap_interface.hpp

    virtual bool setProperty(int, double) { return 0; }
    virtual bool grabFrame() { return true; }
+    virtual bool deviceHandlePoll(const std::vector<int>&, std::vector<int>&, const int64_t &){ return false; }
+    virtual bool setFirstCapture() {return false;}


remove from public interface

Hardcode84 · 2019-08-01T11:26:47Z

modules/videoio/src/cap_interface.hpp

    virtual double getProperty(int) const { return 0; }
    virtual bool setProperty(int, double) { return 0; }
    virtual bool grabFrame() { return true; }
+    virtual bool deviceHandlePoll(const std::vector<int>&, std::vector<int>&, const int64_t &){ return false; }


remove from public interface

…le in test_camera

Hardcode84 · 2019-08-02T08:14:21Z

modules/videoio/perf/perf_input.cpp

    "highgui/video/big_buck_bunny.wmv"
 };

+static char* set_environment_variable()


something like "get_cameras_list"

Hardcode84 · 2019-08-02T08:15:36Z

modules/videoio/src/cap.cpp

+            }
+            if(state[cupture_num] == CAP_CAM_ERROR)
+            {
+                //printf("error cam no. %d \n", cupture_num);


Remove debug code

Hardcode84 · 2019-08-02T08:17:34Z

modules/videoio/src/cap_v4l.cpp

+        int ret = poll(fds.data(), fds.size(), timeout);
+
+        if(ret == -1)
+            return false;


add perror call

Hardcode84 · 2019-08-02T10:18:01Z

modules/videoio/src/cap_v4l.cpp

+            }
+            else
+            {
+                perror("CvCapture pointer is null");


perror checks last errono. Is is not deeded there. Only after poll

Hardcode84 · 2019-08-02T10:19:24Z

modules/videoio/perf/perf_input.cpp

    "highgui/video/big_buck_bunny.wmv"
 };

+static char* set_environment_variable()


Hardcode84 · 2019-08-02T10:22:06Z

Add url to original issue into PR description

mpashchenkov · 2019-08-22T07:04:03Z

Tests passed.

mshabunin · 2019-08-30T10:14:48Z

modules/python/src2/cv2.cpp

 typedef std::vector<DMatch> vector_DMatch;
 typedef std::vector<String> vector_String;
 typedef std::vector<Scalar> vector_Scalar;
-


Please rollback unrelated changes

mshabunin · 2019-08-30T10:16:28Z

modules/videoio/include/opencv2/videoio.hpp

 //! @} videoio_flags_others


+enum statecam


Please use formatting consistent with other enums in this file

mshabunin · 2019-08-30T10:17:56Z

modules/videoio/include/opencv2/videoio/videoio_c.h

 */
 CVAPI(int) cvGrabFrame( CvCapture* capture );

+CVAPI(bool) cvCamerasPoll( const std::vector<CvCapture * >& pointers, std::vector<int>& state, const int64_t & timeout );


Please remove this function from public interface, C-API is deprecated and should not be extended. std::vector is part of C++ standard. If you need this function, declare it in private files (src/*.hpp, src/*.cpp).

mshabunin · 2019-08-30T10:19:27Z

modules/videoio/perf/perf_input.cpp

+#ifndef WINRT
+    return getenv("OPENCV_TEST_CAMERA_LIST");
+#else
+    return OPENCV_TEST_CAMERA_LIST;


There is no such variable, build for WinRT will be broken.

mshabunin · 2019-08-30T10:21:08Z

modules/videoio/perf/perf_input.cpp


+static char* get_cameras_list()
+{
+#ifndef WINRT


Use #ifndef NO_GETENV check here

mshabunin · 2019-08-30T15:01:28Z

modules/videoio/perf/perf_input.cpp

+        path += datapath_dir[step];
+        ++step;
+    }
+    Mat frame1, frame2,  processed1, processed2;


ASSERT_EQUAL(VCM.size(), 2u); Mat frame[2], processed[2];

mshabunin · 2019-08-30T15:06:05Z

modules/videoio/src/cap_v4l.cpp

+        std::vector<int> deviceHandles;
+        for(const auto& ptr_num : pointers)
+        {
+            if(ptr_num)


Why didn't not you use dynamic_cast?

CvCaptureCAM_V4L *ptr = dynamic_cast< CvCaptureCAM_V4L * >(ptr_num); if (!ptr) return false; deviceHandles.push_back(ptr->deviceHandle); ptr->setFirstCapture();

mshabunin · 2019-08-30T15:13:22Z

modules/videoio/src/cap_v4l.cpp

+                    state[struct_num] = CAP_CAM_READY;
+                }
+                else
+                    if((fds[struct_num].revents & POLLERR) != 0)


Please balance braces, its either

if () foo(); else bar();

either

if () { foo(); } else { bar(); }

mshabunin · 2019-08-30T15:15:08Z

modules/videoio/src/cap_v4l.cpp

+    return false;
+}
+
+bool CvCaptureCAM_V4L::setFirstCapture()


What does this method do? There is duplicated code in grabFrame. Duplicated code is bad practice.

mshabunin · 2019-08-30T15:16:48Z

modules/videoio/src/cap_v4l.cpp


+bool CvCaptureCAM_V4L::deviceHandlePoll(const std::vector<int>& deviceHandles, std::vector<int>& state, int64_t timeout)
+{
+     if(!deviceHandles.empty())


Please fix indents in this function.

@param

Implement Camera Multiplexing API * IdideoCapture + two wrong function function waitAny Add errors catcher Stub for Python added. Sifting warnings One test added Two tests for camera and Perf tests added * Perf sync and async tests for waitAny() added, waitAnyInterior() deleted, getDeviceHandle() deleted * Variable OPENCV_TEST_CAMERA_LIST added * Without fps set * ASSERT_FAILED for environment variable * Perf tests is DISABLED_ * --Trailing whitespace * Return false from cap.cpp deleted * Two functions deleted from interface, +range for, +environment variable in test_camera * Space deleted * printf deleted, perror added * CV_WRAP deleted, cv2 cleared from stubs * -- space * default timeout added * @param changed * place of waitAny changed * --whitespace * ++function description * function description changed * revert unused changes * videoio: rework API for VideoCapture::waitAny()

mshabunin reviewed Jul 22, 2019

View reviewed changes

modules/videoio/src/cap_interface.hpp Outdated Show resolved Hide resolved

mshabunin reviewed Jul 23, 2019

View reviewed changes

mshabunin reviewed Jul 25, 2019

View reviewed changes

IdideoCapture + two wrong function

0c11e06

function waitAny Add errors catcher Stub for Python added. Sifting warnings One test added Two tests for camera and Perf tests added

mpashchenkov force-pushed the cam_multiplexing_function_v branch from 1b098cc to 0c11e06 Compare July 29, 2019 07:45

Hardcode84 reviewed Jul 29, 2019

View reviewed changes

mpashchenkov added 7 commits July 30, 2019 12:07

Perf sync and async tests for waitAny() added, waitAnyInterior() dele…

602e984

…ted, getDeviceHandle() deleted

Variable OPENCV_TEST_CAMERA_LIST added

58a4708

Without fps set

a896db6

ASSERT_FAILED for environment variable

a648674

Perf tests is DISABLED_

90b8146

--Trailing whitespace

d84e5e8

Return false from cap.cpp deleted

29d8201

Hardcode84 reviewed Jul 31, 2019

View reviewed changes

Hardcode84 reviewed Aug 1, 2019

View reviewed changes

mpashchenkov added 2 commits August 2, 2019 09:38

Two functions deleted from interface, +range for, +environment variab…

fe4ca37

…le in test_camera

Space deleted

d3e2286

Hardcode84 reviewed Aug 2, 2019

View reviewed changes

printf deleted, perror added

c955d84

Hardcode84 reviewed Aug 2, 2019

View reviewed changes

mpashchenkov added 5 commits August 2, 2019 14:37

CV_WRAP deleted, cv2 cleared from stubs

2aba19e

-- space

604734d

default timeout added

59ada92

@param changed

9693939

place of waitAny changed

6a2e300

mpashchenkov added 3 commits August 15, 2019 16:40

--whitespace

4854687

++function description

c08c692

function description changed

5b51694

mshabunin requested changes Aug 30, 2019

View reviewed changes

alalek added 3 commits November 15, 2019 00:58

Merge remote-tracking branch 'upstream/master'

02b4975

revert unused changes

c70761c

videoio: rework API for VideoCapture::waitAny()

95d5639

mshabunin approved these changes Nov 15, 2019

View reviewed changes

alalek assigned mshabunin Nov 15, 2019

alalek merged commit 1acadd3 into opencv:master Nov 15, 2019

crackwitz mentioned this pull request Apr 5, 2022

Python bindings for VideoCapture::waitAny #21826

Closed

		CV_WRAP virtual bool grab();


		CV_WRAP static bool waitAny(std::vector<VideoCapture>&, std::vector<int>&, const int &);


		std::vector<pollfd> fds;

		for (size_t dhand_num = 0; dhand_num < deviceHandles.size(); ++dhand_num)

Uh oh!

Conversation

mpashchenkov commented Jul 19, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mshabunin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mshabunin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Hardcode84 commented Jul 29, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mpashchenkov commented Jul 19, 2019 •

edited

Loading

Hardcode84 commented Jul 29, 2019 •

edited

Loading