G-API: Python. Gaze Estimation sample. by mpashchenkov · Pull Request #20144 · opencv/opencv

mpashchenkov · 2021-05-24T09:27:18Z

Gaze Estimation sample based on GAPI Python api.

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or other license that is incompatible with OpenCV
The PR is proposed to proper branch
There is reference to original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

Magic commands:

force_builders=Custom,Custom Win,Custom Mac
build_gapi_standalone:Linux x64=ade-0.1.1f
build_gapi_standalone:Win64=ade-0.1.1f
build_gapi_standalone:Mac=ade-0.1.1f
build_gapi_standalone:Linux x64 Debug=ade-0.1.1f

Xbuild_image:Custom=centos:7
Xbuildworker:Custom=linux-1
build_gapi_standalone:Custom=ade-0.1.1f

build_image:Custom=ubuntu-openvino-2021.3.0:20.04
build_image:Custom Win=openvino-2021.2.0
build_image:Custom Mac=openvino-2021.2.0

test_modules:Custom=gapi,python2,python3,java
test_modules:Custom Win=gapi,python2,python3,java
test_modules:Custom Mac=gapi,python2,python3,java

buildworker:Custom=linux-1
# disabled due high memory usage: test_opencl:Custom=ON
test_opencl:Custom=OFF
test_bigdata:Custom=1
test_filter:Custom=*

alalek · 2021-05-24T14:54:42Z

modules/gapi/misc/python/samples/gaze_estimation.py

+import time
+
+def weight_path(model_path):
+    return model_path.split('.')[0] + '.bin'


rfind() should perform better. Or even .endswith('.xml') + model_path[:-3]

(e.g, openvino paths may have dots in near version numbers).

alalek · 2021-05-24T14:55:30Z

modules/gapi/misc/python/samples/gaze_estimation.py

@@ -0,0 +1,373 @@
+import cv2 as cv
+import argparse
+import numpy as np


Please numpy import before cv2 (some issues with some python configurations)

alalek · 2021-05-24T14:58:59Z

modules/gapi/misc/python/samples/gaze_estimation.py

+    cap = cv.VideoCapture(arguments.input)
+    out = cv.VideoWriter(arguments.out,\
+                         cv.VideoWriter_fourcc('M','J','P','G'), 25,\
+                         (int(cap.get(cv.CAP_PROP_FRAME_WIDTH)),\


\

the same

Removed all \

alalek · 2021-05-24T14:59:44Z

modules/gapi/misc/python/samples/gaze_estimation.py

+                         cv.VideoWriter_fourcc('M','J','P','G'), 25,\
+                         (int(cap.get(cv.CAP_PROP_FRAME_WIDTH)),\
+                          int(cap.get(cv.CAP_PROP_FRAME_HEIGHT))))
+    num_frames = cap.get(cv.CAP_PROP_FRAME_COUNT)


not available for live streams / demos (e.g, from cameras)

alalek · 2021-05-24T15:01:06Z

modules/gapi/misc/python/samples/gaze_estimation.py

+    ccomp.setSource(source)
+    ccomp.start()
+
+    cap = cv.VideoCapture(arguments.input)


missing .isOpened() check with descriptive error message

VideoCapture is removed.

alalek · 2021-05-24T15:02:23Z

modules/gapi/misc/python/samples/gaze_estimation.py

+    source = cv.gapi.wip.make_capture_src(arguments.input)
+    ccomp.setSource(source)
+    ccomp.start()
+
+    cap = cv.VideoCapture(arguments.input)


make_capture_src
VideoCapture

conflicting calls:

should be reordered to probe properties before streaming

VideoCapture should be closed first.

VideoCapture is removed.

TolyaTalamanov

Let's start with pylint

TolyaTalamanov · 2021-05-26T09:59:34Z

modules/gapi/misc/python/samples/gaze_estimation.py

+if __name__ == '__main__':
+    parser = argparse.ArgumentParser(description='This is an OpenCV-based version of Gaze Estimation example')
+
+    parser.add_argument('--input',


I would hide this:

def build_argparser(): ... args = build_argparser().parse_args()

TolyaTalamanov · 2021-05-26T10:12:46Z

modules/gapi/misc/python/samples/gaze_estimation.py

+                out_r_st += [1 if st[0] < st[1] else 0]
+        return out_l_st, out_r_st
+
+@cv.gapi.op('custom.copy', in_types=[cv.GMat,], out_types=[cv.GMat])


unnecessary comma

TolyaTalamanov · 2021-06-23T14:36:58Z

modules/gapi/misc/python/samples/gaze_estimation.py

+        return out_poses
+
+
+@cv.gapi.op('custom.ParseEyes', in_types=[cv.GArray.GMat,


Why don't you do like this ?

@cv.gapi.op('custom.ParseEyes', in_types=[cv.GArray.GMat, cv.GArray.Rect, cv.GOpaque.Size], out_types=[cv.GArray.Rect, cv.GArray.Rect, cv.GArray.Point, cv.GArray.Point])

TolyaTalamanov

Everything looks good 👍

Cosmetic fixes are required

TolyaTalamanov · 2021-06-24T07:18:56Z

modules/gapi/misc/python/samples/gaze_estimation.py

+
+@cv.gapi.op('custom.GetStates', in_types=[cv.GArray.GMat, cv.GArray.GMat],
+            out_types=[cv.GArray.Int, cv.GArray.Int])
+class GetStates:


Since GGetStatesImpl, it should be GGetStates.

BTW, how about putting doc string for kernel & operations ?

TolyaTalamanov · 2021-06-24T07:19:08Z

modules/gapi/misc/python/samples/gaze_estimation.py

+
+# FIXME: the operation should be wrapped soon
+@cv.gapi.op('custom.Copy', in_types=[cv.GMat], out_types=[cv.GMat])
+class Copy:


The same GCopy then

TolyaTalamanov · 2021-06-24T07:21:49Z

modules/gapi/misc/python/samples/gaze_estimation.py

+    print('Processing')
+    START_TIME = time.time()
+
+    while state:


Can it be clearer ?

while True: has_frame, ... = ccomp.pull() if not has_frame: break ...

TolyaTalamanov · 2021-06-24T07:22:40Z

modules/gapi/misc/python/samples/gaze_estimation.py

+
+        fps = int(1. / (time.time() - start_time_cycle))
+        frames += 1
+    ALL_TIME = time.time() - START_TIME


total time or execution time

execution time

TolyaTalamanov · 2021-06-24T07:24:28Z

modules/gapi/misc/python/samples/gaze_estimation.py

+
+
+@cv.gapi.kernel(Copy)
+class GCopyImpl:


Why don't you just wrap this one on your own ?
It should be pretty straightforward

Copy is wrapped

TolyaTalamanov · 2021-06-24T07:31:49Z

modules/gapi/misc/python/samples/gaze_estimation.py

+        return left_eyes, right_eyes, midpoints, lmarks
+
+
+@cv.gapi.op('custom.GetStates', in_types=[cv.GArray.GMat, cv.GArray.GMat],


@cv.gapi.op('custom.GetStates', in_types=[cv.GArray.GMat, cv.GArray.GMat], out_types=[cv.GArray.Int, cv.GArray.Int])

TolyaTalamanov · 2021-06-24T07:34:42Z

modules/gapi/misc/python/samples/gaze_estimation.py

+    eye_net = cv.gapi.ie.params('open-closed-eye', ARGUMENTS.eyem,
+                                weight_path(ARGUMENTS.eyem), ARGUMENTS.eyed)
+
+    nets = cv.gapi.networks(face_net, head_pose_net, landmarks_net, gaze_net,


nets = cv.gapi.networks(face_net, head_pose_net, landmarks_net, gaze_net, eye_net)

Looks like you don't exceed the line limitation in python, do you ?

I used 80 symbols limit. Removed line break.

TolyaTalamanov · 2021-06-24T07:34:53Z

modules/gapi/misc/python/samples/gaze_estimation.py

+                            eye_net)
+
+    # Kernels pack
+    kernels = cv.gapi.kernels(GParseEyesImpl, GProcessPosesImpl, GGetStatesImpl,


TolyaTalamanov · 2021-06-24T07:36:42Z

modules/gapi/misc/python/samples/gaze_estimation.py

+
+        FACES_SIZE = len(outr)
+
+        for i in range(FACES_SIZE):


You can just:

for i, out_rect in enumerate(outr): ...

TolyaTalamanov · 2021-06-24T07:38:04Z

modules/gapi/misc/python/samples/gaze_estimation.py

+            cv.rectangle(oimg, r_eyes[i], color_r, 1)
+
+            # Gaze vectors
+            normGazes = np.linalg.norm(outg[i][0])


Every variable in snake_case, right ?

TolyaTalamanov · 2021-06-28T07:09:40Z

modules/gapi/misc/python/samples/gaze_estimation.py

+    """ Parse arguments from comand line
+
+    Return:
+    Pack of arguments from comand line


TolyaTalamanov · 2021-06-28T07:09:59Z

modules/gapi/misc/python/samples/gaze_estimation.py

+
+
+def build_argparser():
+    """ Parse arguments from comand line


TolyaTalamanov · 2021-06-28T07:12:07Z

modules/gapi/misc/python/samples/gaze_estimation.py

+@cv.gapi.op('custom.ParseEyes',
+            in_types=[cv.GArray.GMat, cv.GArray.Rect, cv.GOpaque.Size],
+            out_types=[cv.GArray.Rect, cv.GArray.Rect, cv.GArray.Point, cv.GArray.Point])
+class ParseEyes:


I thought we decided GParseEyes and GParseEyesImpl

TolyaTalamanov · 2021-06-28T14:16:10Z

modules/gapi/misc/python/samples/gaze_estimation.py

+    # ------------------------Execution part------------------------
+    ccomp = comp.compileStreaming(args=cv.gapi.compile_args(kernels, nets))
+    source = cv.gapi.wip.make_capture_src(ARGUMENTS.input)
+    ccomp.setSource(source)


please use:

ccomp.setSource(cv.gin(source))

mpashchenkov · 2021-06-30T08:37:57Z

@alalek, can it be merged?

mpashchenkov · 2021-07-01T10:12:00Z

@alalek, #20144 (comment), build is green. Can it be merged? All comments are fixed.

AlexanderKalistratov · 2021-07-18T14:43:58Z

modules/gapi/misc/python/samples/gaze_estimation.py

+        Return:
+        Arrays with heads poses
+        """
+        out_poses = []


Consider this option:

out_poses = [np.array([ys[0], ps[0], rs[0]]).T for ys, ps, rs in zip(in_ys, in_ps, in_rs)]

Fixed in #20510

AlexanderKalistratov · 2021-07-18T14:46:10Z

modules/gapi/misc/python/samples/gaze_estimation.py

+        lmarks = []
+        num_faces = len(in_landm_per_face)
+        surface = (0, 0, *frame_size)
+        for i in range(num_faces):


consider this option:

for landm_face, rect in zip(in_landm_per_face, in_face_rcs): points = process_landmarks(*rect, landm_face) lmarks.extend(points) rect, midpoint_l = eye_box(points[0], points[1]) left_eyes.append(intersection(surface, rect)) rect, midpoint_r = eye_box(points[2], points[3]) right_eyes.append(intersection(surface, rect)) midpoints.append(midpoint_l) midpoints.append(midpoint_r)

Fixed in #20510

AlexanderKalistratov · 2021-07-18T14:48:06Z

modules/gapi/misc/python/samples/gaze_estimation.py

+        size = len(eyesl)
+        out_l_st = []
+        out_r_st = []
+        for i in range(size):


Consider this option:

out_l_st = [int(st) for eye_l in eyesl for st in (eye_l[:, 0] < eye_l[:, 1]).ravel()] out_r_st = [int(st) for eye_r in eyesr for st in (eye_r[:, 0] < eye_r[:, 1]).ravel()]

Fixed in #20510

G-API: Python. Gaze Estimation sample. * GE pep8 * Added function description, wrapped copy * Applying review comments * One more change * Added gin * Rstrt bb

mpashchenkov added category: python bindings category: samples category: g-api / gapi labels May 24, 2021

alalek reviewed May 24, 2021

View reviewed changes

mpashchenkov assigned TolyaTalamanov May 26, 2021

TolyaTalamanov reviewed May 26, 2021

View reviewed changes

mpashchenkov force-pushed the mp/python-ge branch from 3a4e9b8 to 8b6998b Compare June 2, 2021 09:41

GE pep8

6118363

mpashchenkov force-pushed the mp/python-ge branch from 8b6998b to 6118363 Compare June 23, 2021 08:59

TolyaTalamanov reviewed Jun 24, 2021

View reviewed changes

TolyaTalamanov approved these changes Jun 24, 2021

View reviewed changes

Added function description, wrapped copy

281114a

TolyaTalamanov reviewed Jun 28, 2021

View reviewed changes

Maxim Pashchenkov added 2 commits June 28, 2021 11:29

Applying review comments

1b7d4ce

One more change

2d4220a

TolyaTalamanov reviewed Jun 28, 2021

View reviewed changes

Maxim Pashchenkov added 2 commits June 28, 2021 17:27

Added gin

68719bb

Rstrt bb

c2a2618

alalek merged commit d70053a into opencv:master Jul 1, 2021

AlexanderKalistratov reviewed Jul 18, 2021

View reviewed changes

mpashchenkov mentioned this pull request Aug 6, 2021

G-API: Python. Gaze Estimation sample. Update. #20510

Merged

6 tasks

alalek mentioned this pull request Oct 15, 2021

(5.x) Merge 4.x #20886

Merged

		return out_poses


		@cv.gapi.op('custom.ParseEyes', in_types=[cv.GArray.GMat,

		return left_eyes, right_eyes, midpoints, lmarks


		@cv.gapi.op('custom.GetStates', in_types=[cv.GArray.GMat, cv.GArray.GMat],



		@cv.gapi.kernel(Copy)
		class GCopyImpl:

Uh oh!

Conversation

mpashchenkov commented May 24, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Readiness Checklist

Magic commands:

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TolyaTalamanov left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TolyaTalamanov Jun 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TolyaTalamanov left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

mpashchenkov commented May 24, 2021 •

edited

Loading

TolyaTalamanov Jun 23, 2021 •

edited

Loading