vittrack(gsoc realtime object tracking model) by lpylpy0514 · Pull Request #24201 · opencv/opencv

lpylpy0514 · 2023-08-27T12:02:50Z

Vit tracker(vision transformer tracker) is a much better model for real-time object tracking. Vit tracker can achieve speeds exceeding nanotrack by 20% in single-threaded mode with ARM chip, and the advantage becomes even more pronounced in multi-threaded mode. In addition, on the dataset, vit tracker demonstrates better performance compared to nanotrack. Moreover, vit trackerprovides confidence values during the tracking process, which can be used to determine if the tracking is currently lost.
opencv_zoo: opencv/opencv_zoo#194
opencv_extra: opencv/opencv_extra#1088

Performance comparison is as follows:

NOTE: The speed below is tested by onnxruntime because opencv has poor support for the transformer architecture for now.

ONNX speed test on ARM platform(apple M2)(ms):

thread nums	1	2	3	4
nanotrack	5.25	4.86	4.72	4.49
vit tracker	4.18	2.41	1.97	1.46 (3X)

ONNX speed test on x86 platform(intel i3 10105)(ms):

thread nums	1	2	3	4
nanotrack	3.20	2.75	2.46	2.55
vit tracker	3.84	2.37	2.10	2.01

opencv speed test on x86 platform(intel i3 10105)(ms):

thread nums	1	2	3	4
vit tracker	31.3	31.4	31.4	31.4

preformance test on lasot dataset(AUC is the most important data. Higher AUC means better tracker):

LASOT	AUC	P	Pnorm
nanotrack	46.8	45.0	43.3
vit tracker	48.6	44.8	54.7

https://youtu.be/MJiPnu1ZQRI
In target tracking tasks, the score is an important indicator that can indicate whether the current target is lost. In the video, vit tracker can track the target and display the current score in the upper left corner of the video. When the target is lost, the score drops significantly. While nanotrack will only return 0.9 score in any situation, so that we cannot determine whether the target is lost.

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
The PR is proposed to the proper branch
There is a reference to the original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

zihaomu

Please take a look.

modules/video/include/opencv2/video/tracking.hpp

modules/video/src/tracking/tracker_vit.cpp

modules/video/test/test_trackers.cpp

modules/video/include/opencv2/video/tracking.hpp

zihaomu

LGTM! 👍

modules/video/include/opencv2/video/tracking.hpp

modules/video/src/tracking/tracker_vit.cpp

asmorkalov · 2023-09-08T11:31:36Z

modules/video/src/tracking/tracker_vit.cpp

+    cv::Scalar meanvalue(0.485, 0.456, 0.406);
+    cv::Scalar stdvalue(0.229, 0.224, 0.225);


I propose to move it to parameters. In case if users re-trained the model they can use custom pre-proc params.

Is there any sample code available for reference?

Actually these parameters are from the classic ImageNet image classification. Basically everybody uses these params by default. I dont see anyone changing them in any practical sense.

The model may be re-trained for YUV or use-cases that are not classical RGB imaging in science, medicine, etc.

opencv-alalek · 2023-09-09T03:46:22Z

samples/dnn/vit_tracker.cpp

@@ -0,0 +1,176 @@
+// VitTracker
+// model: https://github.com/opencv/opencv_extra/blob/4.x/testdata/dnn/onnx/models/vitTracker.onnx


Use opencv_zoo too to avoid mess of different sources

Also it makes sense to dump suggested model path in --help mode.

modules/video/src/tracking/tracker_vit.cpp

samples/python/tracker.py

modules/video/include/opencv2/video/tracking.hpp

asmorkalov

👍 Looks good to me in general. Tested manually with Ubuntu and Camera. I propose to extract mean and std to the class parameters with existing default values and then merge.

modules/video/include/opencv2/video/tracking.hpp

VIT track(gsoc realtime object tracking model) #1088 add an onnx model for opencv vttrack PR opencv/opencv#24201

Add VIT track model and demo #194 GSOC Realtime tracking model opencv repo PR link is [here](opencv/opencv#24201)

opencv-alalek · 2023-09-20T06:40:39Z

@asmorkalov Why do you merge large PR without CI checks? We should run extended scope of tests in that case instead of ignoring them.

Debug builds are broken.

VIT track(gsoc realtime object tracking model) opencv#24201 Vit tracker(vision transformer tracker) is a much better model for real-time object tracking. Vit tracker can achieve speeds exceeding nanotrack by 20% in single-threaded mode with ARM chip, and the advantage becomes even more pronounced in multi-threaded mode. In addition, on the dataset, vit tracker demonstrates better performance compared to nanotrack. Moreover, vit trackerprovides confidence values during the tracking process, which can be used to determine if the tracking is currently lost. opencv_zoo: opencv/opencv_zoo#194 opencv_extra: [https://github.com/opencv/opencv_extra/pull/1088](https://github.com/opencv/opencv_extra/pull/1088) # Performance comparison is as follows: NOTE: The speed below is tested by **onnxruntime** because opencv has poor support for the transformer architecture for now. ONNX speed test on ARM platform(apple M2)(ms): | thread nums | 1| 2| 3| 4| |--------|--------|--------|--------|--------| | nanotrack| 5.25| 4.86| 4.72| 4.49| | vit tracker| 4.18| 2.41| 1.97| **1.46 (3X)**| ONNX speed test on x86 platform(intel i3 10105)(ms): | thread nums | 1| 2| 3| 4| |--------|--------|--------|--------|--------| | nanotrack|3.20|2.75|2.46|2.55| | vit tracker|3.84|2.37|2.10|2.01| opencv speed test on x86 platform(intel i3 10105)(ms): | thread nums | 1| 2| 3| 4| |--------|--------|--------|--------|--------| | vit tracker|31.3|31.4|31.4|31.4| preformance test on lasot dataset(AUC is the most important data. Higher AUC means better tracker): |LASOT | AUC| P| Pnorm| |--------|--------|--------|--------| | nanotrack| 46.8| 45.0| 43.3| | vit tracker| 48.6| 44.8| 54.7| [https://youtu.be/MJiPnu1ZQRI](https://youtu.be/MJiPnu1ZQRI) In target tracking tasks, the score is an important indicator that can indicate whether the current target is lost. In the video, vit tracker can track the target and display the current score in the upper left corner of the video. When the target is lost, the score drops significantly. While nanotrack will only return 0.9 score in any situation, so that we cannot determine whether the target is lost. ### Pull Request Readiness Checklist See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request - [x] I agree to contribute to the project under Apache 2 License. - [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [x] The PR is proposed to the proper branch - [ ] There is a reference to the original bug report and related work - [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [ ] The feature is well documented and sample code can be built with the project CMake

Onlybyuse added 3 commits August 27, 2023 19:46

add vttrack model and test code

9ff5ecb

delete some unused code

abaff2f

fix a bug

959d003

zihaomu self-assigned this Aug 28, 2023

zihaomu added the GSoC label Aug 28, 2023

name update to vitTracker

81d0e08

zihaomu changed the title ~~vttrack(gsoc realtime object tracking model)~~ vittrack(gsoc realtime object tracking model) Aug 28, 2023

zihaomu added the category: video label Aug 28, 2023

zihaomu reviewed Aug 28, 2023

View reviewed changes

modules/video/include/opencv2/video/tracking.hpp Outdated Show resolved Hide resolved

modules/video/src/tracking/tracker_vit.cpp Outdated Show resolved Hide resolved

modules/video/test/test_trackers.cpp Outdated Show resolved Hide resolved

zihaomu reviewed Aug 28, 2023

View reviewed changes

modules/video/include/opencv2/video/tracking.hpp Outdated Show resolved Hide resolved

Onlybyuse added 4 commits August 28, 2023 16:09

fix some notes

ffaa2c5

fix some notes

e19794b

add c++ sample code

51b580c

add example code for vittrack

cafad0d

zihaomu approved these changes Sep 8, 2023

View reviewed changes

fengyuentau reviewed Sep 8, 2023

View reviewed changes

modules/video/include/opencv2/video/tracking.hpp Outdated Show resolved Hide resolved

update link for model vittrack

d02187d

lpylpy0514 mentioned this pull request Sep 8, 2023

add vittrack and result opencv/opencv_zoo#194

Merged

fix a bug

37c0cc8

asmorkalov added this to the 4.9.0 milestone Sep 8, 2023

asmorkalov reviewed Sep 8, 2023

View reviewed changes

opencv-alalek reviewed Sep 9, 2023

View reviewed changes

update comments and namespace

c066509

fengyuentau reviewed Sep 12, 2023

View reviewed changes

modules/video/include/opencv2/video/tracking.hpp Outdated Show resolved Hide resolved

update link for model

66d03b1

fengyuentau approved these changes Sep 13, 2023

View reviewed changes

asmorkalov approved these changes Sep 14, 2023

View reviewed changes

move mean and std value to params

5e09b6f

asmorkalov reviewed Sep 14, 2023

View reviewed changes

modules/video/include/opencv2/video/tracking.hpp Outdated Show resolved Hide resolved

move init to constructor

0440318

asmorkalov merged commit 70d7e83 into opencv:4.x Sep 19, 2023

asmorkalov mentioned this pull request Sep 19, 2023

vttrack(gsoc realtime object tracking model) opencv/opencv_extra#1088

Merged

asmorkalov pushed a commit to opencv/opencv_extra that referenced this pull request Sep 19, 2023

Merge pull request #1088 from lpylpy0514:4.x

93cdc72

VIT track(gsoc realtime object tracking model) #1088 add an onnx model for opencv vttrack PR opencv/opencv#24201

asmorkalov pushed a commit to opencv/opencv_zoo that referenced this pull request Sep 19, 2023

Merge pull request #194 from lpylpy0514:main

4347f6a

Add VIT track model and demo #194 GSOC Realtime tracking model opencv repo PR link is [here](opencv/opencv#24201)

asmorkalov mentioned this pull request Sep 20, 2023

Warnings fix on Windows. #24303

Merged

6 tasks

asmorkalov mentioned this pull request Sep 28, 2023

(5.x) Merge 4.x #24338

Merged

		cv::Scalar meanvalue(0.485, 0.456, 0.406);
		cv::Scalar stdvalue(0.229, 0.224, 0.225);

		@@ -0,0 +1,176 @@
		// VitTracker
		// model: https://github.com/opencv/opencv_extra/blob/4.x/testdata/dnn/onnx/models/vitTracker.onnx

Uh oh!

Conversation

lpylpy0514 commented Aug 27, 2023 • edited by asmorkalov Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Performance comparison is as follows:

Pull Request Readiness Checklist

Uh oh!

zihaomu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zihaomu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

asmorkalov Sep 8, 2023

Choose a reason for hiding this comment

Uh oh!

lpylpy0514 Sep 9, 2023

Choose a reason for hiding this comment

Uh oh!

fengyuentau Sep 12, 2023

Choose a reason for hiding this comment

Uh oh!

asmorkalov Sep 14, 2023

Choose a reason for hiding this comment

Uh oh!

opencv-alalek Sep 9, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

asmorkalov left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

opencv-alalek commented Sep 20, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

lpylpy0514 commented Aug 27, 2023 •

edited by asmorkalov

Loading