G-API: Adding a Python script to download onnx models by mpashchenkov · Pull Request #831 · opencv/opencv_extra

mpashchenkov · 2020-12-10T11:26:31Z

Added python script which downloads models from https://github.com/onnx/models
Behavior (UPDATED):

Script creates two directories onnx_models_cache and onnx_models;
onnx_models_cache contains result of git clone (small files);
onnx_models contains large ONNX models which will be used for something;
Checks sha for models from onnx_models folder and skip download if model already exists and pass sha check;
You can remove onnx_models and call script again, then models will be downloaded again;
Keeps constant size of onnx_models_cache and removes lfs folder from onnx_models_cache for that;
Script is tested on Linux and Windows;
Names of models which wasn't downloaded are placed to failedModels.

Also you can call download_onnx_models.py with model_name/print,

model with model_name will be downloaded in this case;
print - prints all models which script can download.

Added new directories:

testdata/gapi;
testdata/gapi/onnx;

alalek · 2020-12-10T15:30:33Z

testdata/gapi/onnx/images/laptop.jpg

Legal/copyright information is required about the used image.

alalek · 2020-12-11T05:33:04Z

git clone
git lfs pull

Until downloaded these LFS files EXIST and store lfs metadata.
It is really hard to debug problem of "missing test file" from tests logs.

Git repository itself should go into dedicated "download_cache" directory.

What is about repository commit revision?
File hashsum (MD5/SHA256)? (to ensure that it is valid/not truncated/etc)

mpashchenkov · 2021-01-15T23:04:04Z

testdata/gapi/onnx/download_onnx_models.py

+    # and "lfs" folder will exist during download
+    lfs_cache_path = CACHE_DIR + '.git/lfs'
+    if os.path.exists(lfs_cache_path):
+        rmdir_with_data(lfs_cache_path)


@alalek, it is questionable step. Cache directory gets some data after calling git lfs pull. I remove this folder and don't know. Should cache directory have constant size (180 MB) or should contain received data (500 mb)? This (without deleting lfs folder) avoids downloading models after deleting onnx_models folder.

mpashchenkov · 2021-02-15T09:51:24Z

@smirnov-alexey, can you review this?

testdata/gapi/onnx/download_onnx_models.py

smirnov-alexey

Looks good to me

OrestChura

Some comments, most are just questions to double-check

testdata/gapi/onnx/download_onnx_models.py

OrestChura

Some comment cavils last, but I'm all good with the script. Thanks!

mpashchenkov · 2021-03-02T10:13:59Z

@alalek, does script have correct behavior (check description)? If yes, can we merge it?

dmatveev · 2021-03-29T13:32:51Z

@alalek can we merge this thing? I believe it is required for a more correct ONNX models coverage we do now.

alalek

Overall looks good!

alalek · 2021-03-29T14:03:43Z

testdata/gapi/onnx/download_onnx_models.py

+def clear_pulled_files():
+    # "lfs" directory contains some data
+    # Remove this folder then "onnx_models_cache" will have standard size - 180 MB
+    # If we remove "onnx_models" folder then models will be downloaded again
+    # and "lfs" folder will exist during download
+    lfs_cache_path = CACHE_DIR + '.git/lfs'
+    if os.path.exists(lfs_cache_path):
+        rmdir_with_data(lfs_cache_path)


Do we really need to clean '.git' internals? Does it break anything?

No, but without cleaning models won't be downloaded again and will be moved from cache. And for models update you should remove cache directory.
It is main question about behavior. Do we need update models from GH when we remove models directory (NOT cache)?

Don't clear the cache now.

alalek · 2021-03-29T14:43:12Z

testdata/gapi/onnx/download_onnx_models.py

+        os.system('git clone --recursive https://github.com/onnx/models.git onnx_models_cache')
+    else:


git clone

Need to fetch+checkout the latest commits from the upstream repository if folder already exists.

Added git pull of master branch.

alalek · 2021-03-29T14:46:10Z

testdata/gapi/onnx/download_onnx_models.py

+
+CUR_DIR = os.getcwd()
+# This directory contains result of "git clone"
+CACHE_DIR = CUR_DIR + '/onnx_models_cache/'


'/onnx_models_cache/'

Please use '/.cache/onnx_models/'

mpashchenkov · 2021-12-23T11:11:10Z

@alalek, Could this be merged if all comments are applied correctly?

mpashchenkov commented Jan 15, 2021

View reviewed changes

smirnov-alexey reviewed Feb 15, 2021

View reviewed changes

testdata/gapi/onnx/download_onnx_models.py Outdated Show resolved Hide resolved

smirnov-alexey approved these changes Feb 15, 2021

View reviewed changes

OrestChura reviewed Feb 18, 2021

View reviewed changes

testdata/gapi/onnx/download_onnx_models.py Show resolved Hide resolved

testdata/gapi/onnx/download_onnx_models.py Outdated Show resolved Hide resolved

testdata/gapi/onnx/download_onnx_models.py Outdated Show resolved Hide resolved

OrestChura approved these changes Feb 18, 2021

View reviewed changes

mpashchenkov assigned OrestChura Mar 2, 2021

alalek reviewed Mar 29, 2021

View reviewed changes

mpashchenkov force-pushed the mp/oe-add-gapi branch from 8287c09 to cf77691 Compare December 13, 2021 08:14

Added parser, removed deleting cache, added pulling

4e325ce

mpashchenkov force-pushed the mp/oe-add-gapi branch from cf77691 to 4e325ce Compare December 13, 2021 09:02

Removed cloning step for simple print of list of models

e9ca64e

mpashchenkov requested a review from alalek December 13, 2021 09:07

alalek approved these changes Dec 23, 2021

View reviewed changes

alalek merged commit 37294e3 into opencv:4.x Dec 23, 2021

alalek mentioned this pull request Dec 30, 2021

(5.x) Merge 4.x #949

Merged

alalek mentioned this pull request Feb 22, 2022

(5.x) Merge 4.x #960

Merged

		os.system('git clone --recursive https://github.com/onnx/models.git onnx_models_cache')
		else:

Conversation

mpashchenkov commented Dec 10, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alalek commented Dec 10, 2020

Uh oh!

alalek commented Dec 11, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mpashchenkov commented Feb 15, 2021

Uh oh!

Uh oh!

smirnov-alexey left a comment

Choose a reason for hiding this comment

Uh oh!

OrestChura left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

OrestChura left a comment

Choose a reason for hiding this comment

Uh oh!

mpashchenkov commented Mar 2, 2021

Uh oh!

dmatveev commented Mar 29, 2021

Uh oh!

alalek left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mpashchenkov commented Dec 23, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

mpashchenkov commented Dec 10, 2020 •

edited

Loading