Scripts for downloading models in DNN samples by sl-sergei · Pull Request #18591 · opencv/opencv

sl-sergei · 2020-10-15T02:19:53Z

Relates: #12186

force_builders=linux,docs,Custom,Custom Mac
build_image:Custom=ubuntu-openvino-2020.4.0:16.04
build_image:Custom Win=openvino-2020.4.0
build_image:Custom Mac=openvino-2020.4.0

test_modules:Custom=dnn,python2,python3,java,samples
test_modules:Custom Win=dnn,python2,python3,java,samples
test_modules:Custom Mac=dnn,python2,python3,java,samples

buildworker:Custom=linux-1
# disabled due high memory usage: test_opencl:Custom=ON
test_opencl:Custom=OFF
test_bigdata:Custom=1
test_filter:Custom=*

sl-sergei · 2020-10-15T10:41:39Z

@alalek Can you please review these changes? Please, let me know, if there is something else needed for this task.

asmorkalov · 2020-10-16T09:34:07Z

@vpisarev please take a look.

vpisarev · 2020-10-20T09:12:56Z

@sl-sergei, thank you!

can you please put force_builders=... comment into the pull request to run it one some build slaves?
I would like to have some decentralized solution (i.e. the info about all downloadable items does not have to be put into one big file), and not necessarily limited to deep learning models. It can also be images, video clips and generally arbitrary data that is needed for a sample or an app. There should be documented program API as well, not just command-line API.

Approximate usage:

import cv2 as cv, sys

try:
   imgpath=cv.download(
       url="https://github.com/opencv/opencv/raw/master/samples/data/lena.jpg",
       sha="12345"
       )
except IOError:
   sys.exit("could not load lena image")

img = cv.imread(imgpath, -1)
...

url and sha are required parameters
dir/save_dir is optional parameter. If it's not set, some environment variable should be used (like OPENCV_SAVE_DIR), and if it's empty, it could be opencv_data subdirectory inside $TEMP or something like that. Saving to current directory may not be very good idea, because sample can be executed from different directories. Saving to the same directory as sample is not a good idea either, because it can be in read-only directory (like /usr/share) or on a slow flash drive etc.
The file should be saved under <save_dir>/<sha_key>/<base_part_of_url> or maybe as <save_dir>/<sha_key>_<base_part_of_url>, like $TEMP/opencv_data/12345/lena.jpg or $TEMP/opencv_data/12345_lena.jpg in this case. Putting file into a dedicated subdirectory or adding sha as a prefix will help to avoid possible name conflicts
timeout could be another optional parameter

sl-sergei · 2020-10-30T08:44:50Z

@vpisarev @alalek Can you please review the changes?

alalek · 2020-10-30T08:58:35Z

cv.download

This means that this functionality should be contributed into Python package with OpenCV itself (somewhere here)

subdirectory inside $TEMP

There are several variants which are not secure (due to be globally writable locations, like /tmp).
Consider checking (reusing / adopting) code for OpenCL binaries cache (review this code).

sl-sergei · 2020-10-30T09:00:28Z

cv.download

This means that this functionality should be contributed into Python package with OpenCV itself (somewhere here)

subdirectory inside $TEMP

There are several variants which are not secure (due to be globally writable locations, like /tmp).
Consider checking (reusing / adopting) code for OpenCL binaries cache (review this code).

Okay, I will do it

vpisarev · 2020-11-09T23:45:46Z

@sl-sergei, thank you! I still do not see how this functionality can be used from another Python script, as in the sample that I put above. Can you, please, modify at least one Python sample to download some data automatically?

sl-sergei · 2020-11-12T17:47:35Z

@alalek I updated script logic to make cache folder selection more secure. @vpisarev I added an example to README

vpisarev · 2020-11-18T17:37:59Z

@sl-sergei, looks good to me! 👍

asmorkalov · 2020-11-20T08:00:21Z

@alalek Could we merge it?

alalek

I don't see changes in current samples.
Is there some "single-click" sample? Which can download all necessary data without passing of extra parameters?

alalek · 2020-11-12T17:53:30Z

samples/dnn/models.yml

 opencv_fd:
+  load_info:
+    url: "https://github.com/opencv/opencv_3rdparty/raw/dnn_samples_face_detector_20170830/res10_300x300_ssd_iter_140000.caffemodel"
+    sha: "15aa726b4d46d9f023526d85537db81cbc8dd566"


Let use sha1 entry name (because there are several SHA implementations or user may want to pass md5 instead).

alalek · 2020-11-13T22:57:39Z

samples/dnn/download_models.py

+            temp_env = os.environ.get("TMPDIR", None)
+            if temp_env is None or not os.path.isdir(temp_env):
+                temp_dir = Path("/tmp")
+                print("Using world accessible cache directory. This may be not secure: ", temp_dir)


We should wrap and reuse C++ implementation instead of adding duplicated logic.

Lets rollback changes and add "TODO" here until we have resolved issues in #18931

sl-sergei · 2020-11-20T08:30:52Z

I don't see changes in current samples.
Is there some "single-click" sample? Which can download all necessary data without passing of extra parameters?

python download_models.py command will parse models.yml and face_detector/weights.meta4 and download all models mentioned there

alalek · 2020-11-23T15:37:30Z

@vpisarev @asmorkalov @sl-sergei Please provide feedback on comments above.

sl-sergei · 2020-11-23T18:23:31Z

@alalek I have already fixed naming for SHA1 key locally and plan to reuse getCacheDirectory, just in the middle of progress

vpisarev · 2020-11-24T21:06:40Z

👍

alalek

samples/dnn/download_models.py

Probably its name is very specific. We may want to download other files (like annotations, datasets) too.

Also we can keep this "as is" and then proxy into cv.downloader from package when it will be available.

alalek · 2020-12-03T01:37:10Z

samples/dnn/README.md

+
+By running the following commands, you will get **MobileNetSSD_deploy.caffemodel** file:
+```bash
+export OPENCV_SAVE_DIR=save_dir_2


Lets introduce more specific name here.

It is unclear what is going to be saved here. May be test results?

Something like:

OPENCV_DOWNLOAD_DATA_PATH

OPENCV_DOWNLOAD_CACHE_DATA_PATH

P.S. _DATA_PATH is aligned with OPENCV_TEST_DATA_PATH

alalek · 2020-12-03T01:38:46Z

samples/dnn/download_models.py

+            temp_env = os.environ.get("TMPDIR", None)
+            if temp_env is None or not os.path.isdir(temp_env):
+                temp_dir = Path("/tmp")
+                print("Using world accessible cache directory. This may be not secure: ", temp_dir)


Lets rollback changes and add "TODO" here until we have resolved issues in #18931

…specific env variable

alalek

Thank you 👍

sl-sergei added 2 commits October 15, 2020 05:16

Initial commit. Utility classes and functions for downloading files

1b6e4ca

updated download script

78d827c

sl-sergei requested a review from alalek October 15, 2020 07:55

Support YAML parsing, update download script and configs

7212f9e

sl-sergei added 2 commits October 15, 2020 16:13

Fix problem with archived files

edefebc

fix models.yml

4485644

sl-sergei changed the title ~~WIP: Download scripts for DNN samples~~ WIP: Scripts for downloading models in DNN samples Oct 16, 2020

asmorkalov added this to the 5.0 milestone Oct 16, 2020

asmorkalov added category: python bindings category: samples labels Oct 16, 2020

asmorkalov assigned vpisarev Oct 16, 2020

sl-sergei added 3 commits October 16, 2020 13:12

Move download utilities to more appropriate place

8a11249

Fix script description

98c8cde

Update README

c19bab3

sl-sergei added 3 commits October 23, 2020 18:26

update utilities for broader range of files

3d02d43

fix loading with no hashsum provided

ebdfcb3

remove unnecessary import

4c7f911

fix for Python2

948abe1

sl-sergei added 4 commits November 12, 2020 09:56

Add usage examples for downloadFile function

f6161d5

Add more secure cache folder selection

5d798fa

Remove trailing whitespaces

b288f89

Fix indentation

05c88ff

Update function interface

0b960a0

sl-sergei changed the title ~~WIP: Scripts for downloading models in DNN samples~~ Scripts for downloading models in DNN samples Nov 18, 2020

asmorkalov requested a review from vpisarev November 18, 2020 12:27

vpisarev approved these changes Nov 18, 2020

View reviewed changes

alalek reviewed Nov 20, 2020

View reviewed changes

sl-sergei mentioned this pull request Nov 26, 2020

Add Python Bindings for getCacheDirectory function #18931

Closed

sl-sergei added 2 commits November 26, 2020 20:59

Change function for temp dir, change entry name in models.yml

f8f0c56

Update getCacheDirectory function call

51d5b43

alalek reviewed Dec 3, 2020

View reviewed changes

sl-sergei added 2 commits December 4, 2020 20:51

Return python implementation for cache directory selection, use more …

2b6d2ba

…specific env variable

Fix whitespace

1792c7f

alalek approved these changes Dec 11, 2020

View reviewed changes

alalek merged commit 1f3255d into opencv:3.4 Dec 11, 2020

alalek mentioned this pull request Dec 11, 2020

Merge 3.4 #19087

Merged

alalek mentioned this pull request Apr 9, 2021

(5.x) Merge 4.x #19885

Merged

Uh oh!

Conversation

sl-sergei commented Oct 15, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sl-sergei commented Oct 15, 2020

Uh oh!

asmorkalov commented Oct 16, 2020

Uh oh!

vpisarev commented Oct 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sl-sergei commented Oct 30, 2020

Uh oh!

alalek commented Oct 30, 2020

Uh oh!

sl-sergei commented Oct 30, 2020

Uh oh!

vpisarev commented Nov 9, 2020

Uh oh!

sl-sergei commented Nov 12, 2020

Uh oh!

vpisarev commented Nov 18, 2020

Uh oh!

asmorkalov commented Nov 20, 2020

Uh oh!

alalek left a comment

Choose a reason for hiding this comment

Uh oh!

alalek Nov 12, 2020

Choose a reason for hiding this comment

Uh oh!

alalek Nov 13, 2020

Choose a reason for hiding this comment

Uh oh!

alalek Dec 3, 2020

Choose a reason for hiding this comment

Uh oh!

sl-sergei commented Nov 20, 2020

Uh oh!

alalek commented Nov 23, 2020

Uh oh!

sl-sergei commented Nov 23, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vpisarev commented Nov 24, 2020

Uh oh!

alalek left a comment

Choose a reason for hiding this comment

Uh oh!

alalek Dec 3, 2020

Choose a reason for hiding this comment

Uh oh!

alalek Dec 3, 2020

Choose a reason for hiding this comment

Uh oh!

alalek left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

sl-sergei commented Oct 15, 2020 •

edited

Loading

vpisarev commented Oct 20, 2020 •

edited

Loading

sl-sergei commented Nov 23, 2020 •

edited

Loading