Support downloading 3rdparty resources from Gitcode & Gitlab-style mirrors#21531
Support downloading 3rdparty resources from Gitcode & Gitlab-style mirrors#21531alalek merged 35 commits intoopencv:4.xfrom
Conversation
…m & githubusercontent.com
…ling ocv_download
…o names for replace github archives and raw githubusercontent
… ocv_download; update md5sum for archives
… are downloaded from release page
|
Current development process doesn't assume that we have any links from other mirrored resources. In general these mirrors are updated only after relevant PRs are merged (and we don't have control for such mirrors). So it makes sense:
P.S. PR with CMake hooks support: #10727. One of usage example is here. |
|
@alalek I re-implemented the mirror support using |
alalek
left a comment
There was a problem hiding this comment.
Thank you for update! Looks much better!
|
Hello @alalek , I noticed that you mentioned this patch breaks setting custom TBB url and hash for CI to test new TBB release. Could you give me an example or more details on this so that I may solve the problem before this Friday? |
alalek
left a comment
There was a problem hiding this comment.
Need to design pipeline of replacing, adding, or updating of 3rdparty components during the development.
Some of such updates requires mirrors updates and/or mirrors configurations updates.
|
@alalek I can provide a document in details about replacing/adding/updating 3rdparty components. However, it can be too long to put into CMake scripts as comments. Do you think we should write a wiki page for such a document? |
There was a problem hiding this comment.
No major objections from my side. See below.
@vpisarev @asmorkalov please review too.
…use warning message for warnings
cmake/OpenCVDownload.cmake
Outdated
| message(WARNING "ocv_init_download: Unknown host detected. Using github.com to download 3rdparty components.") | ||
| elseif(NOT __HOST STREQUAL "github") | ||
| message(STATUS "ocv_init_download: Using ${__HOST}-hosted mirror to download 3rdparty components.") | ||
| ocv_cmake_hook_append(OCV_DOWNLOAD_MIRROR_${__HOST} "${CMAKE_CURRENT_SOURCE_DIR}/cmake/mirrors/${__HOST}.cmake") |
There was a problem hiding this comment.
In the latest commit, either "gitcode.cmake" or "custom.cmake" can be added into hooks. So I think it is not necessary to check whether those files exist.
vpisarev
left a comment
There was a problem hiding this comment.
tested it on my machine (macos); works great
Support downloading 3rdparty resources from Gitcode & Gitlab-style mirrors * replace github.com with gitcode.net for ocv_download * replace raw.githubusercontent.com with gitcode.net for ocv_download * rename funtions and remove some comments * add options for custom mirrors, which simply replace domain github.com & githubusercontent.com * run ocv_init_download once; replace DL_URL with mirrored one when calling ocv_download * fix for empty download links when not using mirror * fix bugs: set(.. .. PARENT_SCOPE) for ocv_init_download; correct macro names for replace github archives and raw githubusercontent * adjusted mirror swapping impl: replace with mirrored link before each ocv_download; update md5sum for archives * fix a bug: macro invoked with incorrect arguments by non-set vars * enclose if statement * workable impl * shorten the var names of two key options * scalable implementation of downloading from mirror and using custom mirror * improve ocv_init_download help message * fix the different extracted directory name in case of ADE & TBB which are downloaded from release page * improve help message printing * Download ADE & TBB using commit ids instead of from release pages * support custom mirrors on downloading archives * improve hints * add missing parentheses * reset ocv_download calls * mirror support implementation using ocv_cmake_hook & ocv_cmake_hook_append * move ocv_init_download into cmake/OpenCVDownload.cmake * move ocv_cmake_hook before checking CMake cache * improve hints when not fetching as git repo * add WORKING_DIRECTORY in execute_process in ocv_init_download * use OPENCV_DOWNLOAD_MIRROR_ID * add custom.cmake for custom mirror * detect github origin * fix broken var name * download from github by default if custom tbb is set * add checksum checks for gitcode.cmake before replacing urls and checksums * add checksum checks for custom.cmake before replacing urls and checkusms * use description specify instead of set for messages in custom.cmake; use warning message for warnings * updates and fixes
Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
Patch to opencv_extra has the same branch name.