Skip to content

CI: update toolchains to use latest GitHub runner images and rewrite set_matrix.py without polly#780

Merged
NeroBurner merged 23 commits intomasterfrom
ci_update_ubuntu_image_20.04
May 20, 2025
Merged

CI: update toolchains to use latest GitHub runner images and rewrite set_matrix.py without polly#780
NeroBurner merged 23 commits intomasterfrom
ci_update_ubuntu_image_20.04

Conversation

@NeroBurner
Copy link

@NeroBurner NeroBurner commented Apr 29, 2025

Update all used GitHub runner images (Linux, Windows and Mac) and update the set_matrix.py script to work without the polly toolchains.


  • Update from the deprecated ubuntu-20.04 images to the currently latest ubuntu-24.04 image. In the process update the gcc version from 7, 8, 9 to 12, 13, 14.
  • Update from the deprecated windows-2019 image with vs-16-2019 toolchain to windows-2022 with vs-17-2022 toolchain
  • Update from deprecated macos-12 image to macos-15
  • rewrite set_matrix.py to generate the CMAKE_TOOLCHAIN_FILE from the toolchain matrix name
  • update CI and remove the dependency on the polly toolchain project
  • remove analyze and sanitize jobs, no-one looks at them and they are a hassle to implement, PR welcome, but for now removed
  • remove the build.sh and build.cmd default build scripts and replace them with a build.py python script
  • remove build_hunter_tests.sh build script and use build.py there as well
  • remove unused jenkins.py file
  • update custom matrix.json and build.sh for the following projects:
    • filament
    • pthreads-win32
    • occt
    • freetype-gl
    • Boost

@NeroBurner NeroBurner marked this pull request as draft April 29, 2025 15:24
@NeroBurner NeroBurner linked an issue Apr 29, 2025 that may be closed by this pull request
@NeroBurner NeroBurner force-pushed the ci_update_ubuntu_image_20.04 branch 2 times, most recently from b23ca88 to c3e7d55 Compare May 5, 2025 05:34
@NeroBurner
Copy link
Author

NeroBurner commented May 5, 2025

from #781

windows-2019 image with vs-16-2019 toolchain will be deprecated next month. Will remove it from build matrix as well

The Windows 2019 Actions runner image will begin deprecation on 2025-06-01 and will be fully unsupported by 2025-06-30

actions/runner-images#12045

@NeroBurner
Copy link
Author

need to update macos-12 as well: #782

Update from the deprecated ubuntu-20.04 images to the currently latest
ubuntu-24.04 image. In the process update the `gcc` version from 7, 8, 9
to 12, 13, 14.
@NeroBurner NeroBurner force-pushed the ci_update_ubuntu_image_20.04 branch from f00adc4 to edbac40 Compare May 19, 2025 11:37
@NeroBurner
Copy link
Author

basic functionality restored, set_matrix.py and build.py should now work.

Now need to update custom matrix.json and build.sh scripts

found the following files:

$ find cmake/projects/ -name "*.sh" -o -name "*.json" -o -name "*.cmd"
cmake/projects/filament/ci/build.sh
cmake/projects/filament/ci/matrix.json
cmake/projects/Qt/generate.sh
cmake/projects/pthreads-win32/ci/matrix.json
cmake/projects/occt/ci/build.sh
cmake/projects/freetype-gl/ci/build.sh
cmake/projects/freetype-gl/ci/matrix.json
cmake/projects/Boost/scripts/create-boost-1-66-ios-fatlib.sh
cmake/projects/Boost/scripts/create-boost-ios-fatlib.sh
cmake/projects/Boost/ci/matrix.json
cmake/projects/Boost/generate.sh

so will have to fix the following projects:

  • filament
  • pthreads-win32
  • occt
  • freetype-gl
  • Boost

To test I'll locally try to successfully run the following command:

python .github/workflows/set_matrix.py Eigen filament pthreads-win32 occt freetype-gl Boost

@NeroBurner NeroBurner force-pushed the ci_update_ubuntu_image_20.04 branch from 3dc0793 to 536cffc Compare May 19, 2025 12:38
@NeroBurner NeroBurner marked this pull request as ready for review May 19, 2025 14:06
@NeroBurner NeroBurner changed the title CI: update Linux toolchains to use Ubuntu 24.04 images CI: update toolchains to use latest GitHub runner images and rewrite set_matrix.py without polly May 19, 2025
@NeroBurner NeroBurner added the infrastructure Problems with the CI infrastructure or CI jobs label May 19, 2025
@NeroBurner NeroBurner self-assigned this May 19, 2025
Noone looks at them and it is a hassle to implement in `set_matrix.py`
- set_matrix: add --projects override for local debugging
- set_matrix: add optional --output parameter
- set_matrix: default output to stdout without indent
- set_matrix: create toolchain files first steps non-complete
- set_matrix: handle specific android toolchains
- set_matrix: handle CMAKE_C_STANDARD and set undetectable
- set_matrix: toolchain creation for vs-xx-xxxx, no generator yet
- set_matrix: toolchain creation for mingw and msys
- set_matrix: newlines and c11 undetectable
- set_matrix: comment about generator fixed and libc++ closing ) fixed
- set_matrix: use android-ndk-api
- set_matrix: set VCVARSALL and generator
- set_matrix: add project_name to error message
- set_matrix: update path to vcvarsall.bat for vs-17-2022
- set_matrix: info about deprecation of vs-16-2019
- set_matrix: specified scripts are mandatory, no default build scripts
- set_matrix: no script for hunter_tests, but VERBOSE=True
- set_matrix: write out project name for CI job name
- set_matrix: set default python version for all jobs
- build.py: copy from jenkins.py and black format
- build.py: call cmake in script
- build.py: sys.exit not just exit
- build.py: fix cmake calling
- build.py: as posixpath for args
- build.py: use full path for TOOLCHAIN
- build.py: toolchain filepath as posix path
- build.py: all paths as posix
- build.py: alsways build with --config Release
- build.py: hunter_sha1 is str, not path
- build.py: only build Release config to save time
- build.py: CMP0069 is on per default since 3.9
- build.py: no chdir, remove all_release flag
- build.py: use cdir instead of recreate
- build.py: try without archive, maybe fixes Windows Store
- build.py: disable zip/archive creation, messes with Windows 10 Store
- build.py: fix release type, delete build dir, no archive
- build.py: handle optional SCRIPT environment variable
- build.py: allow hunter_tests as toolchain, special handling
- build.py: on main and master and GITHUB_USER_PASSWORD upload cache
- build.py: allow toolchain name and path and find file if needed
@NeroBurner NeroBurner force-pushed the ci_update_ubuntu_image_20.04 branch from b278c94 to 28c5ac7 Compare May 19, 2025 14:27
@NeroBurner
Copy link
Author

removed the Eigen and hunter_tests [WIP] commits to be ready for squash-merging

- ci: matrix: remove ndk from toolchain, depends on github runner image
- ci: matrix: use android-ndk-api-xxx
- ci: matrix: macos: move to macos-15
- ci: matrix: windows: move to vs-17-2022 on windows-2022 image
- ci: matrix: use Windows SDK 22000, as it is one of the available
- ci: matrix: remove python and script from default matrix, add comments
- ci: upload generated toolchains folder
- ci: on windows run VCVARSALL if needed
- ci: download artifacts for .github/toolchains folder
- ci: windows: download artifacts downloads folder directly, forgets full path
- ci: linux: download artifacts downloads folder directly, forgets full path
- ci: linux: use build.py script
- ci: run build.py for windows build as well
- ci: use build.py for hunter_tests as well
- ci: use matrix.project
- ci: use TOOLCHAIN with just name, remove hunter_tests job
- ci: on error show output of set_matrix.py
- ci: actually pass matrix.script as SCRIPT env-variable
also update docs such that custom build scripts are called by build.py
@NeroBurner NeroBurner force-pushed the ci_update_ubuntu_image_20.04 branch from 28c5ac7 to 93a5d9c Compare May 19, 2025 14:43
@NeroBurner
Copy link
Author

something is not yet working with the SCRIPT variable/file

 FileNotFoundError: [Errno 2] No such file or directory: '/home/runner/work/hunter/hunter/cmake/projects/filament/ci/build.sh'
PermissionError: [Errno 13] Permission denied: '/home/runner/work/hunter/hunter/cmake/projects/freetype-gl/ci/build.sh'

@NeroBurner
Copy link
Author

made the SCRIPT variable to be relative to repo-root and added executable bit for build.sh files. Now dependencies are installed

@NeroBurner NeroBurner merged commit 0ddc5ce into master May 20, 2025
151 of 209 checks passed
@NeroBurner NeroBurner deleted the ci_update_ubuntu_image_20.04 branch May 20, 2025 09:15
@NeroBurner
Copy link
Author

hyper-joel added a commit to Hypergene/hunter that referenced this pull request Mar 13, 2026
* Replace boostorg.jfrog.io to archives.boost.io URLs (cpp-pm#771)

Fix for cpp-pm#770

* Update minimum CMake to v3.10 (cpp-pm#769)

* Update minimum CMake to v3.10
* Update docs with new CMake minimum 3.10
* Update gate submodule to v0.11.0

Updates minimum CMake to v3.10

* Update Boost to 1.84.0, 1.85.0 and 1.86.0 (cpp-pm#775)

* Update 'Boost' to v1.84.0

Release notes: https://www.boost.org/users/history/version_1_84_0.html

* Update libraries and tests for Boost v1.84.0

* Add example for boost::conbalt
* Add Boost::cobalt to test matrix
* Add example for Boost::redis
* Add Boost::redis to test matrix

* Update 'Boost' to v1.85.0

Release notes: https://www.boost.org/users/history/version_1_85_0.html

* Update libraries and tests for Boost v1.85.0

* Add example for Boost::charconv
* Add Boost::charconv to test matrix

* refactor: move example for BoostProcess from Boost-process to BoostProcess

Renaming because the Boost::process lib in Boost became a lib for v2

* Update 'Boost' to v1.86.0

Release notes: https://www.boost.org/users/history/version_1_86_0.html

* Update libraries and tests for Boost v1.86.0

* Add example for Boost::predef
  * It seems something changed about Boost::predef in 1.86.0. It is listed as a lib but not build as a lib.
* Add Boost::predef to test matrix
* Add Boost::process to test matrix
* Add example for v2 of Boost::process
  * Boost::process changed in the 1.86.0 release.
    * v1 is header only
    * v2 is a library

---------

Co-authored-by: André Falk <andre.falk@tutanota.com>
Co-authored-by: simbahebinbo <lansheng228@163.com>

* hunter_get_lang_standard_flag: Add C++ 23/26 and C17/C23 check (cpp-pm#779)

Fixes: cpp-pm#756

* Add support for VS2022 Build Tools (cpp-pm#783)

* CI: update toolchains to use latest GitHub runner images and rewrite set_matrix.py without polly (cpp-pm#780)

Update all used GitHub runner images (Linux, Windows and Mac) and update the `set_matrix.py` script to work without the `polly` toolchains.

- Update from the deprecated ubuntu-20.04 images to the currently latest ubuntu-24.04 image. In the process update the `gcc` version from 7, 8, 9 to 12, 13, 14.
  - fixes: cpp-pm#778 
- Update from the deprecated `windows-2019` image with `vs-16-2019` toolchain to `windows-2022` with `vs-17-2022` toolchain
  - fixes: cpp-pm#781 
- Update from deprecated `macos-12` image to `macos-15`
  -  fixes: cpp-pm#782 
- rewrite `set_matrix.py` to generate the `CMAKE_TOOLCHAIN_FILE` from the `toolchain` matrix name
- update CI and remove the dependency on the [polly toolchain project](https://github.com/cpp-pm/polly)
- remove `analyze` and `sanitize` jobs, no-one looks at them and they are a hassle to implement, PR welcome, but for now removed
- remove the `build.sh` and `build.cmd` default build scripts and replace them with a `build.py` python script
- remove `build_hunter_tests.sh` build script and use `build.py` there as well
- remove unused `jenkins.py` file
- update custom `matrix.json` and `build.sh` for the following projects:
  - `filament`
  - `pthreads-win32`
  - `occt`
  - `freetype-gl`
  - `Boost`

* update msgpack to v4.1.3 (cpp-pm#644)

* update msgpack to v4.1.3
* update docu and example after review
* Update docs/packages/pkg/msgpack.rst

---------

Co-authored-by: NeroBurner <pyro4hell@gmail.com>

* Add note about updating copyright notes to the manual (cpp-pm#601)

one should update the copyright notices when a new package is created.
Point this out to the user in the guide to create a new package.

Co-authored-by: NeroBurner <pyro4hell@gmail.com>

* libsodium: update version to 1.0.18-p0 (cpp-pm#672)

Update the default version of libsodium to 1.0.18-p0

Fixes: cpp-pm#634

---------

Co-authored-by: NeroBurner <pyro4hell@gmail.com>

* Update civetweb to v1.13-p0 (cpp-pm#304)

* GIT_SUBMODULE fix: File not found .git/HEAD error (cpp-pm#168)

* Fix File not found .git/HEAD error
* Verify if git path output is absolute

* docs: update URL to tar file for boost-components documentation (cpp-pm#561)

* fix link-all.py with high CPU count and little files (cpp-pm#785)

Code changes by https://github.com/aggiehorns

When we have more CPUs than files to link the `files_per_job` variable
got `0`. And then the begin and end indexes got wrong and multiple jobs
tried to link the same file.

Fixes: cpp-pm#664

* set_matrix: simplify regex, fix typos (cpp-pm#786)

Fix some typos in output, variable-names and comments.

The usage of `[\d]` in regex is explicit, but also uneccessarily
complex. Simplify a bit.

* set_matrix: format

* CI: run check-urls: rework script to run on changed files and let it run in CI (cpp-pm#787)

Rework the `check-urls.py` script to make it work with the changed files file from GitHub Actions.

Add a new job to run `check-urls.py` and check projects with changed `hunter.cmake` files and their URLs.

Fixes: cpp-pm#598

* Boost: remove USE_CONFIG_FROM_BOOST hunter option, remove old (cpp-pm#784)

Remove the option `USE_BOOST_CONFIG`. It is now essentially always
enabled.

Remove all Boost versions older than 1.72, as the old versions don't
supply a `BoostConfig.cmake` file yet.

Remove the code and templates to create a `BoostConfig.cmake` with
Hunter.

Update the examples and remove the `-useBoostConfig` example folders.

Fixes: cpp-pm#723

* OpenBLAS: silence warnings (cpp-pm#792)

The OpenBLAS build has uncountable many warnings. Disable them as we
don't need them. Only errors are really interesting.

* docs: fix warning about matrix.json not valid for code highlighting (cpp-pm#793)

Fixing the following warning:
```sh
Warning, treated as error:
hunter/docs/creating-new/test/ci.rst:84:Could not lex literal_block as "JSON". Highlighting skipped.
```

The referenced codeblock is the following:

```rst
.. literalinclude:: ../../../.github/workflows/ci/matrix.json
  :language: JSON
```

The `matrix.json` file contains `//` comments, which are not valid json tripping the highlighting.

Removing the `:language: JSON` part fixes this warning

Fixes: cpp-pm#790

* docs: msgpack: fix spelling warnings/errors (cpp-pm#795)

The documentation for `msgpack` has two spelling errors:

```
writing output... [ 63%] packages/pkg/msgpack
packages/pkg/msgpack.rst:21:msgpackc:["mudpack"]
packages/pkg/msgpack.rst:21:cxx:["xxx", "xx", "cox", "c xx", "OSX"]

...

 Spelling checker messages written to /home/runner/work/hunter/hunter/docs/_spelling/output.txt
Warning, treated as error:
Found 2 misspelled words
```

Update the documentation properly escaping the target names and in the
process making the spellchecker ignoring the "codeblocks".

Fixes: cpp-pm#794

* docs: quick-start: fix link to HUNTER_ROOT env variable (cpp-pm#800)

`HUNTER_ROOT` link goes to old/not available wiki. Update the link to
use an internal reference the the `HUNTER_ROOT` environment variable
section in the documentation.

Fixes: cpp-pm#682

* nlohmann json update to 3.12.0 (cpp-pm#803)

* update nlohmann_json 3.12.0

* docs: add build_docs.sh script, installing debian packages through apt

Replace the `jenskins.sh` script using pip with `make_docs.sh` script
expected to run on Ubuntu.

For apt install set non-interactive env variable.

Update CI to run on `ubuntu-24.04` instead of `ubuntu-latest`.

Remove the files `jenkins.sh` and `requirements.txt`

* docs: enable myst-parser for markdown support, ignore old-wiki md files

* docs: conf.py: explicitly set language "en"

* docs: update version to v0.26 to match release tags

* docs: conf.py: remove deprecated get_html_theme_path call (cpp-pm#810)

For Sphinx `v8.1.3` compatibility remove the `get_html_theme_path` call.

Fixes: cpp-pm#807

* docs: port foo.rst to foo.md (cpp-pm#808)

I like to write documentation in MarkDown better than in `*.rst`.

First step is to have new package documentation in MarkDown. So port the
reference `foo.rst` to `foo.md`.

* docs: fix deprecation warning replacing spelling with spelling:word-list (cpp-pm#811)

When building the docs on Ubuntu 24.04 many deprecation warnings are shown

```
direct use of the spelling directive is deprecated, replace ".. spelling::" with ".. spelling:word-list::"
```

Fixes: cpp-pm#806

* docs: foo.md: remove double space from enumeration

* libdeflate: add new package with v1.24-p0 (cpp-pm#812)

Need to use a patched version of `libdeflate` as the created CMake
config contains absolute paths breaking the caching.

Once ebiggers/libdeflate#434 is merged and
released we can use pure upstream.

Fixes: cpp-pm#804

* libdeflate: update copyright

* TIFF: update to v4.7.0-p0 (cpp-pm#813)

Unpinn OpenCV 4.10 because TIFF 4.7.0 fixes build issues on Linux.
Fixes: cpp-pm#765

Remove old `FindTIFF.cmake` module. Not needed anymore (and probably
wasn't needed for a long time).

Update to use https://github.com/cpp-pm/libtiff/releases/tag/v4.7.0-p0

hunterized dependencies:
- `zlib`: default ON, only dependency available in previous 4.0.2 hunterized version)
- `libdeflate`: default: ON, available in Hunter starting with `v0.26.3` cpp-pm#804
- `jpeg`: default OFF, newly hunterized
- `lzma`: default OFF, newly hunterized
- `zstd`: default OFF, newly hunterized

Not hunterized dependencies:
- `jbig=OFF`
- `lerc=OFF`
- `webp=OFF`

Update the TIFF example to use use `stdio.h` `remove()` instead of `unlink()`.

Fixes: cpp-pm#789

* TIFF: fix mention about zstd being supported

* Add new package OpenJPH 0.23.0 (cpp-pm#814)

Add upstream version of package `OpenJPH`. The `TIFF` dependency is
optional and I had some issues when adding `hunter_add_package(TIFF)`
with missing includes. So adding with `TIFF` disabled.

Fixes: cpp-pm#805

* Imath: update to 3.2.1 (cpp-pm#817)

Update `Imath` package for OpenEXR 3.4.0 compatibility as described in
the `OpenEXR` install documentation (3.4.0+ should use Imath 3.2).

Fixes: cpp-pm#816

* OpenSSL: strip libunwind flags for Clang-based Android NDK (cpp-pm#815)

Clang-based Android NDKs bundle unwind support and do not require explicit linking.
Passing malformed flags like `-l-l:libunwind.a` caused build failures with `ld.lld` on NDK r23+.
This change removes those flags from configure options to ensure successful builds.

* OpenEXR: add version 3.4.0-p0 (cpp-pm#818)

Update `OpenEXR` with Hunterized `v3.4.0-p0` version:
https://github.com/cpp-pm/openexr/releases/tag/v3.4.0-p0

Dependencies:
- add `Imath` as dependency (unchanged)
- add `libdeflate` as depencency, introduced in `v3.2.0`
- removed `zlib` dependency since `v3.2.0`, replaced by `libdeflate`
- add `OpenJPH` as dependency, since `v3.4.0`

Fixes: cpp-pm#788

* CI: install python requests module to fix upload cache script (cpp-pm#820)

On the `master` branch the pipelines fail at the `upload-cache-to-github.py`
script as it is missing the `requests` python module.

Install it before the build to have a working cache upload again.

Fixes: cpp-pm#819

* PNG: update to v1.6.36-p4 with CMake 3.11 requirement (cpp-pm#823)

Update to release https://github.com/cpp-pm/libpng/releases/tag/v1.6.36-p4

* Jpeg: update to v9f-p1 with CMake 3.10 (cpp-pm#822)

Integrate cmakeified version: https://github.com/cpp-pm/jpeg/releases/tag/v9f-p1

Fixes: cpp-pm#821

* CI: install gitpython python module for cache upload script (cpp-pm#829)

Fixes: cpp-pm#827

* giflib: update to v5.2.2-p0 (cpp-pm#828)

use https://github.com/cpp-pm/giflib/releases/tag/v5.2.2-p0 with updated
minimum CMake 3.10 version.

Fixes: cpp-pm#824

* minizip: update to 1.0.1-p4. (cpp-pm#833)

Update to release https://github.com/cpp-pm/minizip/releases/tag/v1.0.1-p4

Fixes: cpp-pm/minizip#1

* ZLIB: update to v1.3.1-p0. (cpp-pm#831)

Update to release https://github.com/cpp-pm/zlib/releases/tag/v1.3.1-p0

Fixes: cpp-pm#735

* OpenCV+Extra+ippicv: add version v4.12.0-p0 (cpp-pm#832)

Update OpenCV, OpenCV-Extra and ippicv to `v4.12.0-p0`.
Those packages belong together and should be updated at once.

- OpenCV: update to 4.12.0-p0
- OpenCV-Extra: update to 4.12.0 release
- ippicv: update to 20250130 for OpenCV 4.12.0
  - keep APPLE version to ippicv 20240201 from OpenCV 4.10.0
- add `WITH_VTK=OFF`
- Unpin OpenCV for all as Linux build is now fixed with fixed
  TIFF/OpenEXR

Fixes: cpp-pm#801

* opencv: update to v4.12.0-p1 with CUDA 12.9 fix (cpp-pm#835)

Update to `4.12.0-p1` with CUDA 12.9 fix cherry-picked:
opencv/opencv@e2d87de

Link to Release: https://github.com/cpp-pm/opencv/releases/tag/v4.12.0-p1

Fixes: cpp-pm#834

* Add Boost version 1.84.0,  1.85.0, and 1.86.0

* Support for Boost 1.86 with added component configuration

* Update download url for Boost

Update download url from jfrog to boost-archive.

---------

Co-authored-by: Aliaksandr Anishchuk <oav.other@gmail.com>
Co-authored-by: Harry Mallon <hjmallon@gmail.com>
Co-authored-by: Dominik Nussbaumer <4436784+tnixeu@users.noreply.github.com>
Co-authored-by: André Falk <andre.falk@tutanota.com>
Co-authored-by: simbahebinbo <lansheng228@163.com>
Co-authored-by: NeroBurner <pyro4hell@gmail.com>
Co-authored-by: Ronny Nowak <dargun@gmx.net>
Co-authored-by: craffael <raffael@casagrande.ch>
Co-authored-by: Graham Reeds <graham.reeds@gmail.com>
Co-authored-by: Rahul Sheth <rahul@snap.com>
Co-authored-by: Joerg-Christian Boehme <joerg@chaosdorf.de>
Co-authored-by: Timothy Pak <httpak@connect.ust.hk>
Co-authored-by: Andrei Svatko <andrei.svatko@regula.by>
Co-authored-by: Olli Wang <olliwang@ollix.com>
Co-authored-by: Ola Engwall <ola.engwall@hypergene.se>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

infrastructure Problems with the CI infrastructure or CI jobs

Projects

None yet

1 participant