Add docker attestation#13066
Add docker attestation#13066rouault merged 10 commits intoOSGeo:masterfrom j9ac9k:add-docker-attestation
Conversation
Make use of the docker attestation capability when generating images. Furthermore, use ARM64 GHA runners for generating ARM64 images, which is significantly faster.
I'd note we currently don't publish Docker images through github action. This is only used for now to test pull requests that modify the Dockerfile recipe. So I'm no sure if attestation will bring any benefit right now, but that's a step in the right direction.
One of the difficulty currently is the build process time, in particular for Alpine arm64 images that don't have a x86_64 cross compiler for arm64. But I didn't realize that github was providing ubuntu 24.04 arm64 images. That's a game changer. So ultimately we'd need to have a x86_64 job producing the x86_64 image, a arm64 producing the arm64 image, and another one waiting for the 2 ones to have finished to create the combined image. |
I cannot type. Co-authored-by: Even Rouault <even.rouault@spatialys.com>
That's what the setup currently is in pdal/pdal, since you can make jobs dependent on each other, you can have the job that generates the manifest run only when the individual images have completed without error. https://github.com/PDAL/PDAL/blob/master/.github/workflows/docker.yml I agree that GHA offering linux/arm64 platforms is a game changer. When testing runtimes for docker images, I went from ~3 hours to ~20 minutes for the ARM images. That said, there is no arm64 image I'm aware of of linuxkit/binfmt:v0.8, hence why I need to specify to use the amd64 image regardless of the platform. I can certainly setup the docker CI to run from master on a daily basis. EDIT: the docker CI job not pushing to GHCR as part of the release process now makes a lot more sense, here I was pulling my hair out wondering why I couldn't get that working, ...when that wasn't how this was made to work to begin with. |
|
Currently the GHA runs ubuntu small and ubuntu full on both amd64 and arm64, but the alpine images run only on amd64. Looking at the tags on ghcr.io/osgeo/gdal, looks like arm64 images of alpine models are distributed, so I should create those as well I assume? |
yes please (The current images pushed to ghcr.io are built from my own PC. Cooking time: > 12 hours. Most of that spent for the arm64 images for alpine-small and alpine-normal) |
|
What should the tag naming scheme be here? Especially for the daily builds from master. Also, not sure how proj versions work their way into this, can you elaborate on that? |
We currently use for master builds: For releases:
For GDAL master builds, the docker/util.sh script defaults to using PROJ master For GDAL releases, currently I manually specify the latest PROJ version. Example for the current in-cooking GDAL 3.11.4 release: |
|
Deleted a previous comment because I couldn't read. The proprietary-sdks, those should only be included the |
they are x64 only (some might be arm64 but we don't bother) and we should not pushed them to ghcr.io given their license. We just test they build |
.github/workflows/docker.yml
Outdated
| tag: ["ubuntu-full", "ubuntu-small", "alpine-small", "alpine-normal"] | ||
| steps: | ||
| - name: Set PUSH_PACKAGE due to schedule | ||
| if: github.event.schedule=='*/0 * * * *' |
There was a problem hiding this comment.
I don't know if environment variables are transferred across jobs, but if they are transferred this will overwrite the PUSH_PACKAGE from the build step in the previous job.
There was a problem hiding this comment.
If you want to pass a value from a step in one job in a workflow to a step in another job in the workflow, you can define the value as a job output. You can then reference this job output from a step in another job. For more information, see Workflow syntax for GitHub Actions.
Extra steps need to be made to pass an environment variable that was set in the step of one job to another job.
Make use of the docker attestation capability when generating images.
Furthermore, use ARM64 GHA runners for generating ARM64 images, which is significantly faster.
What does this PR do?
Add docker attestation features when creating the docker images. Furthermore, have docker use the ARM linux runners when generating the ARM images.
I did my best to test this on my fork, but a lot of this code has to do with the release process of gdal, which I was not able to test as thoroughly as I would have liked (to ensure the attestation for example is indeed distributed). Any guidance towards things that are incorrect would be greatly appreciated.
Furthermore, this PR will make it such that
docker buildxis utilized (which is not the default behavior currently).Happy to explain any of the other changes.
What are related issues/pull requests?
None
Tasklist
Environment
Github Actions (CI).