Further improve image caching for Breeze#22625
Conversation
96a2f88 to
3e0b95e
Compare
|
This will nicely handle moving "to" the new cache. The "inline" image cache should always be used from the remote image in
|
f3f9820 to
043b2ce
Compare
|
I hope that one will make it finally solved with multi-staging and multi-platform builds. I've learned a TON about it and there are still some weird behaviours.
Very, very true. |
9f83b28 to
020a490
Compare
8df1640 to
eede348
Compare
|
Hello! I would really love that one to merge :). It also includes re-enabling hadolint which also brought ARM support last night (and already supports the new dockerfile:1.4 features that we use. |
|
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease. |
eede348 to
f7a1380
Compare
In order to allow "big" rebuilds we remove the image tag before rebuilding, to make sure that the remote and not local image is used as source of cache. This is consequence of using inlined cache. Files are added to the Python Base image image - the base layers will be shared with other stages but we will make sure that the script stages are different for different platforms. Also - when there is no image at all we fail pre-commit. This should handle the situation when we tried to build the image and stopped it in-between. Hadolint released a new version of their checker - with support for the new Dockerfile buildkit features and native support for ARM so we are enabling it back. See hadolint/hadolint#803 Finally we still need some files that we cannot inline, because they are only needed for source build and they are too long to inline (yarn.lock for example). In order to keep the cache working for all umasks we need to bring back group fixing.
f7a1380 to
84dec5a
Compare
|
Random failure. Merging :) |
In order to allow "big" rebuilds we remove the image tag
before rebuilding, to make sure that the remote and not local image
is used as source of cache. This is consequence of using inlined
cache.
Files are added to the Python Base image image - the base
layers will be shared with other stages but we will make sure
that the script stages are different for different platforms.
Also - when there is no image at all we fail pre-commit. This
should handle the situation when we tried to build the image and
stopped it in-between.
Hadolint released a new version of their checker - with support
for the new Dockerfile buildkit features and native support for
ARM so we are enabling it back.
See hadolint/hadolint#803
Finally we still need some files that we cannot inline, because
they are only needed for source build and they are too long
to inline (yarn.lock for example). In order to keep the cache
working for all umasks we need to bring back group fixing.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.