Dockerfile Build Optimizations #9733
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
|
ping me when it's ready |
afdbaac to
7d9baaf
Compare
Member
Author
@auvipy can you please check if this branch builds successfully locally for you? |
auvipy
approved these changes
Aug 1, 2025
Member
Author
|
Finally passing on Mac: make docker-build
[+] Building 1487.7s (47/47) FINISHED
=> [internal] load local bake definitions 0.0s
=> => reading from stdin 718B 0.0s
=> [docs internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 977B 0.0s
=> [celery internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 6.90kB 0.0s
=> [docs internal] load metadata for docker.io/library/python:3.12-slim-bookworm 1.2s
=> [celery internal] load metadata for docker.io/library/debian:bookworm-slim 0.7s
=> [auth] library/python:pull token for registry-1.docker.io 0.0s
=> [docs internal] load .dockerignore 0.1s
=> => transferring context: 348B 0.0s
=> [celery stage-0 1/25] FROM docker.io/library/debian:bookworm-slim@sha256:2424c1850714a4d94666ec928e24d86de958646737b1d113f5b2207be44d37d8 0.0s
=> [docs internal] load build context 0.2s
=> => transferring context: 52.43kB 0.1s
=> CACHED [celery stage-0 2/25] RUN apt-get update && apt-get install -y build-essential libcurl4-openssl-dev apt-utils debconf libffi-dev tk-dev xz-utils ca-certificates curl 0.0s
=> CACHED [celery stage-0 3/25] RUN : "developer" 0.0s
=> CACHED [celery stage-0 4/25] WORKDIR /provisioning 0.0s
=> CACHED [celery stage-0 5/25] RUN useradd -m -s /bin/bash developer 0.0s
=> CACHED [celery stage-0 6/25] RUN curl https://pyenv.run | bash 0.0s
=> CACHED [celery stage-0 7/25] RUN pyenv install 3.13 && pyenv install 3.12 && pyenv install 3.11 && pyenv install 3.10 && pyenv install 3.9 && pyenv install 3.8 && pyenv install pypy3.1 0.0s
=> CACHED [celery stage-0 8/25] RUN pyenv global 3.13 3.12 3.11 3.10 3.9 3.8 pypy3.10 0.0s
=> CACHED [celery stage-0 9/25] WORKDIR /home/developer 0.0s
=> CACHED [celery stage-0 10/25] COPY --chown=1000:1000 requirements /home/developer/requirements 0.0s
=> CACHED [celery stage-0 11/25] COPY --chown=1000:1000 docker/entrypoint /entrypoint 0.0s
=> CACHED [celery stage-0 12/25] RUN chmod gu+x /entrypoint 0.0s
=> CACHED [celery stage-0 13/25] RUN pyenv local 3.13 3.12 3.11 3.10 3.9 3.8 pypy3.10 0.0s
=> CACHED [celery stage-0 14/25] RUN --mount=type=cache,target=/home/developer/.cache/pip pyenv exec python3.13 -m pip install --upgrade pip setuptools wheel && pyenv exec python3.12 -m pip install --upgr 0.0s
=> [celery stage-0 15/25] RUN --mount=type=cache,target=/home/developer/.cache/pip pyenv exec python3.13 -m pip install -r requirements/default.txt -r requirements/dev.txt -r requirements/docs.txt 177.8s
=> [docs 1/10] FROM docker.io/library/python:3.12-slim-bookworm@sha256:9c1d9ed7593f2552a4ea47362ec0d2ddf5923458a53d0c8e30edf8b398c94a31 0.0s
=> CACHED [docs 2/10] RUN apt-get update && apt-get install -y build-essential && apt-get install -y texlive && apt-get install -y texlive-latex-extra && apt-get install -y dvipng && apt-get 0.0s
=> CACHED [docs 3/10] COPY /requirements /requirements 0.0s
=> CACHED [docs 4/10] RUN pip install -r /requirements/docs.txt -r /requirements/default.txt 0.0s
=> [docs 5/10] COPY . /celery 1.1s
=> [docs 6/10] RUN pip install /celery 18.2s
=> [docs 7/10] COPY docker/docs/start /start-docs 0.0s
=> [docs 8/10] RUN sed -i 's/\r$//g' /start-docs 0.1s
=> [docs 9/10] RUN chmod +x /start-docs 0.2s
=> [docs 10/10] WORKDIR /docs 0.0s
=> [docs] exporting to image 0.1s
=> => exporting layers 0.1s
=> => writing image sha256:802166a75251adea55b3a7632a7c2b611645a084bef08d5c3a7626bf50b3eca9 0.0s
=> => naming to docker.io/celery/docs 0.0s
=> [docs] resolving provenance for metadata file 0.0s
=> [celery stage-0 16/25] RUN --mount=type=cache,target=/home/developer/.cache/pip pyenv exec python3.12 -m pip install -r requirements/default.txt -r requirements/dev.txt -r requirements/docs.txt 71.2s
=> [celery stage-0 17/25] RUN --mount=type=cache,target=/home/developer/.cache/pip pyenv exec python3.11 -m pip install -r requirements/default.txt -r requirements/dev.txt -r requirements/docs.txt 61.9s
=> [celery stage-0 18/25] RUN --mount=type=cache,target=/home/developer/.cache/pip pyenv exec python3.10 -m pip install -r requirements/default.txt -r requirements/dev.txt -r requirements/docs.txt 63.1s
=> [celery stage-0 19/25] RUN --mount=type=cache,target=/home/developer/.cache/pip pyenv exec python3.9 -m pip install -r requirements/default.txt -r requirements/dev.txt -r requirements/docs.txt 64.2s
=> [celery stage-0 20/25] RUN --mount=type=cache,target=/home/developer/.cache/pip pyenv exec python3.8 -m pip install -r requirements/default.txt -r requirements/dev.txt -r requirements/docs.txt 61.6s
=> [celery stage-0 21/25] RUN --mount=type=cache,target=/home/developer/.cache/pip pyenv exec pypy3.10 -m pip install -r requirements/default.txt -r requirements/dev.txt -r requirements/docs.txt 831.5s
=> [celery stage-0 22/25] COPY --chown=1000:1000 . /home/developer/celery 0.5s
=> [celery stage-0 23/25] RUN --mount=type=cache,target=/home/developer/.cache/pip pyenv exec python3.13 -m pip install --no-deps -e /home/developer/celery && pyenv exec python3.12 -m pip install --no-de 78.1s
=> [celery stage-0 24/25] WORKDIR /home/developer/celery 0.1s
=> [celery stage-0 25/25] RUN git config --global --add safe.directory /home/developer/celery 0.6s
=> [celery] exporting to image 74.9s
=> => exporting layers 74.9s
=> => writing image sha256:075ebd28935b787f2af2748c2b2047e08a45a70c52871ffb7fce5307541e9473 0.0s
=> => naming to docker.io/celery/celery:dev 0.0s
=> [celery] resolving provenance for metadata file 0.1s
[+] Building 2/2
✔ docs Built 0.0s
✔ celery Built 0.0s |
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR implements Docker build optimizations to improve performance and caching efficiency in the Celery project's Dockerfile. The changes focus on leveraging Docker BuildKit features and restructuring build commands for better layer caching.
- Adds Docker BuildKit pip cache mounts to reduce build times and network usage
- Restructures Python installation and dependency management to optimize layer caching
- Enables Docker BuildKit in the Makefile build command
Reviewed Changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| docker/Dockerfile | Major restructuring with pip cache mounts, separated RUN commands for better caching, and environment variable format standardization |
| Makefile | Enables Docker BuildKit for the docker-build target |
Member
Author
make docker-lintAlso passing :) make docker-lint
[+] Creating 4/4
✔ Container docker-dynamodb-1 Running 0.0s
✔ Container docker-azurite-1 Running 0.0s
✔ Container docker-rabbit-1 Running 0.0s
✔ Container docker-redis-1 Running 0.0s
ROOT: will run in automatically provisioned tox, host /home/developer/.pyenv/versions/3.13.5/bin/python3.13 is missing [requires (has)]: tox-gh-actions
ROOT: install_deps> python -I -m pip install tox tox-gh-actions
ROOT: provision> .tox/.tox/bin/python -m tox -e lint
ROOT: tox-gh-actions won't override envlist because tox is not running in GitHub Actions
lint: install_deps> python -I -m pip install pre-commit -r /home/developer/celery/requirements/test.txt -r /home/developer/celery/requirements/pkgutils.txt
.pkg: install_requires> python -I -m pip install 'setuptools>=40.8.0'
.pkg: _optional_hooks> python /home/developer/celery/.tox/.tox/lib/python3.13/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: get_requires_for_build_editable> python /home/developer/celery/.tox/.tox/lib/python3.13/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: build_editable> python /home/developer/celery/.tox/.tox/lib/python3.13/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
lint: install_package_deps> python -I -m pip install 'backports.zoneinfo[tzdata]>=0.2.1; python_version < "3.9"' 'billiard<5.0,>=4.2.1' 'click-didyoumean>=0.3.0' 'click-plugins>=1.1.1' 'click-repl>=0.2.0' 'click<9.0,>=8.1.2' 'kombu<5.7,>=5.6.0b1' 'python-dateutil>=2.8.2' 'vine<6.0,>=5.1.0'
lint: install_package> python -I -m pip install --force-reinstall --no-deps /home/developer/celery/.tox/.tmp/package/1/celery-5.5.3-0.editable-py3-none-any.whl
lint: commands[0]> pre-commit run --all-files --show-diff-on-failure
[INFO] Initializing environment for https://github.com/asottile/pyupgrade.
[INFO] Initializing environment for https://github.com/PyCQA/flake8.
[INFO] Initializing environment for https://github.com/asottile/yesqa.
[INFO] Initializing environment for https://github.com/codespell-project/codespell.
[INFO] Initializing environment for https://github.com/codespell-project/codespell:tomli.
[INFO] Initializing environment for https://github.com/pre-commit/pre-commit-hooks.
[INFO] Initializing environment for https://github.com/pycqa/isort.
[WARNING] repo `https://github.com/pycqa/isort` uses deprecated stage names (commit, merge-commit, push) which will be removed in a future version. Hint: often `pre-commit autoupdate --repo https://github.com/pycqa/isort` will fix this. if it does not -- consider reporting an issue to that repo.
[INFO] Initializing environment for https://github.com/pre-commit/mirrors-mypy.
[INFO] Installing environment for https://github.com/asottile/pyupgrade.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/PyCQA/flake8.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/asottile/yesqa.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/codespell-project/codespell.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/pre-commit/pre-commit-hooks.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/pycqa/isort.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/pre-commit/mirrors-mypy.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
pyupgrade................................................................Passed
flake8...................................................................Passed
Strip unnecessary `# noqa`s..............................................Passed
codespell................................................................Passed
check for merge conflicts................................................Passed
check toml...............................................................Passed
check yaml...............................................................Passed
mixed line ending........................................................Passed
isort....................................................................Passed
mypy.....................................................................Passed
lint: OK (166.86=setup[97.42]+cmd[69.44] seconds)
congratulations :) (172.63 seconds) |
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Member
Author
Ready for review! @auvipy |
Nusnus
commented
Aug 3, 2025
auvipy
approved these changes
Aug 4, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.