Celery Docker Upgrades by Nusnus · Pull Request #8531 · celery/celery

Nusnus · 2023-09-21T08:19:46Z

Allow build, lint, test & bash with the celery container using make commands.
Added CI to build the docker image to make sure it doesn't get broken in a future PR without knowing.
Default to Python 3.11 in the Dockerfile if not set otherwise via .python-version (pyenv).

For example running make docker-lint from the celery repo root:

make docker-lint

[+] Creating 4/0
 ✔ Container docker-azurite-1   Running                                                                                                                                           0.0s
 ✔ Container docker-rabbit-1    Running                                                                                                                                           0.0s
 ✔ Container docker-dynamodb-1  Running                                                                                                                                           0.0s
 ✔ Container docker-redis-1     Running                                                                                                                                           0.0s
ROOT: will run in automatically provisioned tox, host /home/developer/.pyenv/versions/3.11.5/bin/python3.11 is missing [requires (has)]: tox-gh-actions
ROOT: recreate env because python changed version_info=[3, 9, 13, 'final', 0]->[3, 11, 5, 'final', 0] | executable='/Users/nusnus/.pyenv/versions/3.9.13/bin/python3.9'->'/home/developer/.pyenv/versions/3.11.5/bin/python3.11' | virtualenv version='20.24.4'->'20.24.5'
ROOT: remove tox env folder /home/developer/celery/.tox/.tox
ROOT: install_deps> python -I -m pip install tox tox-gh-actions
ROOT: provision> .tox/.tox/bin/python -m tox -e lint
ROOT: tox-gh-actions won't override envlist because tox is not running in GitHub Actions
.pkg: _optional_hooks> python /home/developer/celery/.tox/.tox/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: get_requires_for_build_editable> python /home/developer/celery/.tox/.tox/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: build_editable> python /home/developer/celery/.tox/.tox/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
lint: install_package> python -I -m pip install --force-reinstall --no-deps /home/developer/celery/.tox/.tmp/package/17/celery-5.3.4-0.editable-py3-none-any.whl
lint: commands[0]> pre-commit run --all-files --show-diff-on-failure
[INFO] Initializing environment for https://github.com/asottile/pyupgrade.
[INFO] Initializing environment for https://github.com/PyCQA/flake8.
[INFO] Initializing environment for https://github.com/asottile/yesqa.
[INFO] Initializing environment for https://github.com/pre-commit/pre-commit-hooks.
[INFO] Initializing environment for https://github.com/pycqa/isort.
[INFO] Initializing environment for https://github.com/pre-commit/mirrors-mypy.
[INFO] Installing environment for https://github.com/asottile/pyupgrade.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/PyCQA/flake8.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/asottile/yesqa.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/pre-commit/pre-commit-hooks.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/pycqa/isort.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/pre-commit/mirrors-mypy.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
pyupgrade................................................................Passed
flake8...................................................................Passed
Strip unnecessary `# noqa`s..............................................Passed
check for merge conflicts................................................Passed
check toml...............................................................Passed
check yaml...............................................................Passed
mixed line ending........................................................Passed
isort....................................................................Passed
mypy.....................................................................Passed
.pkg: _exit> python /home/developer/celery/.tox/.tox/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
  lint: OK (35.25=setup[3.34]+cmd[31.92] seconds)
  congratulations :) (37.49 seconds)

The make help is also modified:

make help
docs                 - Build documentation.
test-all             - Run tests for all supported python versions.
distcheck ---------- - Check distribution for problems.
  test               - Run unittests using current python.
  lint ------------  - Check codebase for problems.
    apicheck         - Check API reference coverage.
    configcheck      - Check configuration reference coverage.
    readmecheck      - Check README.rst encoding.
    contribcheck     - Check CONTRIBUTING.rst encoding
    flakes --------  - Check code for syntax and style errors.
      flakecheck     - Run flake8 on the source code.
      flakepluscheck - Run flakeplus on the source code.
readme               - Regenerate README.rst file.
contrib              - Regenerate CONTRIBUTING.rst file
clean-dist --------- - Clean all distribution build artifacts.
  clean-git-force    - Remove all uncommitted files.
  clean ------------ - Non-destructive clean
    clean-pyc        - Remove .pyc/__pycache__ files
    clean-docs       - Remove documentation build artifacts.
    clean-build      - Remove setup artifacts.
bump                 - Bump patch version number.
bump-minor           - Bump minor version number.
bump-major           - Bump major version number.
release              - Make PyPI release.

Docker-specific commands:
  docker-build			- Build celery docker container.
  docker-lint        		- Run tox -e lint on docker container.
  docker-unit-tests		- Run unit tests on docker container, use '-- -k <TEST NAME>' for specific test run.
  docker-bash        		- Get a bash shell inside the container.

… tests via the docker compose broker and backend containers

…cker-unit-tests (partially supported), docker-bash

Nusnus · 2023-09-21T09:38:06Z

Updated PR message @auvipy .
PR is ready for review.

Nusnus · 2023-09-21T09:43:26Z

The integration tests are partially working in the docker environment so this PR doesn't fix 100% per-say the problem from #8447, but it should at least improve on the current capability, allowing the integration tests to even work with the broker and backend from the docker compose (which was 100% broken before - see changes in tox.ini).

…rrectly

Nusnus · 2023-09-21T09:54:52Z

Changed a string from '' to "" to trigger full CI to make sure this change doesn't create unwanted side effects.

codecov · 2023-09-21T09:58:17Z

Codecov Report

Patch and project coverage have no change.

Comparison is base (65ac2ac) 87.45% compared to head (9f331ce) 87.45%.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #8531   +/-   ##
=======================================
  Coverage   87.45%   87.45%           
=======================================
  Files         148      148           
  Lines       18499    18499           
  Branches     3158     3158           
=======================================
  Hits        16179    16179           
  Misses       2032     2032           
  Partials      288      288

Flag	Coverage Δ
unittests	`87.42% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

.github/workflows/docker.yml

* Added -docker environment to tox envlist to allow running integration tests via the docker compose broker and backend containers * Set default python for celery docker container to 3.11 from 3.8 * Added make commands: docker-build, docker-lint, docker-unit-tests, docker-unit-tests (partially supported), docker-bash * Added new Docker CI Workflow to validate the docker image is built correctly * No-op code change to trigger full CI

Nusnus added 3 commits September 21, 2023 09:56

Added -docker environment to tox envlist to allow running integration…

61ecbb7

… tests via the docker compose broker and backend containers

Set default python for celery docker container to 3.11 from 3.8

9dde3b2

Added make commands: docker-build, docker-lint, docker-unit-tests, do…

b2082f4

…cker-unit-tests (partially supported), docker-bash

Nusnus added Component: Docker Image CI labels Sep 21, 2023

Nusnus added this to the 5.3.x milestone Sep 21, 2023

Nusnus self-assigned this Sep 21, 2023

Nusnus force-pushed the docker_stuff branch 3 times, most recently from 5026d94 to e2a814e Compare September 21, 2023 08:37

auvipy self-requested a review September 21, 2023 08:45

Nusnus force-pushed the docker_stuff branch from e2a814e to 08fc734 Compare September 21, 2023 09:29

Nusnus marked this pull request as ready for review September 21, 2023 09:34

Nusnus requested review from auvipy and removed request for auvipy September 21, 2023 09:34

Nusnus added 2 commits September 21, 2023 12:51

Added new Docker CI Workflow to validate the docker image is built co…

ee2f026

…rrectly

No-op code change to trigger full CI

9f331ce

Nusnus force-pushed the docker_stuff branch from 08fc734 to 9f331ce Compare September 21, 2023 09:53

auvipy approved these changes Sep 21, 2023

View reviewed changes

.github/workflows/docker.yml Show resolved Hide resolved

auvipy merged commit 20b396d into celery:main Sep 21, 2023

Nusnus deleted the docker_stuff branch September 21, 2023 12:22

Nusnus mentioned this pull request Sep 21, 2023

Running tests in Docker fails #8447

Closed

18 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Celery Docker Upgrades#8531

Celery Docker Upgrades#8531
auvipy merged 5 commits intocelery:mainfrom
Katz-Consulting-Group:docker_stuff

Nusnus commented Sep 21, 2023 •

edited

Loading

Uh oh!

Nusnus commented Sep 21, 2023 •

edited

Loading

Uh oh!

Nusnus commented Sep 21, 2023

Uh oh!

Nusnus commented Sep 21, 2023

Uh oh!

codecov bot commented Sep 21, 2023 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

Nusnus commented Sep 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Nusnus commented Sep 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Nusnus commented Sep 21, 2023

Uh oh!

Nusnus commented Sep 21, 2023

Uh oh!

codecov bot commented Sep 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Nusnus commented Sep 21, 2023 •

edited

Loading

Nusnus commented Sep 21, 2023 •

edited

Loading

codecov bot commented Sep 21, 2023 •

edited

Loading