Refactor integration and smoke tests CI #9855

Nusnus · 2025-08-09T21:53:35Z

No description provided.

codecov · 2025-08-09T21:57:51Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 78.63%. Comparing base (33eb148) to head (c169268).
⚠️ Report is 71 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #9855   +/-   ##
=======================================
  Coverage   78.63%   78.63%           
=======================================
  Files         153      153           
  Lines       19198    19198           
  Branches     2546     2546           
=======================================
  Hits        15096    15096           
- Misses       3807     3809    +2     
+ Partials      295      293    -2

Flag	Coverage Δ
unittests	`78.61% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

auvipy

wah only 3 failures! I guess we can now focus on the remaining pr

auvipy

only one test is failing now

Nusnus · 2025-08-11T10:17:02Z

only one test is failing now

Yeah but I had to rerun many times.
I'm trying to find the source, I'm suspecting something with Redis but I'm still not sure.

I'll keep investigating bro, LMK if you have any idea/tip!

auvipy · 2025-08-11T11:06:07Z

could be redis multi threading or race condition issue or this could be in billiard or may be not. #9837 (comment)

…s from 2 -> 3 (fail/retry faster)

…ition (celery#9770)" This reverts commit 6d8bfd1.

auvipy · 2025-08-13T05:06:37Z

only one old known test is failing ::test_chain::test_chain_error_handler_with_eta PASSED
t/integration/test_canvas.py::test_chain::test_groupresult_serialization +++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++

  File "/opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/threading.py", line 937, in _bootstrap
    self._bootstrap_inner()
  File "/opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/threading.py", line 980, in _bootstrap_inner
    self.run()
  File "/opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/threading.py", line 917, in run
    self._target(*self._args, **self._kwargs)
  File "/home/runner/_work/celery/celery/celery/contrib/testing/worker.py", line 77, in start
    return super().start()
  File "/home/runner/_work/celery/celery/celery/worker/worker.py", line 203, in start
    self.blueprint.start(self)
  File "/home/runner/_work/celery/celery/celery/bootsteps.py", line 116, in start
    step.start(parent)
  File "/home/runner/_work/celery/celery/celery/bootsteps.py", line 365, in start
    return self.obj.start()
  File "/home/runner/_work/celery/celery/celery/worker/consumer/consumer.py", line 341, in start
    blueprint.start(self)
  File "/home/runner/_work/celery/celery/celery/bootsteps.py", line 116, in start
    step.start(parent)
  File "/home/runner/_work/celery/celery/celery/worker/consumer/consumer.py", line 772, in start
    c.loop(*c.loop_args())
  File "/home/runner/_work/celery/celery/celery/worker/loops.py", line 97, in asynloop
    next(loop)
  File "/home/runner/_work/celery/celery/.tox/3.9-integration-redis/lib/python3.9/site-packages/kombu/asynchronous/hub.py", line 317, in create_loop
    events = poll(poll_timeout)
  File "/home/runner/_work/celery/celery/.tox/3.9-integration-redis/lib/python3.9/site-packages/kombu/utils/eventio.py", line 83, in poll
    return self._epoll.poll(timeout if timeout is not None else -1)
~~~~~~~~~~~~~~~~~~~~~ Stack of Thread-1 (139766358075072) ~~~~~~~~~~~~~~~~~~~~~~
  File "/opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/threading.py", line 937, in _bootstrap
    self._bootstrap_inner()
  File "/opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/threading.py", line 980, in _bootstrap_inner
    self.run()
  File "/opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/threading.py", line 917, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/logging/handlers.py", line 1537, in _monitor
    record = self.dequeue(True)
  File "/opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/logging/handlers.py", line 1486, in dequeue
    return self.queue.get(block)
  File "/home/runner/_work/celery/celery/.tox/3.9-integration-redis/lib/python3.9/site-packages/billiard/queues.py", line 99, in get
    res = self._recv_bytes()
  File "/home/runner/_work/celery/celery/.tox/3.9-integration-redis/lib/python3.9/site-packages/billiard/connection.py", line 240, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/home/runner/_work/celery/celery/.tox/3.9-integration-redis/lib/python3.9/site-packages/billiard/connection.py", line 453, in _recv_bytes
    buf = self._recv(4)
  File "/home/runner/_work/celery/celery/.tox/3.9-integration-redis/lib/python3.9/site-packages/billiard/connection.py", line 419, in _recv
    chunk = read(handle, remaining)
+++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++
Error: Final attempt failed. Timeout of 900000ms hit

Nusnus · 2025-08-13T12:28:19Z

only one old known test is failing ::test_chain::test_chain_error_handler_with_eta PASSED t/integration/test_canvas.py::test_chain::test_groupresult_serialization +++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++

  File "/opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/threading.py", line 937, in _bootstrap
    self._bootstrap_inner()
  File "/opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/threading.py", line 980, in _bootstrap_inner
    self.run()
  File "/opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/threading.py", line 917, in run
    self._target(*self._args, **self._kwargs)
  File "/home/runner/_work/celery/celery/celery/contrib/testing/worker.py", line 77, in start
    return super().start()
  File "/home/runner/_work/celery/celery/celery/worker/worker.py", line 203, in start
    self.blueprint.start(self)
  File "/home/runner/_work/celery/celery/celery/bootsteps.py", line 116, in start
    step.start(parent)
  File "/home/runner/_work/celery/celery/celery/bootsteps.py", line 365, in start
    return self.obj.start()
  File "/home/runner/_work/celery/celery/celery/worker/consumer/consumer.py", line 341, in start
    blueprint.start(self)
  File "/home/runner/_work/celery/celery/celery/bootsteps.py", line 116, in start
    step.start(parent)
  File "/home/runner/_work/celery/celery/celery/worker/consumer/consumer.py", line 772, in start
    c.loop(*c.loop_args())
  File "/home/runner/_work/celery/celery/celery/worker/loops.py", line 97, in asynloop
    next(loop)
  File "/home/runner/_work/celery/celery/.tox/3.9-integration-redis/lib/python3.9/site-packages/kombu/asynchronous/hub.py", line 317, in create_loop
    events = poll(poll_timeout)
  File "/home/runner/_work/celery/celery/.tox/3.9-integration-redis/lib/python3.9/site-packages/kombu/utils/eventio.py", line 83, in poll
    return self._epoll.poll(timeout if timeout is not None else -1)
~~~~~~~~~~~~~~~~~~~~~ Stack of Thread-1 (139766358075072) ~~~~~~~~~~~~~~~~~~~~~~
  File "/opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/threading.py", line 937, in _bootstrap
    self._bootstrap_inner()
  File "/opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/threading.py", line 980, in _bootstrap_inner
    self.run()
  File "/opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/threading.py", line 917, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/logging/handlers.py", line 1537, in _monitor
    record = self.dequeue(True)
  File "/opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/logging/handlers.py", line 1486, in dequeue
    return self.queue.get(block)
  File "/home/runner/_work/celery/celery/.tox/3.9-integration-redis/lib/python3.9/site-packages/billiard/queues.py", line 99, in get
    res = self._recv_bytes()
  File "/home/runner/_work/celery/celery/.tox/3.9-integration-redis/lib/python3.9/site-packages/billiard/connection.py", line 240, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/home/runner/_work/celery/celery/.tox/3.9-integration-redis/lib/python3.9/site-packages/billiard/connection.py", line 453, in _recv_bytes
    buf = self._recv(4)
  File "/home/runner/_work/celery/celery/.tox/3.9-integration-redis/lib/python3.9/site-packages/billiard/connection.py", line 419, in _recv
    chunk = read(handle, remaining)
+++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++
Error: Final attempt failed. Timeout of 900000ms hit

Almost there!
Hehe..

…tdown

This reverts commit 3da612d.

…rker shutdown" This reverts commit cebcb2b.

…dition (celery#9770)" This reverts commit b10a55c.

…ion tests

This reverts commit ade8cbd.

Nusnus self-assigned this Aug 9, 2025

Nusnus force-pushed the cifix branch 3 times, most recently from e98f537 to 2c7c397 Compare August 9, 2025 22:41

auvipy reviewed Aug 10, 2025

View reviewed changes

auvipy added this to the 5.6.0 milestone Aug 10, 2025

auvipy reviewed Aug 11, 2025

View reviewed changes

Nusnus added 6 commits August 12, 2025 14:35

Disabled tests test_multiprocess_producer and test_multithread_producer

5c601c9

Refactor integration tests CI

e8b54e5

Disable test_quorum_queue_qos_cluster_simulation.py

250518a

Reduce integration tests timeout from 30m -> 20m and increase attempt…

a6d6f2c

…s from 2 -> 3 (fail/retry faster)

Increase max attempts from 3 -> 5 with 1m break between each retry

64fc4dc

TMP Dont wait for unit tests

3da612d

Nusnus force-pushed the cifix branch from b2db0dd to 3da612d Compare August 12, 2025 11:37

Changed retry settings

7823276

Nusnus force-pushed the cifix branch from ff24045 to 7823276 Compare August 12, 2025 14:03

Nusnus added 2 commits August 12, 2025 19:07

Revert "Add xfail test for RabbitMQ quorum queue global QoS race cond…

b10a55c

…ition (celery#9770)" This reverts commit 6d8bfd1.

Remove test_quorum_queue_qos_cluster_simulation.py from CI

89dccb1

Nusnus force-pushed the cifix branch 4 times, most recently from a060103 to 6926cec Compare August 12, 2025 22:21

Nusnus force-pushed the cifix branch 3 times, most recently from 5165dd0 to 05b10e9 Compare August 13, 2025 17:35

Prevent the billiard QueueListener from deadlocking during worker shu…

cebcb2b

…tdown

Nusnus force-pushed the cifix branch from 9135da1 to cebcb2b Compare August 13, 2025 18:00

Nusnus added 4 commits August 13, 2025 21:40

Revert "TMP Dont wait for unit tests"

91590b4

This reverts commit 3da612d.

Run smoke if integration passed

8cb1720

Changed retry settings

76bf1ef

Disable test_groupresult_serialization

ade8cbd

Nusnus force-pushed the cifix branch from 02f20eb to ade8cbd Compare August 14, 2025 10:37

Nusnus added 3 commits August 25, 2025 20:12

timeout 5, attempts 10, instead of 30m x 2 attempts

d421d42

Split integration test jobs to be test per module

f9ff637

Disabled dep with unit tests (for faster testing)

e8594c3

Nusnus force-pushed the cifix branch from 9e6909a to e8594c3 Compare August 25, 2025 18:24

Nusnus added 6 commits August 25, 2025 22:46

Run all integration jobs together

8ff2e37

Split smoke test jobs

6c61e4e

Simplifed python-package.yml

e33a727

Cleanup

2d33206

max-parallel: 4

79c3810

fixed smoked tests ci

57e085e

Nusnus changed the title ~~Refactor integration tests CI~~ Refactor integration and smoke tests CI Aug 25, 2025

Nusnus added 7 commits August 26, 2025 01:04

Removed Python 3.10-3.12 from the integration and smoke tests CI

947ebce

Integration & Smoke run only if unit tests pass

d0f3848

Reduced more python versions for now (integration min/max, smoke-max)

4cd7048

Revert "Prevent the billiard QueueListener from deadlocking during wo…

282aa3f

…rker shutdown" This reverts commit cebcb2b.

Reapply "Add xfail test for RabbitMQ quorum queue global QoS race con…

9c83bec

…dition (celery#9770)" This reverts commit b10a55c.

Added back test_quorum_queue_qos_cluster_simulation to the integrat…

9ac7c36

…ion tests

max-parallel: 5

1541f15

Nusnus force-pushed the cifix branch from 2b0275d to 1541f15 Compare August 26, 2025 00:08

Revert "Disable test_groupresult_serialization"

c169268

This reverts commit ade8cbd.

Nusnus marked this pull request as ready for review August 26, 2025 01:00

Nusnus merged commit 777d92f into celery:main Aug 26, 2025
199 of 203 checks passed

Nusnus deleted the cifix branch August 26, 2025 01:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Refactor integration and smoke tests CI #9855

Refactor integration and smoke tests CI #9855

Uh oh!

Nusnus commented Aug 9, 2025

Uh oh!

codecov bot commented Aug 9, 2025 •

edited

Loading

Uh oh!

auvipy left a comment

Uh oh!

auvipy left a comment

Uh oh!

Nusnus commented Aug 11, 2025

Uh oh!

auvipy commented Aug 11, 2025

Uh oh!

auvipy commented Aug 13, 2025

Uh oh!

Nusnus commented Aug 13, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Refactor integration and smoke tests CI #9855

Refactor integration and smoke tests CI #9855

Uh oh!

Conversation

Nusnus commented Aug 9, 2025

Uh oh!

codecov bot commented Aug 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

auvipy left a comment

Choose a reason for hiding this comment

Uh oh!

auvipy left a comment

Choose a reason for hiding this comment

Uh oh!

Nusnus commented Aug 11, 2025

Uh oh!

auvipy commented Aug 11, 2025

Uh oh!

auvipy commented Aug 13, 2025

Uh oh!

Nusnus commented Aug 13, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov bot commented Aug 9, 2025 •

edited

Loading