Skip to content

Fix test_storage_rabbitmq timeouts#84234

Merged
pamarcos merged 5 commits intoClickHouse:masterfrom
pamarcos:improve-test_storage_rabbitmq
Jul 24, 2025
Merged

Fix test_storage_rabbitmq timeouts#84234
pamarcos merged 5 commits intoClickHouse:masterfrom
pamarcos:improve-test_storage_rabbitmq

Conversation

@pamarcos
Copy link
Copy Markdown
Member

@pamarcos pamarcos commented Jul 22, 2025 •
edited
Loading

Part of #71049
Closes #81221

Changelog category (leave one):

  • CI Fix or Improvement (changelog entry is not required)

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

...

Documentation entry for user-facing changes

  • Documentation is written (mandatory for new features)

pamarcos added 4 commits July 22, 2025 14:44
@pamarcos
Refactor test_failed_connection to use common code
61983e6 
Increase the deadline of test_failed_connection in case data is consumed.
  Sometimes the 240 seconds is not enough:
    - https://s3.amazonaws.com/clickhouse-test-reports/PRs/84008/2f687ac0378b5773333dfab111697271a77712b4//integration_tests_aarch64_distributed_plan_4_4/integration_run_test_storage_rabbitmq_test_failed_connection_py_0.log
    - https://s3.amazonaws.com/clickhouse-test-reports/PRs/84008/ba8e3a94ef29096dcc3ce1e69db6e1ad548936fc//integration_tests_release_1_4/integration_run_test_storage_rabbitmq_test_failed_connection_py_0.log
    - https://s3.amazonaws.com/clickhouse-test-reports/PRs/84008/2f687ac0378b5773333dfab111697271a77712b4//integration_tests_release_1_4/integration_run_test_storage_rabbitmq_test_failed_connection_py_0.log
  Even though some other times it takes less than 20 seconds to finish:
    - https://s3.amazonaws.com/clickhouse-test-reports/REFs/master/9d200fac5a5bda20729c02296404f429b4bb0ce9//integration_tests_release_3_4/integration_run_test_storage_rabbitmq_test_failed_connection_py_0.log
@pamarcos
Increase the deadline of test_rabbitmq_sharding_between_queues_publis…
e0d6c2a 
…h in case data is consumed.

https://s3.amazonaws.com/clickhouse-test-reports/PRs/84079/912469e0a9a87c6fc7d77f386b688f589e799b7f//integration_tests_tsan_1_6/integration_run_test_storage_rabbitmq_test_py_0.log
@pamarcos
Consolidate in a single function the check
045f2a4
@pamarcos
Reuse check_expected_result_polling as much as possible
b1a3040 
This will make the tests more resilient
@clickhouse-gh
Copy link
Copy Markdown
Contributor

clickhouse-gh bot commented Jul 22, 2025 •
edited
Loading

Workflow [PR], commit [0d56f80]

Summary: ❌

job_name test_name status info comment
Integration tests (release, 3/4) failure
Job Timeout Expired FAIL

@clickhouse-gh clickhouse-gh bot added the pr-ci label Jul 22, 2025
@nikitamikhaylov nikitamikhaylov assigned pamarcos and unassigned pamarcos Jul 22, 2025
@pamarcos pamarcos added the 🍃 green ci 🌿 Fixing flaky tests in CI label Jul 23, 2025
@pamarcos pamarcos changed the title Improve test_storage_rabbitmq to avoid timeouts Fix test_storage_rabbitmq timeouts Jul 23, 2025
@kssenii kssenii self-assigned this Jul 23, 2025
kssenii
kssenii reviewed Jul 23, 2025
View reviewed changes
tests/integration/test_storage_rabbitmq/test.py
Comment on lines +81 to +82
if result > prev_result:
deadline += 1
Copy link
Copy Markdown
Member

@kssenii kssenii Jul 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May be also have a hard deadline after which we can no longer increase?

Copy link
Copy Markdown
Member Author

@pamarcos pamarcos Jul 23, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't set any hard deadline on purpose because ideally, I'd like the deadline to keep increasing as long as there's new data consumed. pytest already has a 900s timeout, so it'll make sure it won't take infinite time

@pamarcos pamarcos enabled auto-merge July 23, 2025 11:56
@pamarcos pamarcos added pr-must-backport-synced The `*-must-backport` labels are synced into the cloud Sync PR pr-must-backport Pull request should be backported intentionally. Use this label with great care! labels Jul 23, 2025
@pamarcos pamarcos added this pull request to the merge queue Jul 24, 2025
Merged via the queue into ClickHouse:master with commit 4c0c2e8 Jul 24, 2025
121 of 123 checks passed
@pamarcos pamarcos deleted the improve-test_storage_rabbitmq branch July 24, 2025 08:56
@robot-ch-test-poll3 robot-ch-test-poll3 added the pr-synced-to-cloud The PR is synced to the cloud repo label Jul 24, 2025
This was referenced Jul 24, 2025
Closed
Merged
Merged
Merged
Merged
Merged
robot-ch-test-poll added a commit that referenced this pull request Jul 24, 2025
@robot-ch-test-poll
Merge pull request #84349 from ClickHouse/cherrypick/25.3/84234
7643fb3 
Cherry pick #84234 to 25.3: Fix test_storage_rabbitmq timeouts
robot-clickhouse added a commit that referenced this pull request Jul 24, 2025
@robot-clickhouse
Backport #84234 to 25.3: Fix test_storage_rabbitmq timeouts
904bfee
@robot-ch-test-poll robot-ch-test-poll mentioned this pull request Jul 24, 2025
Merged
robot-ch-test-poll added a commit that referenced this pull request Jul 24, 2025
@robot-ch-test-poll
Merge pull request #84350 from ClickHouse/cherrypick/25.4/84234
9c86870 
Cherry pick #84234 to 25.4: Fix test_storage_rabbitmq timeouts
robot-clickhouse added a commit that referenced this pull request Jul 24, 2025
@robot-clickhouse
Backport #84234 to 25.4: Fix test_storage_rabbitmq timeouts
09fa7f9
@robot-ch-test-poll robot-ch-test-poll mentioned this pull request Jul 24, 2025
Closed
robot-ch-test-poll added a commit that referenced this pull request Jul 24, 2025
@robot-ch-test-poll
Merge pull request #84351 from ClickHouse/cherrypick/25.5/84234
7303af9 
Cherry pick #84234 to 25.5: Fix test_storage_rabbitmq timeouts
robot-clickhouse added a commit that referenced this pull request Jul 24, 2025
@robot-clickhouse
Backport #84234 to 25.5: Fix test_storage_rabbitmq timeouts
b31ecb4
@robot-ch-test-poll robot-ch-test-poll mentioned this pull request Jul 24, 2025
Closed
robot-ch-test-poll added a commit that referenced this pull request Jul 24, 2025
@robot-ch-test-poll
Merge pull request #84352 from ClickHouse/cherrypick/25.6/84234
516f832 
Cherry pick #84234 to 25.6: Fix test_storage_rabbitmq timeouts
robot-clickhouse added a commit that referenced this pull request Jul 24, 2025
@robot-clickhouse
Backport #84234 to 25.6: Fix test_storage_rabbitmq timeouts
701b444
@robot-ch-test-poll robot-ch-test-poll mentioned this pull request Jul 24, 2025
Merged
robot-ch-test-poll added a commit that referenced this pull request Jul 24, 2025
@robot-ch-test-poll
Merge pull request #84353 from ClickHouse/cherrypick/25.7/84234
16d6c5f 
Cherry pick #84234 to 25.7: Fix test_storage_rabbitmq timeouts
robot-clickhouse added a commit that referenced this pull request Jul 24, 2025
@robot-clickhouse
Backport #84234 to 25.7: Fix test_storage_rabbitmq timeouts
c423e36
@robot-ch-test-poll robot-ch-test-poll mentioned this pull request Jul 24, 2025
Merged
@pamarcos pamarcos removed the pr-synced-to-cloud The PR is synced to the cloud repo label Jul 24, 2025
@robot-clickhouse-ci-1 robot-clickhouse-ci-1 added the pr-synced-to-cloud The PR is synced to the cloud repo label Jul 24, 2025
@robot-ch-test-poll2 robot-ch-test-poll2 added the pr-backports-created Backport PRs are successfully created, it won't be processed by CI script anymore label Jul 24, 2025
clickhouse-gh bot added a commit that referenced this pull request Jul 24, 2025
@clickhouse-gh
Merge pull request #84363 from ClickHouse/backport/25.7/84234
dcad4e1 
Backport #84234 to 25.7: Fix test_storage_rabbitmq timeouts
clickhouse-gh bot added a commit that referenced this pull request Jul 24, 2025
@clickhouse-gh
Merge pull request #84362 from ClickHouse/backport/25.6/84234
5d04bea 
Backport #84234 to 25.6: Fix test_storage_rabbitmq timeouts
@pamarcos pamarcos removed pr-synced-to-cloud The PR is synced to the cloud repo pr-must-backport-synced The `*-must-backport` labels are synced into the cloud Sync PR labels Jul 25, 2025
@robot-ch-test-poll4 robot-ch-test-poll4 added pr-backports-created-cloud deprecated label, NOOP pr-must-backport-synced The `*-must-backport` labels are synced into the cloud Sync PR labels Jul 25, 2025
@robot-clickhouse-ci-1 robot-clickhouse-ci-1 added the pr-synced-to-cloud The PR is synced to the cloud repo label Jul 25, 2025
pamarcos added a commit that referenced this pull request Jul 25, 2025
@pamarcos
Merge pull request #84359 from ClickHouse/backport/25.3/84234
cd38dc8 
Backport #84234 to 25.3: Fix test_storage_rabbitmq timeouts
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kssenii kssenii kssenii approved these changes

Assignees

@kssenii kssenii

Labels

🍃 green ci 🌿 Fixing flaky tests in CI pr-backports-created Backport PRs are successfully created, it won't be processed by CI script anymore pr-backports-created-cloud deprecated label, NOOP pr-ci pr-must-backport Pull request should be backported intentionally. Use this label with great care! pr-must-backport-synced The `*-must-backport` labels are synced into the cloud Sync PR pr-synced-to-cloud The PR is synced to the cloud repo

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

test_rabbitmq_sharding_between_queues_publish is flaky

8 participants

@pamarcos @kssenii @nikitamikhaylov @robot-ch-test-poll2 @robot-ch-test-poll4 @robot-ch-test-poll3 @rschu1ze @robot-clickhouse-ci-1