Skip to content

Fix flaky test_throttling#93344

Merged
azat merged 1 commit intomasterfrom
fix-flaky-test_throttling
Jan 3, 2026
Merged

Fix flaky test_throttling#93344
azat merged 1 commit intomasterfrom
fix-flaky-test_throttling

Conversation

@serxa
Copy link
Copy Markdown
Member

@serxa serxa commented Jan 2, 2026

Changelog category (leave one):

  • Not for changelog (changelog entry is not required)

Fixes rare flakiness of tests that check config changes on the fly w/o a server restart. Unfortunately, there is no reliable way to check for an unlimited case, so we only increase the arbitrary check from "<1" to "<3". This is because in the limited case, it should be ">3", and we need to distinguish between these cases.

Fixes: #93251
Fixes: #86098
Fixes: #86963
Fixes: #85563

@clickhouse-gh
Copy link
Copy Markdown
Contributor

clickhouse-gh bot commented Jan 2, 2026

Workflow [PR], commit [14eb365]

Summary:

@clickhouse-gh clickhouse-gh bot added the pr-not-for-changelog This PR should not be mentioned in the changelog label Jan 2, 2026
@azat azat self-assigned this Jan 3, 2026
@azat azat changed the title Fix flaky test_throttling::test_remote_write_throttling_reload Fix flaky test_throttling Jan 3, 2026
@azat azat added the 🍃 green ci 🌿 Fixing flaky tests in CI label Jan 3, 2026

_, took = elapsed(node.query, f"select * from data")
assert took < 1
assert took < 3
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks OK, though I think we can do better, we need to measure the query_duration_ms from system.query_log, since for sanitizers even starting the client binary can take sometime

For instance here https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=93275&sha=340f689ca711904c1c473f347ecd1b003bd0d849&name_0=PR&name_1=Integration%20tests%20%28amd_tsan%2C%206%2F6%29&name_1=Integration%20tests%20%28amd_tsan%2C%206%2F6%29

It reports 1.0x seconds, but I don't see it on the server side - https://pastila.nl/?00328ceb/1edad0ab0cccc0bb0facfe662450c4ce#i+/GGP//s0aRB9BPHMdxrQ==GCM

And the slowest part is MinIO

2026.01.03 17:45:15.763555 [ 15150 ] {2b7d44eb-96d7-493c-b7b7-ecf1a314f14d} <Test> AWSClient: Make S3 request to: https://minio1:9001/root/data/qyu/rnkvbdzoeehuysdsjvzgykfswmooe, aws sdk attempt: 1, clickhouse attempt: 1, kind: Write
2026.01.03 17:45:16.285502 [ 15150 ] {2b7d44eb-96d7-493c-b7b7-ecf1a314f14d} <Trace> WriteBufferFromS3: Single part upload has completed. Details: bucket root, key data/qyu/rnkvbdzoeehuysdsjvzgykfswmooe, size 8003050

I tried to hack MinIO to forbid fsync, but it did not work - #92645

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But let's merge as-is to make CI feel better

@azat azat added this pull request to the merge queue Jan 3, 2026
Merged via the queue into master with commit f140437 Jan 3, 2026
131 checks passed
@azat azat deleted the fix-flaky-test_throttling branch January 3, 2026 23:39
@robot-ch-test-poll3 robot-ch-test-poll3 added the pr-synced-to-cloud The PR is synced to the cloud repo label Jan 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

🍃 green ci 🌿 Fixing flaky tests in CI pr-not-for-changelog This PR should not be mentioned in the changelog pr-synced-to-cloud The PR is synced to the cloud repo

Projects

None yet

3 participants