Skip to content

tests: fix test_dictionaries_all_layouts_separate_sources flakiness#84969

Merged
azat merged 1 commit intoClickHouse:masterfrom
azat:tests/test_dictionaries_all_layouts_separate_sources
Aug 3, 2025
Merged

tests: fix test_dictionaries_all_layouts_separate_sources flakiness#84969
azat merged 1 commit intoClickHouse:masterfrom
azat:tests/test_dictionaries_all_layouts_separate_sources

Conversation

@azat
Copy link
Copy Markdown
Member

@azat azat commented Aug 3, 2025

The problem is that the certificate file can be rewritten by another dictionary setup, look at 1:

First of all server contains fatal errors in logs, and it should be a starting point:

Traceback (most recent call last):
  File "/http_server.py", line 116, in <module>
    start_server(
  File "/http_server.py", line 90, in start_server
    httpd.socket = ssl.wrap_socket(
  File "/usr/lib/python3.10/ssl.py", line 1468, in wrap_socket
    context.load_cert_chain(certfile, keyfile)
ssl.SSLError: [SSL] PEM lib (_ssl.c:3900)

Next we can look at pytest logs:

2025-08-01 13:26:29.588000 [ 652 ] DEBUG : Command:[docker exec roottestdictionariesalllayoutsseparatesourceshttps-gw2-clickhouse_hs-1 bash -c mkdir -p $(dirname /fake_cert.pem) && echo ... | base64 --decode > /fake_cert.pem] (cluster.py:119, run_and_check)
2025-08-01 13:26:29.857000 [ 652 ] DEBUG : Command:[docker exec roottestdictionariesalllayoutsseparatesourceshttps-gw2-clickhouse_hs-1 bash -c mkdir -p $(dirname /fake_cert.pem) && echo ... | base64 --decode > /fake_cert.pem] (cluster.py:119, run_and_check)
2025-08-01 13:26:29.909000 [ 652 ] DEBUG : run container_id:roottestdictionariesalllayoutsseparatesourceshttps-gw2-clickhouse_hs-1 detach:True nothrow:False cmd: ['bash', '-c', 'python3 /http_server.py --data-path=/table_SourceHTTPS_hashed_.tsv --schema=https --host=clickhouse_hs --port=5556 --cert-path=/fake_cert.pem >> /var/log/clickhouse-server/http_server.py.log 2>&1'] (cluster.py:2170, exec_in_container)

It does multiple copying of fake_cert.pem, but, it is not synced with reading of this file in http-server.py

So, let's make helpers for copying from/to container atomic.

Fixes: #81246

Changelog category (leave one):

  • Not for changelog (changelog entry is not required)

@clickhouse-gh
Copy link
Copy Markdown
Contributor

clickhouse-gh bot commented Aug 3, 2025

Workflow [PR], commit [0a92698]

Summary:

@clickhouse-gh clickhouse-gh bot added the pr-not-for-changelog This PR should not be mentioned in the changelog label Aug 3, 2025
The problem is that the certificate file can be rewritten by another
dictionary setup, look at [1]:

First of all server contains fatal errors in logs, and it should be a
starting point:

    Traceback (most recent call last):
      File "/http_server.py", line 116, in <module>
        start_server(
      File "/http_server.py", line 90, in start_server
        httpd.socket = ssl.wrap_socket(
      File "/usr/lib/python3.10/ssl.py", line 1468, in wrap_socket
        context.load_cert_chain(certfile, keyfile)
    ssl.SSLError: [SSL] PEM lib (_ssl.c:3900)

Next we can look at pytest logs:

    2025-08-01 13:26:29.588000 [ 652 ] DEBUG : Command:[docker exec roottestdictionariesalllayoutsseparatesourceshttps-gw2-clickhouse_hs-1 bash -c mkdir -p $(dirname /fake_cert.pem) && echo ... | base64 --decode > /fake_cert.pem] (cluster.py:119, run_and_check)
    2025-08-01 13:26:29.857000 [ 652 ] DEBUG : Command:[docker exec roottestdictionariesalllayoutsseparatesourceshttps-gw2-clickhouse_hs-1 bash -c mkdir -p $(dirname /fake_cert.pem) && echo ... | base64 --decode > /fake_cert.pem] (cluster.py:119, run_and_check)
    2025-08-01 13:26:29.909000 [ 652 ] DEBUG : run container_id:roottestdictionariesalllayoutsseparatesourceshttps-gw2-clickhouse_hs-1 detach:True nothrow:False cmd: ['bash', '-c', 'python3 /http_server.py --data-path=/table_SourceHTTPS_hashed_.tsv --schema=https --host=clickhouse_hs --port=5556 --cert-path=/fake_cert.pem >> /var/log/clickhouse-server/http_server.py.log 2>&1'] (cluster.py:2170, exec_in_container)

It does multiple copying of fake_cert.pem, but, it is not synced with
reading of this file in http-server.py

So, let's make helpers for copying from/to container atomic.

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/json.html?REF=master&sha=8420ce5c332f5cad97c804d2be9f8a7ba7366588&name_0=MasterCI&name_1=Integration%20tests%20%28release%2C%206%2F6%29&name_1=Integration%20tests%20%28release%2C%206%2F6%29
@azat azat force-pushed the tests/test_dictionaries_all_layouts_separate_sources branch from 89d2c45 to 0a92698 Compare August 3, 2025 14:40
@azat azat enabled auto-merge August 3, 2025 17:01
@azat azat added this pull request to the merge queue Aug 3, 2025
Merged via the queue into ClickHouse:master with commit c9e8c43 Aug 3, 2025
124 checks passed
@azat azat deleted the tests/test_dictionaries_all_layouts_separate_sources branch August 3, 2025 18:17
@robot-clickhouse robot-clickhouse added the pr-synced-to-cloud The PR is synced to the cloud repo label Aug 3, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

🍃 green ci 🌿 Fixing flaky tests in CI pr-not-for-changelog This PR should not be mentioned in the changelog pr-synced-to-cloud The PR is synced to the cloud repo

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky test_dictionaries_all_layouts_separate_sources/test_https.py

3 participants