Skip to content

Flaky test_checking_s3_blobs_paranoid/test.py::test_when_s3_broken_pipe_at_upload_is_retried #65530

@pamarcos

Description

@pamarcos

test_checking_s3_blobs_paranoid/test.py::test_when_s3_broken_pipe_at_upload_is_retried seems to be flaky.

=================================== FAILURES ===================================
________________ test_when_s3_broken_pipe_at_upload_is_retried _________________
[gw1] linux -- Python 3.10.12 /usr/bin/python3

cluster = <helpers.cluster.ClickHouseCluster object at 0xffa1e9b935e0>
broken_s3 = <helpers.s3_mocks.broken_s3.MockControl object at 0xffa1e939c8e0>

    def test_when_s3_broken_pipe_at_upload_is_retried(cluster, broken_s3):
        node = cluster.instances["node"]
    
        broken_s3.setup_fake_multpartuploads()
        broken_s3.setup_at_part_upload(
            count=3,
            after=2,
            action="broken_pipe",
        )
    
        insert_query_id = f"TEST_WHEN_S3_BROKEN_PIPE_AT_UPLOAD"
        node.query(
            f"""
            INSERT INTO
                TABLE FUNCTION s3(
                    'http://resolver:8083/root/data/test_when_s3_broken_pipe_at_upload_is_retried',
                    'minio', 'minio123',
                    'CSV', auto, 'none'
                )
            SELECT
                *
            FROM system.numbers
            LIMIT 1000000
            SETTINGS
                s3_max_single_part_upload_size=100,
                s3_min_upload_part_size=100000,
                s3_check_objects_after_upload=0
            """,
            query_id=insert_query_id,
        )
    
        create_multipart, upload_parts, s3_errors = get_multipart_counters(
            node, insert_query_id, log_type="QueryFinish"
        )
    
        assert create_multipart == 1
        assert upload_parts == 69
>       assert s3_errors == 3
E       assert 4 == 3

test_checking_s3_blobs_paranoid/test.py:315: AssertionError

I was able to reproduce it with some runs always with the same assert 4 == 3 error:

tests/integration/runner --binary $PWD/build/programs/clickhouse  --odbc-bridge-binary $PWD/build/programs/clickhouse-odbc-bridge --base-configs-dir $PWD/programs/server/ --command
pip install pytest-flakefinder
pytest --flake-finder --flake-runs=9999 test_checking_s3_blobs_paranoid/test.py::test_when_s3_broken_pipe_at_upload_is_retried

I left it running 600 times and it failed 5 times.

Metadata

Metadata

Assignees

No one assigned

    Labels

    flaky testflaky test found by CItestingSpecial issue with list of bugs found by CI

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions