Skip to content

test_restore_db_replica/test.py::test_query_after_restore_db_replica is flaky #85664

@rschu1ze

Description

@rschu1ze

Example failure (from here):

>           assert node_1.query_with_retry(
                f"SELECT table FROM system.tables WHERE database='{exclusive_database_name}' ORDER BY table",
                retry_count=10,
                sleep_time=1,
                check_callback=lambda tables: tables == exists_table_name,
            )
E           assert ''
E            +  where '' = <bound method ClickHouseInstance.query_with_retry of <helpers.cluster.ClickHouseInstance object at 0x7f2683624070>>("SELECT table FROM system.tables WHERE database='repl_db_test_query_after_restore_db_replica_alter_table_with_exists_table_with_restart__njwfjz' ORDER BY table", retry_count=10, sleep_time=1, check_callback=<function test_query_after_restore_db_replica.<locals>.<lambda> at 0x7f268188dc60>)
E            +    where <bound method ClickHouseInstance.query_with_retry of <helpers.cluster.ClickHouseInstance object at 0x7f2683624070>> = <helpers.cluster.ClickHouseInstance object at 0x7f2683624070>.query_with_retry

test_restore_db_replica/test.py:280: AssertionError
------------------------------ Captured log call -------------------------------
2025-08-14 12:05:38.474000 [ 698 ] DEBUG : Executing query 
                CREATE DATABASE repl_db_test_query_after_restore_db_replica_alter_table_with_exists_table_with_restart__njwfjz
                ENGINE=Replicated("/clickhouse/repl_db_test_query_after_restore_db_replica_alter_table_with_exists_table_with_restart__njwfjz", '{shard}', '{replica}')
             on node1 (cluster.py:4100, query)
2025-08-14 12:05:38.638000 [ 698 ] DEBUG : Executing query 
                CREATE DATABASE repl_db_test_query_after_restore_db_replica_alter_table_with_exists_table_with_restart__njwfjz
                ENGINE=Replicated("/clickhouse/repl_db_test_query_after_restore_db_replica_alter_table_with_exists_table_with_restart__njwfjz", '{shard}', '{replica}')
             on node2 (cluster.py:4100, query)
2025-08-14 12:05:38.803000 [ 698 ] DEBUG : Executing query 
            CREATE TABLE repl_db_test_query_after_restore_db_replica_alter_table_with_exists_table_with_restart__njwfjz.exists_table_zoywis (n UInt32)
            ENGINE = ReplicatedMergeTree
            ORDER BY n PARTITION BY n % 10
            SETTINGS distributed_ddl_task_timeout=5;
         on node1 (cluster.py:4100, query)
2025-08-14 12:05:39.068000 [ 698 ] DEBUG : Executing query 
            INSERT INTO repl_db_test_query_after_restore_db_replica_alter_table_with_exists_table_with_restart__njwfjz.exists_table_zoywis SELECT number FROM numbers(1000)
         on node1 (cluster.py:4100, query)
2025-08-14 12:05:39.333000 [ 698 ] DEBUG : get_instance_ip instance_name=zoo1 (cluster.py:2241, get_instance_ip)
2025-08-14 12:05:39.335000 [ 698 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestrestoredbreplica-gw2-zoo1-1/json HTTP/11" 200 None (connectionpool.py:546, _make_request)
2025-08-14 12:05:39.335000 [ 698 ] DEBUG : get_kazoo_client: zoo1, ip:172.16.1.4, port:2181, use_ssl:False (cluster.py:3727, get_kazoo_client)
2025-08-14 12:05:39.336000 [ 698 ] INFO : Connecting to 172.16.1.4(172.16.1.4):2181, use_ssl: False (connection.py:650, _connect)
2025-08-14 12:05:39.337000 [ 698 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit)
2025-08-14 12:05:39.339000 [ 698 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback)
2025-08-14 12:05:39.339000 [ 698 ] DEBUG : Sending request(xid=1): GetChildren(path='/clickhouse/repl_db_test_query_after_restore_db_replica_alter_table_with_exists_table_with_restart__njwfjz', watcher=None) (connection.py:312, _submit)

Test case history

Metadata

Metadata

Assignees

No one assigned

    Labels

    flaky testflaky test found by CI

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions