Skip to content

Test test_quorum_inserts/test.py is flaky #67604

@vitlibar

Description

@vitlibar

Test test_quorum_inserts/test.py::test_insert_quorum_with_keeper_loss_connection has failed with the following exception kazoo.exceptions.ConnectionLoss (see):

=================================== FAILURES ===================================
________________ test_insert_quorum_with_keeper_loss_connection ________________
[gw4] linux -- Python 3.10.12 /usr/bin/python3

    def test_insert_quorum_with_keeper_loss_connection():
        zero.query(
            "DROP TABLE IF EXISTS test_insert_quorum_with_keeper_fail ON CLUSTER cluster"
        )
        create_query = (
            "CREATE TABLE test_insert_quorum_with_keeper_loss"
            "(a Int8, d Date) "
            "Engine = ReplicatedMergeTree('/clickhouse/tables/{table}', '{replica}') "
            "ORDER BY a "
        )
    
        zero.query(create_query)
        first.query(create_query)
    
        first.query("SYSTEM STOP FETCHES test_insert_quorum_with_keeper_loss")
    
        zero.query("SYSTEM ENABLE FAILPOINT replicated_merge_tree_commit_zk_fail_after_op")
        zero.query("SYSTEM ENABLE FAILPOINT replicated_merge_tree_insert_retry_pause")
    
        with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
            insert_future = executor.submit(
                lambda: zero.query(
                    "INSERT INTO test_insert_quorum_with_keeper_loss(a,d) VALUES(1, '2011-01-01')",
                    settings={"insert_quorum_timeout": 150000},
                )
            )
    
            pm = PartitionManager()
            pm.drop_instance_zk_connections(zero)
    
            retries = 0
            zk = cluster.get_kazoo_client("zoo1")
            while True:
                if (
>                   zk.exists(
                        "/clickhouse/tables/test_insert_quorum_with_keeper_loss/replicas/zero/is_active"
                    )
                    is None
                ):

test_quorum_inserts/test.py:402: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/local/lib/python3.10/dist-packages/kazoo/client.py:1123: in exists
    return self.exists_async(path, watch=watch).get()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <kazoo.handlers.threading.AsyncResult object at 0x7f2304e9d630>
block = True, timeout = None

    def get(self, block=True, timeout=None):
        """Return the stored value or raise the exception.
    
        If there is no value raises TimeoutError.
    
        """
        with self._condition:
            if self._exception is not _NONE:
                if self._exception is None:
                    return self.value
                raise self._exception
            elif block:
                self._condition.wait(timeout)
                if self._exception is not _NONE:
                    if self._exception is None:
                        return self.value
>                   raise self._exception
E                   kazoo.exceptions.ConnectionLoss

/usr/local/lib/python3.10/dist-packages/kazoo/handlers/utils.py:86: ConnectionLoss
----------------------------- Captured stdout call -----------------------------
replica is still active
replica is still active
replica is still active
replica is still active
replica is still active
replica is still active
replica is still active
replica is still active
replica is still active
replica is still active
replica is still active

Other failures: https://play.clickhouse.com/play?user=play#c2VsZWN0IAp0ZXN0X25hbWUsIGNoZWNrX3N0YXJ0X3RpbWUsIHB1bGxfcmVxdWVzdF9udW1iZXIsIHJlcG9ydF91cmwKZnJvbSBjaGVja3Mgd2hlcmUgJzIwMjQtMDEtMDEnIDw9IGNoZWNrX3N0YXJ0X3RpbWUgYW5kIHRlc3RfbmFtZSBsaWtlICcldGVzdF9pbnNlcnRfcXVvcnVtX3dpdGhfa2VlcGVyX2xvc3NfY29ubmVjdGlvbiUnIGFuZCB0ZXN0X3N0YXR1cyBpbiAoJ0ZBSUwnLCAnRkxBS1knKSBvcmRlciBieSBjaGVja19zdGFydF90aW1l

Metadata

Metadata

Assignees

Labels

flaky testflaky test found by CItestingSpecial issue with list of bugs found by CI

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions