-
Notifications
You must be signed in to change notification settings - Fork 8.3k
Test test_quorum_inserts/test.py is flaky #67604
Copy link
Copy link
Closed
Labels
flaky testflaky test found by CIflaky test found by CItestingSpecial issue with list of bugs found by CISpecial issue with list of bugs found by CI
Description
Test test_quorum_inserts/test.py::test_insert_quorum_with_keeper_loss_connection has failed with the following exception kazoo.exceptions.ConnectionLoss (see):
=================================== FAILURES ===================================
________________ test_insert_quorum_with_keeper_loss_connection ________________
[gw4] linux -- Python 3.10.12 /usr/bin/python3
def test_insert_quorum_with_keeper_loss_connection():
zero.query(
"DROP TABLE IF EXISTS test_insert_quorum_with_keeper_fail ON CLUSTER cluster"
)
create_query = (
"CREATE TABLE test_insert_quorum_with_keeper_loss"
"(a Int8, d Date) "
"Engine = ReplicatedMergeTree('/clickhouse/tables/{table}', '{replica}') "
"ORDER BY a "
)
zero.query(create_query)
first.query(create_query)
first.query("SYSTEM STOP FETCHES test_insert_quorum_with_keeper_loss")
zero.query("SYSTEM ENABLE FAILPOINT replicated_merge_tree_commit_zk_fail_after_op")
zero.query("SYSTEM ENABLE FAILPOINT replicated_merge_tree_insert_retry_pause")
with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
insert_future = executor.submit(
lambda: zero.query(
"INSERT INTO test_insert_quorum_with_keeper_loss(a,d) VALUES(1, '2011-01-01')",
settings={"insert_quorum_timeout": 150000},
)
)
pm = PartitionManager()
pm.drop_instance_zk_connections(zero)
retries = 0
zk = cluster.get_kazoo_client("zoo1")
while True:
if (
> zk.exists(
"/clickhouse/tables/test_insert_quorum_with_keeper_loss/replicas/zero/is_active"
)
is None
):
test_quorum_inserts/test.py:402:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/lib/python3.10/dist-packages/kazoo/client.py:1123: in exists
return self.exists_async(path, watch=watch).get()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <kazoo.handlers.threading.AsyncResult object at 0x7f2304e9d630>
block = True, timeout = None
def get(self, block=True, timeout=None):
"""Return the stored value or raise the exception.
If there is no value raises TimeoutError.
"""
with self._condition:
if self._exception is not _NONE:
if self._exception is None:
return self.value
raise self._exception
elif block:
self._condition.wait(timeout)
if self._exception is not _NONE:
if self._exception is None:
return self.value
> raise self._exception
E kazoo.exceptions.ConnectionLoss
/usr/local/lib/python3.10/dist-packages/kazoo/handlers/utils.py:86: ConnectionLoss
----------------------------- Captured stdout call -----------------------------
replica is still active
replica is still active
replica is still active
replica is still active
replica is still active
replica is still active
replica is still active
replica is still active
replica is still active
replica is still active
replica is still active
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
flaky testflaky test found by CIflaky test found by CItestingSpecial issue with list of bugs found by CISpecial issue with list of bugs found by CI