Skip to content

Flaky test ddl_worker_with_loopback_hosts #88328

@korowa

Description

@korowa

CIDB: link

ON CLUSTER table creation fails with either Attempt to read after eof:

Trace
E   helpers.client.QueryRuntimeException: Client failed! Return code: 32, stderr: Error on processing query: Code: 32. DB::Exception: Attempt to read after eof: while receiving packet from 172.16.5.5:9000, local address: 172.16.5.1:56828. (ATTEMPT_TO_READ_AFTER_EOF), Stack trace (when copying this message, always include the lines below):
E   
E   0. ./contrib/llvm-project/libcxx/include/__exception/exception.h:113: Poco::Exception::Exception(String const&, int) @ 0x0000000029f07de0
E   1. ./ci/tmp/build/./src/Common/Exception.cpp:128: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001376cdb4
E   2. DB::Exception::Exception(String&&, int, String, bool) @ 0x00000000093299ee
E   3. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x00000000093293f0
E   4. DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000933ae3a
E   5. ./ci/tmp/build/./src/IO/VarInt.cpp:13: DB::throwReadAfterEOF() @ 0x00000000138c49ef
E   6. ./src/IO/VarInt.h:83: DB::Connection::receivePacket() @ 0x00000000224aadca
E   7. ./ci/tmp/build/./src/Client/ClientBase.cpp:1403: DB::ClientBase::receiveAndProcessPacket(std::shared_ptr<DB::IAST>, bool) @ 0x000000002244bcb2
E   8. ./ci/tmp/build/./src/Client/ClientBase.cpp:1378: DB::ClientBase::receiveResult(std::shared_ptr<DB::IAST>, int, bool) @ 0x000000002244b295
E   9. ./ci/tmp/build/./src/Client/ClientBase.cpp:1287: DB::ClientBase::processOrdinaryQuery(String, std::shared_ptr<DB::IAST>) @ 0x0000000022449a2b
E   10. ./ci/tmp/build/./src/Client/ClientBase.cpp:2305: DB::ClientBase::processParsedSingleQuery(std::basic_string_view<char, std::char_traits<char>>, std::shared_ptr<DB::IAST>, bool&, unsigned long) @ 0x000000002244733a
E   11. ./ci/tmp/build/./src/Client/ClientBase.cpp:2680: DB::ClientBase::executeMultiQuery(String const&) @ 0x00000000224569ed
E   12. ./ci/tmp/build/./src/Client/ClientBase.cpp:2921: DB::ClientBase::processQueryText(String const&) @ 0x00000000224582f5
E   13. ./ci/tmp/build/./src/Client/ClientBase.cpp:3734: DB::ClientBase::runNonInteractive() @ 0x00000000224651f5
E   14. ./ci/tmp/build/./programs/client/Client.cpp:394: DB::Client::main(std::vector<String, std::allocator<String>> const&) @ 0x0000000013a97bce
E   15. ./ci/tmp/build/./programs/client/Client.cpp:0: non-virtual thunk to DB::Client::main(std::vector<String, std::allocator<String>> const&) @ 0x0000000013a98d61
E   16. ./ci/tmp/build/./base/poco/Util/src/Application.cpp:315: Poco::Util::Application::run() @ 0x000000002a05c4ff
E   17. ./ci/tmp/build/./programs/client/Client.cpp:1226: mainEntryClickHouseClient(int, char**) @ 0x0000000013aa810c
E   18. ./ci/tmp/build/./programs/main.cpp:380: main @ 0x000000000931bb65
E   19. ? @ 0x0000000000029d90
E   20. ? @ 0x0000000000029e40
E   21. _start @ 0x000000000926e02e
E    (version 25.10.1.2015 (official build))
E   (query: CREATE TABLE t1 ON CLUSTER 'test_cluster' (x INT) ENGINE=MergeTree() ORDER BY x)

or with Distributed DDL task timeout (not on master branch though)

Trace
E   helpers.client.QueryRuntimeException: Client failed! Return code: 159, stderr: Received exception from server (version 25.10.1):
E   Code: 159. DB::Exception: Received from 172.16.5.6:9000. DB::Exception: Distributed DDL task /clickhouse/task_queue/ddl/query-0000000000 is not finished on 1 of 2 hosts (0 of them are currently executing the task, 0 are inactive). They are going to execute the query in background. Was waiting for 10.575958071 seconds, which is longer than distributed_ddl_task_timeout. Stack trace:
E   
E   0. ./contrib/llvm-project/libcxx/include/__exception/exception.h:113: Poco::Exception::Exception(String const&, int) @ 0x0000000029f4a920
E   1. ./ci/tmp/build/./src/Common/Exception.cpp:129: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000137981f4
E   2. DB::Exception::Exception(String&&, int, String, bool) @ 0x00000000093539ee
E   3. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x00000000093533f0
E   4. ./src/Common/Exception.h:141: DB::Exception::Exception<String&, unsigned long&, unsigned long, unsigned long&, unsigned long, double, char const*>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type, std::type_identity<unsigned long>::type, std::type_identity<double>::type, std::type_identity<char const*>::type>, String&, unsigned long&, unsigned long&&, unsigned long&, unsigned long&&, double&&, char const*&&) @ 0x000000001c8e7aa5
E   5. ./ci/tmp/build/./src/Interpreters/DDLOnClusterQueryStatusSource.cpp:85: DB::DDLOnClusterQueryStatusSource::handleTimeoutExceeded() @ 0x000000001cc77b4b
E   6. ./ci/tmp/build/./src/Interpreters/DistributedQueryStatusSource.cpp:206: DB::DistributedQueryStatusSource::generate() @ 0x000000001c5fa79b
E   7. ./ci/tmp/build/./src/Processors/ISource.cpp:144: DB::ISource::tryGenerate() @ 0x0000000022813638
E   8. ./ci/tmp/build/./src/Processors/ISource.cpp:110: DB::ISource::work() @ 0x00000000228131eb
E   9. ./ci/tmp/build/./src/Processors/Executors/ExecutionThreadContext.cpp:53: DB::ExecutionThreadContext::executeTask() @ 0x000000002283c192
E   10. ./ci/tmp/build/./src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::executeStepImpl(unsigned long, DB::IAcquiredSlot*, std::atomic<bool>*) @ 0x0000000022828b98
E   11. ./ci/tmp/build/./src/Processors/Executors/PipelineExecutor.cpp:279: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x0000000022827e18
E   12. ./ci/tmp/build/./src/Processors/Executors/PipelineExecutor.cpp:136: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x00000000228278ac
E   13. ./ci/tmp/build/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:76: void std::__function::__policy_invoker<void ()>::__call_impl[abi:ne190107]<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true, true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000022847e11
E   14. ./contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000013947423
E   15. ./contrib/llvm-project/libcxx/include/__type_traits/invoke.h:117: void* std::__thread_proxy[abi:ne190107]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x00000000139504dc
E   16. __tsan_thread_start_func @ 0x00000000092c1418
E   17. ? @ 0x0000000000094ac3
E   18. ? @ 0x0000000000126850
E   . (TIMEOUT_EXCEEDED)
E   (query: CREATE TABLE t1 ON CLUSTER 'test_cluster' (x INT) ENGINE=MergeTree() ORDER BY x)

Metadata

Metadata

Assignees

Labels

flaky testflaky test found by CI

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions