-
Notifications
You must be signed in to change notification settings - Fork 8.3k
Fuzzing 02950_part_offset_as_primary_key.sql with parallel replicas #87653
Copy link
Copy link
Closed
Labels
fuzzProblem found by one of the fuzzersProblem found by one of the fuzzersred CIVERY flaky testVERY flaky test
Description
Describe the bug
Query fails with
<Fatal> : Logical error: 'Replica 1 decided to read in WithOrder mode, not in Default. This is a bug'.
How to reproduce
Run 02950_part_offset_as_primary_key with enabled parallel replicas
Error message and/or stacktrace
clickhouse-server.err.log:2025.09.25 19:42:53.381502 [ 83184 ] {28c04ffb-3819-4b98-a761-a55ae944ce14} <Fatal> : Logical error: 'Replica 1 decided to read in WithOrder mode, not in Default. This is a bug'.
clickhouse-server.err.log:2025.09.25 19:42:53.391921 [ 83184 ] {28c04ffb-3819-4b98-a761-a55ae944ce14} <Fatal> : Stack trace (when copying this message, always include the lines below):
clickhouse-server.err.log:2025.09.25 19:42:53.392118 [ 1622 ] {} <Fatal> BaseDaemon: ########## Short fault info ############
clickhouse-server.err.log:2025.09.25 19:42:53.392135 [ 1622 ] {} <Fatal> BaseDaemon: (version 25.10.1.692 (official build), build id: C27877DCE6D8A529E405E86B2B0172EA06EB8F42, git hash: 89a7b88905dddd20d7fe35897c0f26e4218b57e7, architecture: x86_64) (from thread 83184) Received signal 6
clickhouse-server.err.log:2025.09.25 19:42:53.392140 [ 1622 ] {} <Fatal> BaseDaemon: Signal description: Aborted
clickhouse-server.err.log:2025.09.25 19:42:53.392142 [ 1622 ] {} <Fatal> BaseDaemon:
clickhouse-server.err.log:2025.09.25 19:42:53.392161 [ 1622 ] {} <Fatal> BaseDaemon: Stack trace: 0x00007f01ff5529fd 0x00007f01ff4fe476 0x00007f01ff4e47f3 0x000055b5adeca153 0x000055b5adeca9ac 0x000055b5adecac6c 0x000055b5a728334e 0x000055b5a7282f40 0x000055b5b792bcab 0x000055b5b791e866 0x000055b5b304f3a1 0x000055b5b304de1b 0x000055b5b304ef3e 0x000055b5b81d0fe6 0x000055b5b7e50c76 0x000055b5b7e6bd59 0x000055b5b7e5f5d2 0x000055b5b7e60a85 0x000055b5ae0105d5 0x000055b5ae015d26 0x000055b5ae00e020 0x000055b5ae013e3b 0x00007f01ff550ac3 0x00007f01ff5e2850
clickhouse-server.err.log:2025.09.25 19:42:53.392167 [ 1622 ] {} <Fatal> BaseDaemon: ########################################
clickhouse-server.err.log:2025.09.25 19:42:53.392199 [ 1622 ] {} <Fatal> BaseDaemon: (version 25.10.1.692 (official build), build id: C27877DCE6D8A529E405E86B2B0172EA06EB8F42, git hash: 89a7b88905dddd20d7fe35897c0f26e4218b57e7) (from thread 83184) (query_id: 28c04ffb-3819-4b98-a761-a55ae944ce14) (query: -- nothing to read
clickhouse-server.err.log:2025.09.25 19:42:53.392213 [ 1622 ] {} <Fatal> BaseDaemon:
clickhouse-server.err.log:2025.09.25 19:42:53.392230 [ 1622 ] {} <Fatal> BaseDaemon: Stack trace: 0x00007f01ff5529fd 0x00007f01ff4fe476 0x00007f01ff4e47f3 0x000055b5adeca153 0x000055b5adeca9ac 0x000055b5adecac6c 0x000055b5a728334e 0x000055b5a7282f40 0x000055b5b792bcab 0x000055b5b791e866 0x000055b5b304f3a1 0x000055b5b304de1b 0x000055b5b304ef3e 0x000055b5b81d0fe6 0x000055b5b7e50c76 0x000055b5b7e6bd59 0x000055b5b7e5f5d2 0x000055b5b7e60a85 0x000055b5ae0105d5 0x000055b5ae015d26 0x000055b5ae00e020 0x000055b5ae013e3b 0x00007f01ff550ac3 0x00007f01ff5e2850
clickhouse-server.err.log:2025.09.25 19:42:53.392279 [ 1622 ] {} <Fatal> BaseDaemon: 3. ? @ 0x00000000000969fd
clickhouse-server.err.log:2025.09.25 19:42:53.392289 [ 1622 ] {} <Fatal> BaseDaemon: 4. ? @ 0x0000000000042476
clickhouse-server.err.log:2025.09.25 19:42:53.392303 [ 1622 ] {} <Fatal> BaseDaemon: 5. ? @ 0x00000000000287f3
clickhouse-server.err.log:2025.09.25 19:42:53.401233 [ 1622 ] {} <Fatal> BaseDaemon: 6. ./ci/tmp/build/./src/Common/Exception.cpp:51: DB::abortOnFailedAssertion(String const&, void* const*, unsigned long, unsigned long) @ 0x000000000fdce153
clickhouse-server.err.log:2025.09.25 19:42:53.410609 [ 1622 ] {} <Fatal> BaseDaemon: 7. ./ci/tmp/build/./src/Common/Exception.cpp:84: DB::handle_error_code(String const&, std::basic_string_view<char, std::char_traits<char>>, int, bool, std::vector<void*, std::allocator<void*>> const&) @ 0x000000000fdce9ac
clickhouse-server.err.log:2025.09.25 19:42:53.421274 [ 1622 ] {} <Fatal> BaseDaemon: 8. ./ci/tmp/build/./src/Common/Exception.cpp:135: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000fdcec6c
clickhouse-server.err.log:2025.09.25 19:42:53.465900 [ 1622 ] {} <Fatal> BaseDaemon: 9. DB::Exception::Exception(String&&, int, String, bool) @ 0x000000000918734e
clickhouse-server.err.log:2025.09.25 19:42:53.486682 [ 1622 ] {} <Fatal> BaseDaemon: 10. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000009186f40
clickhouse-server.err.log:2025.09.25 19:42:53.517681 [ 1622 ] {} <Fatal> BaseDaemon: 11. ./src/Common/Exception.h:141: DB::Exception::Exception<unsigned long&, std::basic_string_view<char, std::char_traits<char>>, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<unsigned long&>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, unsigned long&, std::basic_string_view<char, std::char_traits<char>>&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001982fcab
clickhouse-server.err.log:2025.09.25 19:42:53.547460 [ 1622 ] {} <Fatal> BaseDaemon: 12. ./ci/tmp/build/./src/Storages/MergeTree/ParallelReplicasReadingCoordinator.cpp:1153: DB::ParallelReplicasReadingCoordinator::handleRequest(DB::ParallelReadRequest) @ 0x0000000019822866
clickhouse-server.err.log:2025.09.25 19:42:53.580278 [ 1622 ] {} <Fatal> BaseDaemon: 13. ./ci/tmp/build/./src/QueryPipeline/RemoteQueryExecutor.cpp:764: DB::RemoteQueryExecutor::processMergeTreeReadTaskRequest(DB::ParallelReadRequest) @ 0x0000000014f533a1
clickhouse-server.err.log:2025.09.25 19:42:53.610153 [ 1622 ] {} <Fatal> BaseDaemon: 14. ./ci/tmp/build/./src/QueryPipeline/RemoteQueryExecutor.cpp:637: DB::RemoteQueryExecutor::processPacket(DB::Packet) @ 0x0000000014f51e1b
clickhouse-server.err.log:2025.09.25 19:42:53.641087 [ 1622 ] {} <Fatal> BaseDaemon: 15. ./ci/tmp/build/./src/QueryPipeline/RemoteQueryExecutor.cpp:585: DB::RemoteQueryExecutor::readAsync() @ 0x0000000014f52f3e
clickhouse-server.err.log:2025.09.25 19:42:53.651022 [ 1622 ] {} <Fatal> BaseDaemon: 16. ./ci/tmp/build/./src/Processors/Sources/RemoteSource.cpp:187: DB::RemoteSource::tryGenerate() @ 0x000000001a0d4fe6
clickhouse-server.err.log:2025.09.25 19:42:53.657267 [ 1622 ] {} <Fatal> BaseDaemon: 17. ./ci/tmp/build/./src/Processors/ISource.cpp:110: DB::ISource::work() @ 0x0000000019d54c76
clickhouse-server.err.log:2025.09.25 19:42:53.660547 [ 1622 ] {} <Fatal> BaseDaemon: 18.0. inlined from ./ci/tmp/build/./src/Processors/Executors/ExecutionThreadContext.cpp:53: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*)
clickhouse-server.err.log:2025.09.25 19:42:53.660586 [ 1622 ] {} <Fatal> BaseDaemon: 18. ./ci/tmp/build/./src/Processors/Executors/ExecutionThreadContext.cpp:102: DB::ExecutionThreadContext::executeTask() @ 0x0000000019d6fd59
clickhouse-server.err.log:2025.09.25 19:42:53.672085 [ 1622 ] {} <Fatal> BaseDaemon: 19. ./ci/tmp/build/./src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::executeStepImpl(unsigned long, DB::IAcquiredSlot*, std::atomic<bool>*) @ 0x0000000019d635d2
clickhouse-server.err.log:2025.09.25 19:42:53.685177 [ 1622 ] {} <Fatal> BaseDaemon: 20.0. inlined from ./ci/tmp/build/./src/Processors/Executors/PipelineExecutor.cpp:279: DB::PipelineExecutor::executeSingleThread(unsigned long, DB::IAcquiredSlot*)
clickhouse-server.err.log:2025.09.25 19:42:53.685203 [ 1622 ] {} <Fatal> BaseDaemon: 20.1. inlined from ./ci/tmp/build/./src/Processors/Executors/PipelineExecutor.cpp:565: operator()
clickhouse-server.err.log:2025.09.25 19:42:53.685226 [ 1622 ] {} <Fatal> BaseDaemon: 20.2. inlined from ./contrib/llvm-project/libcxx/include/__type_traits/invoke.h:149: decltype(std::declval<DB::PipelineExecutor::spawnThreads(std::shared_ptr<DB::IAcquiredSlot>)::$_0&>()()) std::__invoke[abi:ne190107]<DB::PipelineExecutor::spawnThreads(std::shared_ptr<DB::IAcquiredSlot>)::$_0&>(DB::PipelineExecutor::spawnThreads(std::shared_ptr<DB::IAcquiredSlot>)::$_0&)
clickhouse-server.err.log:2025.09.25 19:42:53.685240 [ 1622 ] {} <Fatal> BaseDaemon: 20.3. inlined from ./contrib/llvm-project/libcxx/include/__type_traits/invoke.h:224: void std::__invoke_void_return_wrapper<void, true>::__call[abi:ne190107]<DB::PipelineExecutor::spawnThreads(std::shared_ptr<DB::IAcquiredSlot>)::$_0&>(DB::PipelineExecutor::spawnThreads(std::shared_ptr<DB::IAcquiredSlot>)::$_0&)
clickhouse-server.err.log:2025.09.25 19:42:53.685248 [ 1622 ] {} <Fatal> BaseDaemon: 20.4. inlined from ./contrib/llvm-project/libcxx/include/__functional/function.h:210: ?
clickhouse-server.err.log:2025.09.25 19:42:53.685253 [ 1622 ] {} <Fatal> BaseDaemon: 20. ./contrib/llvm-project/libcxx/include/__functional/function.h:610: ? @ 0x0000000019d64a85
clickhouse-server.err.log:2025.09.25 19:42:53.694856 [ 1622 ] {} <Fatal> BaseDaemon: 21.0. inlined from ./contrib/llvm-project/libcxx/include/__functional/function.h:716: ?
clickhouse-server.err.log:2025.09.25 19:42:53.694885 [ 1622 ] {} <Fatal> BaseDaemon: 21.1. inlined from ./contrib/llvm-project/libcxx/include/__functional/function.h:989: ?
clickhouse-server.err.log:2025.09.25 19:42:53.694908 [ 1622 ] {} <Fatal> BaseDaemon: 21. ./ci/tmp/build/./src/Common/ThreadPool.cpp:802: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::worker() @ 0x000000000ff145d5
clickhouse-server.err.log:2025.09.25 19:42:53.701992 [ 1622 ] {} <Fatal> BaseDaemon: 22.0. inlined from ./contrib/llvm-project/libcxx/include/__type_traits/invoke.h:117: decltype(*std::declval<ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool*&>().*std::declval<void (ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::*&)()>()()) std::__invoke[abi:ne190107]<void (ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::*&)(), ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool*&, void>(void (ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::*&)(), ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool*&)
clickhouse-server.err.log:2025.09.25 19:42:53.702018 [ 1622 ] {} <Fatal> BaseDaemon: 22.1. inlined from ./contrib/llvm-project/libcxx/include/tuple:1354: decltype(auto) std::__apply_tuple_impl[abi:ne190107]<void (ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::*&)(), std::tuple<ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool*>&, 0ul>(void (ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::*&)(), std::tuple<ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool*>&, std::__tuple_indices<0ul>)
clickhouse-server.err.log:2025.09.25 19:42:53.702034 [ 1622 ] {} <Fatal> BaseDaemon: 22.2. inlined from ./contrib/llvm-project/libcxx/include/tuple:1358: decltype(auto) std::apply[abi:ne190107]<void (ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::*&)(), std::tuple<ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool*>&>(void (ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::*&)(), std::tuple<ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool*>&)
clickhouse-server.err.log:2025.09.25 19:42:53.702048 [ 1622 ] {} <Fatal> BaseDaemon: 22. ./src/Common/ThreadPool.h:312: ThreadFromGlobalPoolImpl<false, true>::ThreadFromGlobalPoolImpl<void (ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::*)(), ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool*>(void (ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::*&&)(), ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool*&&)::'lambda'()::operator()() @ 0x000000000ff19d26
clickhouse-server.err.log:2025.09.25 19:42:53.708893 [ 1622 ] {} <Fatal> BaseDaemon: 23.0. inlined from ./contrib/llvm-project/libcxx/include/__functional/function.h:716: ?
clickhouse-server.err.log:2025.09.25 19:42:53.708918 [ 1622 ] {} <Fatal> BaseDaemon: 23.1. inlined from ./contrib/llvm-project/libcxx/include/__functional/function.h:989: ?
clickhouse-server.err.log:2025.09.25 19:42:53.708927 [ 1622 ] {} <Fatal> BaseDaemon: 23. ./ci/tmp/build/./src/Common/ThreadPool.cpp:812: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x000000000ff12020
clickhouse-server.err.log:2025.09.25 19:42:53.722658 [ 1622 ] {} <Fatal> BaseDaemon: 24.0. inlined from ./contrib/llvm-project/libcxx/include/__type_traits/invoke.h:117: decltype(*std::declval<ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>().*std::declval<void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)()>()()) std::__invoke[abi:ne190107]<void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*, void>(void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*&&)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*&&)
clickhouse-server.err.log:2025.09.25 19:42:53.722691 [ 1622 ] {} <Fatal> BaseDaemon: 24.1. inlined from ./contrib/llvm-project/libcxx/include/__thread/thread.h:192: void std::__thread_execute[abi:ne190107]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*, 2ul>(std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>&, std::__tuple_indices<2ul>)
clickhouse-server.err.log:2025.09.25 19:42:53.722704 [ 1622 ] {} <Fatal> BaseDaemon: 24. ./contrib/llvm-project/libcxx/include/__thread/thread.h:201: void* std::__thread_proxy[abi:ne190107]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x000000000ff17e3b
clickhouse-server.err.log:2025.09.25 19:42:53.722727 [ 1622 ] {} <Fatal> BaseDaemon: 25. ? @ 0x0000000000094ac3
clickhouse-server.err.log:2025.09.25 19:42:53.722739 [ 1622 ] {} <Fatal> BaseDaemon: 26. ? @ 0x0000000000126850
clickhouse-server.err.log:2025.09.25 19:42:53.935549 [ 1622 ] {} <Fatal> BaseDaemon: Integrity check of the executable successfully passed (checksum: 79AB97D48C7413F03C4A3955953DADDC)
clickhouse-server.err.log:2025.09.25 19:42:55.277309 [ 1622 ] {} <Fatal> BaseDaemon: Report this error to https://github.com/ClickHouse/ClickHouse/issues
clickhouse-server.err.log:2025.09.25 19:42:55.277445 [ 1622 ] {} <Fatal> BaseDaemon: Changed settings: min_compress_block_size = 1749775, max_compress_block_size = 377813, max_block_size = 53635, min_external_table_block_size_bytes = 1, max_joined_block_size_rows = 60145, max_insert_threads = 1, max_threads = 3, max_read_buffer_size = 938090, connect_timeout_with_failover_ms = 2000, connect_timeout_with_failover_secure_ms = 3000, use_hedged_requests = false, idle_connection_timeout = 36000, s3_max_get_rps = 1000000, s3_max_get_burst = 2000000, s3_max_put_rps = 1000000, s3_max_put_burst = 2000000, s3_check_objects_after_upload = true, max_remote_read_network_bandwidth = 1000000000000, max_remote_write_network_bandwidth = 1000000000000, max_local_read_bandwidth = 1000000000000, max_local_write_bandwidth = 1000000000000, stream_like_engine_allow_direct_select = true, enable_multiple_prewhere_read_steps = false, replication_wait_for_inactive_replica_timeout = 30, compile_expressions = false, min_count_to_compile_expression = 0, min_count_to_compile_aggregate_expression = 0, group_by_two_level_threshold = 234971, group_by_two_level_threshold_bytes = 8627332, allow_nonconst_timezone_arguments = true, min_chunk_bytes_for_parallel_parsing = 16535326, output_format_parallel_formatting = false, merge_tree_coarse_index_granularity = 7, min_bytes_to_use_direct_io = 10737418240, min_bytes_to_use_mmap_io = 10737418240, log_queries = true, insert_quorum_timeout = 60000, merge_tree_read_split_ranges_into_intersecting_and_non_intersecting_injection_probability = 0., http_wait_end_of_query = true, http_response_buffer_size = 4646614, fsync_metadata = true, query_plan_join_swap_table = false, http_send_timeout = 60., http_receive_timeout = 60., use_index_for_in_with_subqueries_max_values = 1000000000, opentelemetry_start_trace_probability = 0.10000000149011612, enable_vertical_final = false, max_rows_to_read = 20000000, max_bytes_to_read = 1, max_bytes_to_read_leaf = 1000000000000, max_rows_to_group_by = 10000000000, max_bytes_before_externa
Note
02950_part_offset_as_primary_key is in parallel_replicas_blacklist.txt but it's not expected to fail with Logical error: 'Replica 1 decided to read in WithOrder mode, not in Default. This is a bug'.. This issue supposed to be fixed by #80652 but it looks there are still some cases
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
fuzzProblem found by one of the fuzzersProblem found by one of the fuzzersred CIVERY flaky testVERY flaky test