Skip to content

Fuzzing 02950_part_offset_as_primary_key.sql with parallel replicas #87653

@devcrafter

Description

@devcrafter

Describe the bug

Query fails with

<Fatal> : Logical error: 'Replica 1 decided to read in WithOrder mode, not in Default. This is a bug'.

https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=87644&sha=51ec319faacede1a0f7864baf9085d0a6dbd8867&name_0=PR

How to reproduce

Run 02950_part_offset_as_primary_key with enabled parallel replicas

Error message and/or stacktrace

clickhouse-server.err.log:2025.09.25 19:42:53.381502 [ 83184 ] {28c04ffb-3819-4b98-a761-a55ae944ce14} <Fatal> : Logical error: 'Replica 1 decided to read in WithOrder mode, not in Default. This is a bug'.
clickhouse-server.err.log:2025.09.25 19:42:53.391921 [ 83184 ] {28c04ffb-3819-4b98-a761-a55ae944ce14} <Fatal> : Stack trace (when copying this message, always include the lines below):
clickhouse-server.err.log:2025.09.25 19:42:53.392118 [ 1622 ] {} <Fatal> BaseDaemon: ########## Short fault info ############
clickhouse-server.err.log:2025.09.25 19:42:53.392135 [ 1622 ] {} <Fatal> BaseDaemon: (version 25.10.1.692 (official build), build id: C27877DCE6D8A529E405E86B2B0172EA06EB8F42, git hash: 89a7b88905dddd20d7fe35897c0f26e4218b57e7, architecture: x86_64) (from thread 83184) Received signal 6
clickhouse-server.err.log:2025.09.25 19:42:53.392140 [ 1622 ] {} <Fatal> BaseDaemon: Signal description: Aborted
clickhouse-server.err.log:2025.09.25 19:42:53.392142 [ 1622 ] {} <Fatal> BaseDaemon: 
clickhouse-server.err.log:2025.09.25 19:42:53.392161 [ 1622 ] {} <Fatal> BaseDaemon: Stack trace: 0x00007f01ff5529fd 0x00007f01ff4fe476 0x00007f01ff4e47f3 0x000055b5adeca153 0x000055b5adeca9ac 0x000055b5adecac6c 0x000055b5a728334e 0x000055b5a7282f40 0x000055b5b792bcab 0x000055b5b791e866 0x000055b5b304f3a1 0x000055b5b304de1b 0x000055b5b304ef3e 0x000055b5b81d0fe6 0x000055b5b7e50c76 0x000055b5b7e6bd59 0x000055b5b7e5f5d2 0x000055b5b7e60a85 0x000055b5ae0105d5 0x000055b5ae015d26 0x000055b5ae00e020 0x000055b5ae013e3b 0x00007f01ff550ac3 0x00007f01ff5e2850
clickhouse-server.err.log:2025.09.25 19:42:53.392167 [ 1622 ] {} <Fatal> BaseDaemon: ########################################
clickhouse-server.err.log:2025.09.25 19:42:53.392199 [ 1622 ] {} <Fatal> BaseDaemon: (version 25.10.1.692 (official build), build id: C27877DCE6D8A529E405E86B2B0172EA06EB8F42, git hash: 89a7b88905dddd20d7fe35897c0f26e4218b57e7) (from thread 83184) (query_id: 28c04ffb-3819-4b98-a761-a55ae944ce14) (query: -- nothing to read
clickhouse-server.err.log:2025.09.25 19:42:53.392213 [ 1622 ] {} <Fatal> BaseDaemon: 
clickhouse-server.err.log:2025.09.25 19:42:53.392230 [ 1622 ] {} <Fatal> BaseDaemon: Stack trace: 0x00007f01ff5529fd 0x00007f01ff4fe476 0x00007f01ff4e47f3 0x000055b5adeca153 0x000055b5adeca9ac 0x000055b5adecac6c 0x000055b5a728334e 0x000055b5a7282f40 0x000055b5b792bcab 0x000055b5b791e866 0x000055b5b304f3a1 0x000055b5b304de1b 0x000055b5b304ef3e 0x000055b5b81d0fe6 0x000055b5b7e50c76 0x000055b5b7e6bd59 0x000055b5b7e5f5d2 0x000055b5b7e60a85 0x000055b5ae0105d5 0x000055b5ae015d26 0x000055b5ae00e020 0x000055b5ae013e3b 0x00007f01ff550ac3 0x00007f01ff5e2850
clickhouse-server.err.log:2025.09.25 19:42:53.392279 [ 1622 ] {} <Fatal> BaseDaemon: 3. ? @ 0x00000000000969fd
clickhouse-server.err.log:2025.09.25 19:42:53.392289 [ 1622 ] {} <Fatal> BaseDaemon: 4. ? @ 0x0000000000042476
clickhouse-server.err.log:2025.09.25 19:42:53.392303 [ 1622 ] {} <Fatal> BaseDaemon: 5. ? @ 0x00000000000287f3
clickhouse-server.err.log:2025.09.25 19:42:53.401233 [ 1622 ] {} <Fatal> BaseDaemon: 6. ./ci/tmp/build/./src/Common/Exception.cpp:51: DB::abortOnFailedAssertion(String const&, void* const*, unsigned long, unsigned long) @ 0x000000000fdce153
clickhouse-server.err.log:2025.09.25 19:42:53.410609 [ 1622 ] {} <Fatal> BaseDaemon: 7. ./ci/tmp/build/./src/Common/Exception.cpp:84: DB::handle_error_code(String const&, std::basic_string_view<char, std::char_traits<char>>, int, bool, std::vector<void*, std::allocator<void*>> const&) @ 0x000000000fdce9ac
clickhouse-server.err.log:2025.09.25 19:42:53.421274 [ 1622 ] {} <Fatal> BaseDaemon: 8. ./ci/tmp/build/./src/Common/Exception.cpp:135: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000fdcec6c
clickhouse-server.err.log:2025.09.25 19:42:53.465900 [ 1622 ] {} <Fatal> BaseDaemon: 9. DB::Exception::Exception(String&&, int, String, bool) @ 0x000000000918734e
clickhouse-server.err.log:2025.09.25 19:42:53.486682 [ 1622 ] {} <Fatal> BaseDaemon: 10. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000009186f40
clickhouse-server.err.log:2025.09.25 19:42:53.517681 [ 1622 ] {} <Fatal> BaseDaemon: 11. ./src/Common/Exception.h:141: DB::Exception::Exception<unsigned long&, std::basic_string_view<char, std::char_traits<char>>, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<unsigned long&>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, unsigned long&, std::basic_string_view<char, std::char_traits<char>>&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001982fcab
clickhouse-server.err.log:2025.09.25 19:42:53.547460 [ 1622 ] {} <Fatal> BaseDaemon: 12. ./ci/tmp/build/./src/Storages/MergeTree/ParallelReplicasReadingCoordinator.cpp:1153: DB::ParallelReplicasReadingCoordinator::handleRequest(DB::ParallelReadRequest) @ 0x0000000019822866
clickhouse-server.err.log:2025.09.25 19:42:53.580278 [ 1622 ] {} <Fatal> BaseDaemon: 13. ./ci/tmp/build/./src/QueryPipeline/RemoteQueryExecutor.cpp:764: DB::RemoteQueryExecutor::processMergeTreeReadTaskRequest(DB::ParallelReadRequest) @ 0x0000000014f533a1
clickhouse-server.err.log:2025.09.25 19:42:53.610153 [ 1622 ] {} <Fatal> BaseDaemon: 14. ./ci/tmp/build/./src/QueryPipeline/RemoteQueryExecutor.cpp:637: DB::RemoteQueryExecutor::processPacket(DB::Packet) @ 0x0000000014f51e1b
clickhouse-server.err.log:2025.09.25 19:42:53.641087 [ 1622 ] {} <Fatal> BaseDaemon: 15. ./ci/tmp/build/./src/QueryPipeline/RemoteQueryExecutor.cpp:585: DB::RemoteQueryExecutor::readAsync() @ 0x0000000014f52f3e
clickhouse-server.err.log:2025.09.25 19:42:53.651022 [ 1622 ] {} <Fatal> BaseDaemon: 16. ./ci/tmp/build/./src/Processors/Sources/RemoteSource.cpp:187: DB::RemoteSource::tryGenerate() @ 0x000000001a0d4fe6
clickhouse-server.err.log:2025.09.25 19:42:53.657267 [ 1622 ] {} <Fatal> BaseDaemon: 17. ./ci/tmp/build/./src/Processors/ISource.cpp:110: DB::ISource::work() @ 0x0000000019d54c76
clickhouse-server.err.log:2025.09.25 19:42:53.660547 [ 1622 ] {} <Fatal> BaseDaemon: 18.0. inlined from ./ci/tmp/build/./src/Processors/Executors/ExecutionThreadContext.cpp:53: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*)
clickhouse-server.err.log:2025.09.25 19:42:53.660586 [ 1622 ] {} <Fatal> BaseDaemon: 18. ./ci/tmp/build/./src/Processors/Executors/ExecutionThreadContext.cpp:102: DB::ExecutionThreadContext::executeTask() @ 0x0000000019d6fd59
clickhouse-server.err.log:2025.09.25 19:42:53.672085 [ 1622 ] {} <Fatal> BaseDaemon: 19. ./ci/tmp/build/./src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::executeStepImpl(unsigned long, DB::IAcquiredSlot*, std::atomic<bool>*) @ 0x0000000019d635d2
clickhouse-server.err.log:2025.09.25 19:42:53.685177 [ 1622 ] {} <Fatal> BaseDaemon: 20.0. inlined from ./ci/tmp/build/./src/Processors/Executors/PipelineExecutor.cpp:279: DB::PipelineExecutor::executeSingleThread(unsigned long, DB::IAcquiredSlot*)
clickhouse-server.err.log:2025.09.25 19:42:53.685203 [ 1622 ] {} <Fatal> BaseDaemon: 20.1. inlined from ./ci/tmp/build/./src/Processors/Executors/PipelineExecutor.cpp:565: operator()
clickhouse-server.err.log:2025.09.25 19:42:53.685226 [ 1622 ] {} <Fatal> BaseDaemon: 20.2. inlined from ./contrib/llvm-project/libcxx/include/__type_traits/invoke.h:149: decltype(std::declval<DB::PipelineExecutor::spawnThreads(std::shared_ptr<DB::IAcquiredSlot>)::$_0&>()()) std::__invoke[abi:ne190107]<DB::PipelineExecutor::spawnThreads(std::shared_ptr<DB::IAcquiredSlot>)::$_0&>(DB::PipelineExecutor::spawnThreads(std::shared_ptr<DB::IAcquiredSlot>)::$_0&)
clickhouse-server.err.log:2025.09.25 19:42:53.685240 [ 1622 ] {} <Fatal> BaseDaemon: 20.3. inlined from ./contrib/llvm-project/libcxx/include/__type_traits/invoke.h:224: void std::__invoke_void_return_wrapper<void, true>::__call[abi:ne190107]<DB::PipelineExecutor::spawnThreads(std::shared_ptr<DB::IAcquiredSlot>)::$_0&>(DB::PipelineExecutor::spawnThreads(std::shared_ptr<DB::IAcquiredSlot>)::$_0&)
clickhouse-server.err.log:2025.09.25 19:42:53.685248 [ 1622 ] {} <Fatal> BaseDaemon: 20.4. inlined from ./contrib/llvm-project/libcxx/include/__functional/function.h:210: ?
clickhouse-server.err.log:2025.09.25 19:42:53.685253 [ 1622 ] {} <Fatal> BaseDaemon: 20. ./contrib/llvm-project/libcxx/include/__functional/function.h:610: ? @ 0x0000000019d64a85
clickhouse-server.err.log:2025.09.25 19:42:53.694856 [ 1622 ] {} <Fatal> BaseDaemon: 21.0. inlined from ./contrib/llvm-project/libcxx/include/__functional/function.h:716: ?
clickhouse-server.err.log:2025.09.25 19:42:53.694885 [ 1622 ] {} <Fatal> BaseDaemon: 21.1. inlined from ./contrib/llvm-project/libcxx/include/__functional/function.h:989: ?
clickhouse-server.err.log:2025.09.25 19:42:53.694908 [ 1622 ] {} <Fatal> BaseDaemon: 21. ./ci/tmp/build/./src/Common/ThreadPool.cpp:802: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::worker() @ 0x000000000ff145d5
clickhouse-server.err.log:2025.09.25 19:42:53.701992 [ 1622 ] {} <Fatal> BaseDaemon: 22.0. inlined from ./contrib/llvm-project/libcxx/include/__type_traits/invoke.h:117: decltype(*std::declval<ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool*&>().*std::declval<void (ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::*&)()>()()) std::__invoke[abi:ne190107]<void (ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::*&)(), ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool*&, void>(void (ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::*&)(), ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool*&)
clickhouse-server.err.log:2025.09.25 19:42:53.702018 [ 1622 ] {} <Fatal> BaseDaemon: 22.1. inlined from ./contrib/llvm-project/libcxx/include/tuple:1354: decltype(auto) std::__apply_tuple_impl[abi:ne190107]<void (ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::*&)(), std::tuple<ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool*>&, 0ul>(void (ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::*&)(), std::tuple<ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool*>&, std::__tuple_indices<0ul>)
clickhouse-server.err.log:2025.09.25 19:42:53.702034 [ 1622 ] {} <Fatal> BaseDaemon: 22.2. inlined from ./contrib/llvm-project/libcxx/include/tuple:1358: decltype(auto) std::apply[abi:ne190107]<void (ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::*&)(), std::tuple<ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool*>&>(void (ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::*&)(), std::tuple<ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool*>&)
clickhouse-server.err.log:2025.09.25 19:42:53.702048 [ 1622 ] {} <Fatal> BaseDaemon: 22. ./src/Common/ThreadPool.h:312: ThreadFromGlobalPoolImpl<false, true>::ThreadFromGlobalPoolImpl<void (ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::*)(), ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool*>(void (ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::*&&)(), ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool*&&)::'lambda'()::operator()() @ 0x000000000ff19d26
clickhouse-server.err.log:2025.09.25 19:42:53.708893 [ 1622 ] {} <Fatal> BaseDaemon: 23.0. inlined from ./contrib/llvm-project/libcxx/include/__functional/function.h:716: ?
clickhouse-server.err.log:2025.09.25 19:42:53.708918 [ 1622 ] {} <Fatal> BaseDaemon: 23.1. inlined from ./contrib/llvm-project/libcxx/include/__functional/function.h:989: ?
clickhouse-server.err.log:2025.09.25 19:42:53.708927 [ 1622 ] {} <Fatal> BaseDaemon: 23. ./ci/tmp/build/./src/Common/ThreadPool.cpp:812: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x000000000ff12020
clickhouse-server.err.log:2025.09.25 19:42:53.722658 [ 1622 ] {} <Fatal> BaseDaemon: 24.0. inlined from ./contrib/llvm-project/libcxx/include/__type_traits/invoke.h:117: decltype(*std::declval<ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>().*std::declval<void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)()>()()) std::__invoke[abi:ne190107]<void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*, void>(void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*&&)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*&&)
clickhouse-server.err.log:2025.09.25 19:42:53.722691 [ 1622 ] {} <Fatal> BaseDaemon: 24.1. inlined from ./contrib/llvm-project/libcxx/include/__thread/thread.h:192: void std::__thread_execute[abi:ne190107]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*, 2ul>(std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>&, std::__tuple_indices<2ul>)
clickhouse-server.err.log:2025.09.25 19:42:53.722704 [ 1622 ] {} <Fatal> BaseDaemon: 24. ./contrib/llvm-project/libcxx/include/__thread/thread.h:201: void* std::__thread_proxy[abi:ne190107]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x000000000ff17e3b
clickhouse-server.err.log:2025.09.25 19:42:53.722727 [ 1622 ] {} <Fatal> BaseDaemon: 25. ? @ 0x0000000000094ac3
clickhouse-server.err.log:2025.09.25 19:42:53.722739 [ 1622 ] {} <Fatal> BaseDaemon: 26. ? @ 0x0000000000126850
clickhouse-server.err.log:2025.09.25 19:42:53.935549 [ 1622 ] {} <Fatal> BaseDaemon: Integrity check of the executable successfully passed (checksum: 79AB97D48C7413F03C4A3955953DADDC)
clickhouse-server.err.log:2025.09.25 19:42:55.277309 [ 1622 ] {} <Fatal> BaseDaemon: Report this error to https://github.com/ClickHouse/ClickHouse/issues
clickhouse-server.err.log:2025.09.25 19:42:55.277445 [ 1622 ] {} <Fatal> BaseDaemon: Changed settings: min_compress_block_size = 1749775, max_compress_block_size = 377813, max_block_size = 53635, min_external_table_block_size_bytes = 1, max_joined_block_size_rows = 60145, max_insert_threads = 1, max_threads = 3, max_read_buffer_size = 938090, connect_timeout_with_failover_ms = 2000, connect_timeout_with_failover_secure_ms = 3000, use_hedged_requests = false, idle_connection_timeout = 36000, s3_max_get_rps = 1000000, s3_max_get_burst = 2000000, s3_max_put_rps = 1000000, s3_max_put_burst = 2000000, s3_check_objects_after_upload = true, max_remote_read_network_bandwidth = 1000000000000, max_remote_write_network_bandwidth = 1000000000000, max_local_read_bandwidth = 1000000000000, max_local_write_bandwidth = 1000000000000, stream_like_engine_allow_direct_select = true, enable_multiple_prewhere_read_steps = false, replication_wait_for_inactive_replica_timeout = 30, compile_expressions = false, min_count_to_compile_expression = 0, min_count_to_compile_aggregate_expression = 0, group_by_two_level_threshold = 234971, group_by_two_level_threshold_bytes = 8627332, allow_nonconst_timezone_arguments = true, min_chunk_bytes_for_parallel_parsing = 16535326, output_format_parallel_formatting = false, merge_tree_coarse_index_granularity = 7, min_bytes_to_use_direct_io = 10737418240, min_bytes_to_use_mmap_io = 10737418240, log_queries = true, insert_quorum_timeout = 60000, merge_tree_read_split_ranges_into_intersecting_and_non_intersecting_injection_probability = 0., http_wait_end_of_query = true, http_response_buffer_size = 4646614, fsync_metadata = true, query_plan_join_swap_table = false, http_send_timeout = 60., http_receive_timeout = 60., use_index_for_in_with_subqueries_max_values = 1000000000, opentelemetry_start_trace_probability = 0.10000000149011612, enable_vertical_final = false, max_rows_to_read = 20000000, max_bytes_to_read = 1, max_bytes_to_read_leaf = 1000000000000, max_rows_to_group_by = 10000000000, max_bytes_before_externa

Note

02950_part_offset_as_primary_key is in parallel_replicas_blacklist.txt but it's not expected to fail with Logical error: 'Replica 1 decided to read in WithOrder mode, not in Default. This is a bug'.. This issue supposed to be fixed by #80652 but it looks there are still some cases

Metadata

Metadata

Assignees

Labels

fuzzProblem found by one of the fuzzersred CIVERY flaky test

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions