-
Notifications
You must be signed in to change notification settings - Fork 8.3k
Flaky CREATE TABLE on ReplicatedDatabase with Log/Memory engines (ReplicatedDatabase DDL task is not finished on 1 of 3 hosts) #81382
Description
Company or project name
No response
Describe what's wrong
Pretty rare flaky tests with DatabaseReplicated configuration in CI and Log and Memory table engines:
https://play.clickhouse.com/play?user=play#U0VMRUNUIHRlc3RfZHVyYXRpb25fbXMsIGNoZWNrX3N0YXJ0X3RpbWUsIGNoZWNrX25hbWUsIHRlc3RfbmFtZSwgcmVwb3J0X3VybApGUk9NIGNoZWNrcwpXSEVSRSAxCiAgICBBTkQgY2hlY2tfc3RhcnRfdGltZSA+PSBub3coKSAtIElOVEVSVkFMIDYwIGRheQogICAgLS1BTkQgKGhlYWRfcmVmID0gJ21hc3RlcicgQU5EIHN0YXJ0c1dpdGgoaGVhZF9yZXBvLCAnQ2xpY2tIb3VzZS8nKSkKICAgIEFORCB0ZXN0X3N0YXR1cyAhPSAnU0tJUFBFRCcKICAgIEFORCAodGVzdF9zdGF0dXMgTElLRSAnRiUnIE9SIHRlc3Rfc3RhdHVzIExJS0UgJ0UlJykgCiAgICBBTkQgY2hlY2tfc3RhdHVzICE9ICdzdWNjZXNzJyBhbmQgY2hlY2tfc3RhdHVzIT0nZXJyb3InICBhbmQgdGVzdF9jb250ZXh0X3JhdyBsaWtlICclUmVwbGljYXRlZERhdGFiYXNlIERETCB0YXNrICUgaXMgbm90IGZpbmlzaGVkJScgIGFuZCAoIHRlc3RfY29udGV4dF9yYXcgbGlrZSAnJU1lbW9yeSUnIG9yIHRlc3RfY29udGV4dF9yYXcgbGlrZSAnJUxvZyUnICkgYW5kIC8qIGZpbHRlciBiYWQgcmVwb3J0cyAqLyB0ZXN0X2R1cmF0aW9uX21zIDwgNjAwMDAwCk9SREVSIEJZIGNoZWNrX3N0YXJ0X3RpbWUsIGNoZWNrX25hbWUsIHRlc3RfbmFtZQ==
Complete error
2025-06-05 15:36:28 Reason: return code: 159
2025-06-05 15:36:28 Received exception from server (version 25.6.1):
2025-06-05 15:36:28 Code: 159. DB::Exception: Received from localhost:9000. DB::Exception: ReplicatedDatabase DDL task /test/clickhouse/db/test_b3tzodd8/log/query-0000000004 is not finished on 1 of 3 hosts (0 of them are currently executing the task, 0 are inactive). They are going to execute the query in background. Was waiting for 120.641226251 seconds, which is longer than distributed_ddl_task_timeout. (TIMEOUT_EXCEEDED)
2025-06-05 15:36:28 (query: CREATE TABLE mann_whitney_test (left Float64, right UInt8) ENGINE = Memory;)
2025-06-05 15:36:28 , result:
2025-06-05 15:36:28
2025-06-05 15:36:28
2025-06-05 15:36:28
2025-06-05 15:36:28 stdout:
2025-06-05 15:36:28
2025-06-05 15:36:28
2025-06-05 15:36:28 Settings used in the test: --max_insert_threads 2 --group_by_two_level_threshold 1000000 --group_by_two_level_threshold_bytes 45603169 --distributed_aggregation_memory_efficient 0 --fsync_metadata 0 --output_format_parallel_formatting 1 --input_format_parallel_parsing 1 --min_chunk_bytes_for_parallel_parsing 2783958 --max_read_buffer_size 864226 --prefer_localhost_replica 0 --max_block_size 47403 --max_joined_block_size_rows 20532 --max_threads 3 --optimize_append_index 1 --use_hedged_requests 0 --optimize_if_chain_to_multiif 0 --optimize_if_transform_strings_to_enum 1 --optimize_read_in_order 0 --optimize_or_like_chain 0 --optimize_substitute_columns 1 --enable_multiple_prewhere_read_steps 1 --read_in_order_two_level_merge_threshold 57 --optimize_aggregation_in_order 0 --aggregation_in_order_max_block_bytes 30005437 --use_uncompressed_cache 1 --min_bytes_to_use_direct_io 1 --min_bytes_to_use_mmap_io 9366087240 --local_filesystem_read_method read --remote_filesystem_read_method read --local_filesystem_read_prefetch 1 --filesystem_cache_segments_batch_size 2 --read_from_filesystem_cache_if_exists_otherwise_bypass_cache 0 --throw_on_error_from_cache_on_write_operations 0 --remote_filesystem_read_prefetch 1 --allow_prefetched_read_pool_for_remote_filesystem 1 --filesystem_prefetch_max_memory_usage 32Mi --filesystem_prefetches_limit 0 --filesystem_prefetch_min_bytes_for_single_read_task 1Mi --filesystem_prefetch_step_marks 0 --filesystem_prefetch_step_bytes 0 --compile_expressions 0 --compile_aggregate_expressions 1 --compile_sort_description 1 --merge_tree_coarse_index_granularity 26 --optimize_distinct_in_order 0 --max_bytes_before_remerge_sort 1700797872 --min_compress_block_size 3111358 --max_compress_block_size 1414154 --merge_tree_compact_parts_min_granules_to_multibuffer_read 103 --optimize_sorting_by_input_stream_properties 0 --http_response_buffer_size 522475 --http_wait_end_of_query False --enable_memory_bound_merging_of_aggregation_results 0 --min_count_to_compile_expression 3 --min_count_to_compile_aggregate_expression 0 --min_count_to_compile_sort_description 0 --session_timezone Mexico/BajaSur --use_page_cache_for_disks_without_file_cache False --page_cache_inject_eviction True --merge_tree_read_split_ranges_into_intersecting_and_non_intersecting_injection_probability 0.92 --prefer_external_sort_block_bytes 0 --cross_join_min_rows_to_compress 100000000 --cross_join_min_bytes_to_compress 0 --min_external_table_block_size_bytes 0 --max_parsing_threads 0 --optimize_functions_to_subcolumns 1 --parallel_replicas_local_plan 0 --query_plan_join_swap_table auto --enable_vertical_final 0 --optimize_extract_common_expressions 0 --use_query_condition_cache 0 --secondary_indices_enable_bulk_filtering 0 --use_skip_indexes_if_final 1 --max_bytes_before_external_sort 0 --max_bytes_before_external_group_by 0 --max_bytes_ratio_before_external_sort 0.0 --max_bytes_ratio_before_external_group_by 0.08 --use_skip_indexes_if_final_exact_mode 1
2025-06-05 15:36:28
2025-06-05 15:36:28 MergeTree settings used in test: --ratio_of_defaults_for_sparse_serialization 1.0 --prefer_fetch_merged_part_size_threshold 10737418240 --vertical_merge_algorithm_min_rows_to_activate 1000000 --vertical_merge_algorithm_min_columns_to_activate 100 --allow_vertical_merges_from_compact_to_wide_parts 1 --min_merge_bytes_to_use_direct_io 6371087164 --index_granularity_bytes 22568169 --merge_max_block_size 1150 --index_granularity 29110 --min_bytes_for_wide_part 201069676 --marks_compress_block_size 60318 --primary_key_compress_block_size 49923 --replace_long_file_name_to_hash 0 --max_file_name_length 128 --min_bytes_for_full_part_storage 415190351 --compact_parts_max_bytes_to_buffer 176080533 --compact_parts_max_granules_to_buffer 148 --compact_parts_merge_max_bytes_to_prefetch_part 27996067 --cache_populated_by_fetch 0 --concurrent_part_removal_threshold 100 --old_parts_lifetime 10 --prewarm_mark_cache 0 --use_const_adaptive_granularity 0 --enable_index_granularity_compression 0 --enable_block_number_column 1 --enable_block_offset_column 0 --use_primary_key_cache 0 --prewarm_primary_key_cache 1
2025-06-05 15:36:28
2025-06-05 15:36:28 Database: test_b3tzodd8
Does it reproduce on the most recent release?
Yes
Expected behavior
No response
Error message and/or stacktrace
No response
Additional context
No response