Skip to content

Flaky CREATE TABLE on ReplicatedDatabase with Log/Memory engines (ReplicatedDatabase DDL task is not finished on 1 of 3 hosts) #81382

@fm4v

Description

@fm4v

Company or project name

No response

Describe what's wrong

Pretty rare flaky tests with DatabaseReplicated configuration in CI and Log and Memory table engines:
https://play.clickhouse.com/play?user=play#U0VMRUNUIHRlc3RfZHVyYXRpb25fbXMsIGNoZWNrX3N0YXJ0X3RpbWUsIGNoZWNrX25hbWUsIHRlc3RfbmFtZSwgcmVwb3J0X3VybApGUk9NIGNoZWNrcwpXSEVSRSAxCiAgICBBTkQgY2hlY2tfc3RhcnRfdGltZSA+PSBub3coKSAtIElOVEVSVkFMIDYwIGRheQogICAgLS1BTkQgKGhlYWRfcmVmID0gJ21hc3RlcicgQU5EIHN0YXJ0c1dpdGgoaGVhZF9yZXBvLCAnQ2xpY2tIb3VzZS8nKSkKICAgIEFORCB0ZXN0X3N0YXR1cyAhPSAnU0tJUFBFRCcKICAgIEFORCAodGVzdF9zdGF0dXMgTElLRSAnRiUnIE9SIHRlc3Rfc3RhdHVzIExJS0UgJ0UlJykgCiAgICBBTkQgY2hlY2tfc3RhdHVzICE9ICdzdWNjZXNzJyBhbmQgY2hlY2tfc3RhdHVzIT0nZXJyb3InICBhbmQgdGVzdF9jb250ZXh0X3JhdyBsaWtlICclUmVwbGljYXRlZERhdGFiYXNlIERETCB0YXNrICUgaXMgbm90IGZpbmlzaGVkJScgIGFuZCAoIHRlc3RfY29udGV4dF9yYXcgbGlrZSAnJU1lbW9yeSUnIG9yIHRlc3RfY29udGV4dF9yYXcgbGlrZSAnJUxvZyUnICkgYW5kIC8qIGZpbHRlciBiYWQgcmVwb3J0cyAqLyB0ZXN0X2R1cmF0aW9uX21zIDwgNjAwMDAwCk9SREVSIEJZIGNoZWNrX3N0YXJ0X3RpbWUsIGNoZWNrX25hbWUsIHRlc3RfbmFtZQ==

Complete error

2025-06-05 15:36:28 Reason: return code:  159
2025-06-05 15:36:28 Received exception from server (version 25.6.1):
2025-06-05 15:36:28 Code: 159. DB::Exception: Received from localhost:9000. DB::Exception: ReplicatedDatabase DDL task /test/clickhouse/db/test_b3tzodd8/log/query-0000000004 is not finished on 1 of 3 hosts (0 of them are currently executing the task, 0 are inactive). They are going to execute the query in background. Was waiting for 120.641226251 seconds, which is longer than distributed_ddl_task_timeout. (TIMEOUT_EXCEEDED)
2025-06-05 15:36:28 (query: CREATE TABLE mann_whitney_test (left Float64, right UInt8) ENGINE = Memory;)
2025-06-05 15:36:28 , result:
2025-06-05 15:36:28 
2025-06-05 15:36:28 
2025-06-05 15:36:28 
2025-06-05 15:36:28 stdout:
2025-06-05 15:36:28 
2025-06-05 15:36:28 
2025-06-05 15:36:28 Settings used in the test: --max_insert_threads 2 --group_by_two_level_threshold 1000000 --group_by_two_level_threshold_bytes 45603169 --distributed_aggregation_memory_efficient 0 --fsync_metadata 0 --output_format_parallel_formatting 1 --input_format_parallel_parsing 1 --min_chunk_bytes_for_parallel_parsing 2783958 --max_read_buffer_size 864226 --prefer_localhost_replica 0 --max_block_size 47403 --max_joined_block_size_rows 20532 --max_threads 3 --optimize_append_index 1 --use_hedged_requests 0 --optimize_if_chain_to_multiif 0 --optimize_if_transform_strings_to_enum 1 --optimize_read_in_order 0 --optimize_or_like_chain 0 --optimize_substitute_columns 1 --enable_multiple_prewhere_read_steps 1 --read_in_order_two_level_merge_threshold 57 --optimize_aggregation_in_order 0 --aggregation_in_order_max_block_bytes 30005437 --use_uncompressed_cache 1 --min_bytes_to_use_direct_io 1 --min_bytes_to_use_mmap_io 9366087240 --local_filesystem_read_method read --remote_filesystem_read_method read --local_filesystem_read_prefetch 1 --filesystem_cache_segments_batch_size 2 --read_from_filesystem_cache_if_exists_otherwise_bypass_cache 0 --throw_on_error_from_cache_on_write_operations 0 --remote_filesystem_read_prefetch 1 --allow_prefetched_read_pool_for_remote_filesystem 1 --filesystem_prefetch_max_memory_usage 32Mi --filesystem_prefetches_limit 0 --filesystem_prefetch_min_bytes_for_single_read_task 1Mi --filesystem_prefetch_step_marks 0 --filesystem_prefetch_step_bytes 0 --compile_expressions 0 --compile_aggregate_expressions 1 --compile_sort_description 1 --merge_tree_coarse_index_granularity 26 --optimize_distinct_in_order 0 --max_bytes_before_remerge_sort 1700797872 --min_compress_block_size 3111358 --max_compress_block_size 1414154 --merge_tree_compact_parts_min_granules_to_multibuffer_read 103 --optimize_sorting_by_input_stream_properties 0 --http_response_buffer_size 522475 --http_wait_end_of_query False --enable_memory_bound_merging_of_aggregation_results 0 --min_count_to_compile_expression 3 --min_count_to_compile_aggregate_expression 0 --min_count_to_compile_sort_description 0 --session_timezone Mexico/BajaSur --use_page_cache_for_disks_without_file_cache False --page_cache_inject_eviction True --merge_tree_read_split_ranges_into_intersecting_and_non_intersecting_injection_probability 0.92 --prefer_external_sort_block_bytes 0 --cross_join_min_rows_to_compress 100000000 --cross_join_min_bytes_to_compress 0 --min_external_table_block_size_bytes 0 --max_parsing_threads 0 --optimize_functions_to_subcolumns 1 --parallel_replicas_local_plan 0 --query_plan_join_swap_table auto --enable_vertical_final 0 --optimize_extract_common_expressions 0 --use_query_condition_cache 0 --secondary_indices_enable_bulk_filtering 0 --use_skip_indexes_if_final 1 --max_bytes_before_external_sort 0 --max_bytes_before_external_group_by 0 --max_bytes_ratio_before_external_sort 0.0 --max_bytes_ratio_before_external_group_by 0.08 --use_skip_indexes_if_final_exact_mode 1
2025-06-05 15:36:28 
2025-06-05 15:36:28 MergeTree settings used in test: --ratio_of_defaults_for_sparse_serialization 1.0 --prefer_fetch_merged_part_size_threshold 10737418240 --vertical_merge_algorithm_min_rows_to_activate 1000000 --vertical_merge_algorithm_min_columns_to_activate 100 --allow_vertical_merges_from_compact_to_wide_parts 1 --min_merge_bytes_to_use_direct_io 6371087164 --index_granularity_bytes 22568169 --merge_max_block_size 1150 --index_granularity 29110 --min_bytes_for_wide_part 201069676 --marks_compress_block_size 60318 --primary_key_compress_block_size 49923 --replace_long_file_name_to_hash 0 --max_file_name_length 128 --min_bytes_for_full_part_storage 415190351 --compact_parts_max_bytes_to_buffer 176080533 --compact_parts_max_granules_to_buffer 148 --compact_parts_merge_max_bytes_to_prefetch_part 27996067 --cache_populated_by_fetch 0 --concurrent_part_removal_threshold 100 --old_parts_lifetime 10 --prewarm_mark_cache 0 --use_const_adaptive_granularity 0 --enable_index_granularity_compression 0 --enable_block_number_column 1 --enable_block_offset_column 0 --use_primary_key_cache 0 --prewarm_primary_key_cache 1
2025-06-05 15:36:28 
2025-06-05 15:36:28 Database: test_b3tzodd8

Does it reproduce on the most recent release?

Yes

Expected behavior

No response

Error message and/or stacktrace

No response

Additional context

No response

Metadata

Metadata

Assignees

Labels

flaky testflaky test found by CI

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions