Skip to content

Improve 03652_memory_usage_headers#90142

Merged
nihalzp merged 6 commits intoClickHouse:masterfrom
cwurm:progress_memory_usage
Nov 17, 2025
Merged

Improve 03652_memory_usage_headers#90142
nihalzp merged 6 commits intoClickHouse:masterfrom
cwurm:progress_memory_usage

Conversation

@cwurm
Copy link
Copy Markdown
Member

@cwurm cwurm commented Nov 16, 2025 •
edited by nihalzp
Loading

Resolves #90148.

Changelog category (leave one):

  • CI Fix or Improvement (changelog entry is not required)

Details

Original PR: #88393
Revert after test became flaky: #90136
Revert-revert: #90137

The test failures have in common that the first query fails with:

Code: 210. DB::Exception: I/O error: Broken pipe, while writing to socket ([::ffff:127.0.0.1]:8123 -> [::ffff:127.0.0.1]:40816): While executing NumbersRange. (NETWORK_ERROR) (version 25.11.1.2784)

There are probably many reasons why this can happen (never happened locally, I guess my laptop is fast and has stable local networking), but what we can try:

  • Remove cancel_http_readonly_queries_on_client_close = 1 (not needed for the test, and it might explain the broken connections)
  • Reduce memory usage of the first query by grouping fewer numbers, but slowing down execution speed, so it still gets progress headers
  • Make all tests follow the same format

@cwurm cwurm requested a review from nihalzp November 16, 2025 12:46
@clickhouse-gh
Copy link
Copy Markdown
Contributor

clickhouse-gh bot commented Nov 16, 2025 •
edited
Loading

Workflow [PR], commit [1a65ea0]

Summary: ✅

@clickhouse-gh clickhouse-gh bot added the pr-ci label Nov 16, 2025
@cwurm cwurm mentioned this pull request Nov 16, 2025
Merged
@nihalzp nihalzp self-assigned this Nov 16, 2025
@nihalzp
Copy link
Copy Markdown
Member

nihalzp commented Nov 16, 2025

@cwurm Stateless tests (arm_asan, targeted) is failing.

2025-11-17 01:13:42 Reason: return code:  1
2025-11-17 01:13:42 /home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/tests/queries/0_stateless/test_sobsd9k6/03652_memory_usage_headers.sh.debuglog:
2025-11-17 01:13:42 ++ [2025-11-17 01:11:32] [:16] curl -q -s --max-time 120 -s -S -v -N -G 'http://localhost:8123/?max_insert_threads=1&group_by_two_level_threshold=293215&group_by_two_level_threshold_bytes=49083116&distributed_aggregation_memory_efficient=1&fsync_metadata=1&output_format_parallel_formatting=0&input_format_parallel_parsing=0&min_chunk_bytes_for_parallel_parsing=11262525&max_read_buffer_size=860339&prefer_localhost_replica=1&max_block_size=44860&max_joined_block_size_rows=50619&joined_block_split_single_row=0&join_output_by_rowlist_perkey_rows_threshold=89&max_threads=2&optimize_append_index=0&use_hedged_requests=1&optimize_if_chain_to_multiif=0&optimize_if_transform_strings_to_enum=1&optimize_read_in_order=0&optimize_or_like_chain=1&optimize_substitute_columns=0&enable_multiple_prewhere_read_steps=1&read_in_order_two_level_merge_threshold=7&optimize_aggregation_in_order=1&aggregation_in_order_max_block_bytes=49949702&use_uncompressed_cache=1&min_bytes_to_use_direct_io=4352738259&min_bytes_to_use_mmap_io=10737418240&local_filesystem_read_method=read&remote_filesystem_read_method=read&local_filesystem_read_prefetch=1&filesystem_cache_segments_batch_size=5&read_from_filesystem_cache_if_exists_otherwise_bypass_cache=0&throw_on_error_from_cache_on_write_operations=1&remote_filesystem_read_prefetch=1&allow_prefetched_read_pool_for_remote_filesystem=1&filesystem_prefetch_max_memory_usage=64Mi&filesystem_prefetches_limit=0&filesystem_prefetch_min_bytes_for_single_read_task=1Mi&filesystem_prefetch_step_marks=50&filesystem_prefetch_step_bytes=0&compile_expressions=0&compile_aggregate_expressions=1&compile_sort_description=1&merge_tree_coarse_index_granularity=30&optimize_distinct_in_order=1&max_bytes_before_remerge_sort=969821950&min_compress_block_size=397107&max_compress_block_size=1187465&merge_tree_compact_parts_min_granules_to_multibuffer_read=116&optimize_sorting_by_input_stream_properties=0&http_response_buffer_size=8029979&http_wait_end_of_query=True&enable_memory_bound_merging_of_aggregation_results=1&min_count_to_compile_expression=3&min_count_to_compile_aggregate_expression=3&min_count_to_compile_sort_description=3&session_timezone=Mexico%2FBajaSur&use_page_cache_for_disks_without_file_cache=True&page_cache_inject_eviction=True&merge_tree_read_split_ranges_into_intersecting_and_non_intersecting_injection_probability=0.81&prefer_external_sort_block_bytes=0&cross_join_min_rows_to_compress=0&cross_join_min_bytes_to_compress=0&min_external_table_block_size_bytes=100000000&max_parsing_threads=1&optimize_functions_to_subcolumns=0&parallel_replicas_local_plan=0&query_plan_join_swap_table=auto&enable_vertical_final=1&optimize_extract_common_expressions=1&use_async_executor_for_materialized_views=0&use_query_condition_cache=1&secondary_indices_enable_bulk_filtering=0&use_skip_indexes_if_final=1&use_skip_indexes_on_data_read=1&optimize_rewrite_like_perfect_affix=0&input_format_parquet_use_native_reader_v3=0&enable_lazy_columns_replication=1&allow_special_serialization_kinds_in_output_formats=1&max_bytes_before_external_sort=10737418240&max_bytes_before_external_group_by=29909651&max_bytes_ratio_before_external_sort=0&max_bytes_ratio_before_external_group_by=0&use_skip_indexes_if_final_exact_mode=1&database=test_sobsd9k6&log_comment=03652_memory_usage_headers.sh-test_sobsd9k6' --data-urlencode 'query=SELECT number, avg(number) FROM numbers(1e6) GROUP BY number FORMAT Null' --data-urlencode send_progress_in_http_headers=1 --data-urlencode http_headers_progress_interval_ms=10 --data-urlencode max_execution_speed=200000 --data-urlencode timeout_before_checking_execution_speed=0
2025-11-17 01:13:42 + [2025-11-17 01:11:42] [:15] CURL_OUTPUT='*   Trying 127.0.0.1:8123...
2025-11-17 01:13:42 * Connected to localhost (127.0.0.1) port 8123 (#0)
2025-11-17 01:13:42 > GET /?max_insert_threads=1&group_by_two_level_threshold=293215&group_by_two_level_threshold_bytes=49083116&distributed_aggregation_memory_efficient=1&fsync_metadata=1&output_format_parallel_formatting=0&input_format_parallel_parsing=0&min_chunk_bytes_for_parallel_parsing=11262525&max_read_buffer_size=860339&prefer_localhost_replica=1&max_block_size=44860&max_joined_block_size_rows=50619&joined_block_split_single_row=0&join_output_by_rowlist_perkey_rows_threshold=89&max_threads=2&optimize_append_index=0&use_hedged_requests=1&optimize_if_chain_to_multiif=0&optimize_if_transform_strings_to_enum=1&optimize_read_in_order=0&optimize_or_like_chain=1&optimize_substitute_columns=0&enable_multiple_prewhere_read_steps=1&read_in_order_two_level_merge_threshold=7&optimize_aggregation_in_order=1&aggregation_in_order_max_block_bytes=49949702&use_uncompressed_cache=1&min_bytes_to_use_direct_io=4352738259&min_bytes_to_use_mmap_io=10737418240&local_filesystem_read_method=read&remote_filesystem_read_method=read&local_filesystem_read_prefetch=1&filesystem_cache_segments_batch_size=5&read_from_filesystem_cache_if_exists_otherwise_bypass_cache=0&throw_on_error_from_cache_on_write_operations=1&remote_filesystem_read_prefetch=1&allow_prefetched_read_pool_for_remote_filesystem=1&filesystem_prefetch_max_memory_usage=64Mi&filesystem_prefetches_limit=0&filesystem_prefetch_min_bytes_for_single_read_task=1Mi&filesystem_prefetch_step_marks=50&filesystem_prefetch_step_bytes=0&compile_expressions=0&compile_aggregate_expressions=1&compile_sort_description=1&merge_tree_coarse_index_granularity=30&optimize_distinct_in_order=1&max_bytes_before_remerge_sort=969821950&min_compress_block_size=397107&max_compress_block_size=1187465&merge_tree_compact_parts_min_granules_to_multibuffer_read=116&optimize_sorting_by_input_stream_properties=0&http_response_buffer_size=8029979&http_wait_end_of_query=True&enable_memory_bound_merging_of_aggregation_results=1&min_count_to_compile_expression=3&min_count_to_compile_aggregate_expression=3&min_count_to_compile_sort_description=3&session_timezone=Mexico%2FBajaSur&use_page_cache_for_disks_without_file_cache=True&page_cache_inject_eviction=True&merge_tree_read_split_ranges_into_intersecting_and_non_intersecting_injection_probability=0.81&prefer_external_sort_block_bytes=0&cross_join_min_rows_to_compress=0&cross_join_min_bytes_to_compress=0&min_external_table_block_size_bytes=100000000&max_parsing_threads=1&optimize_functions_to_subcolumns=0&parallel_replicas_local_plan=0&query_plan_join_swap_table=auto&enable_vertical_final=1&optimize_extract_common_expressions=1&use_async_executor_for_materialized_views=0&use_query_condition_cache=1&secondary_indices_enable_bulk_filtering=0&use_skip_indexes_if_final=1&use_skip_indexes_on_data_read=1&optimize_rewrite_like_perfect_affix=0&input_format_parquet_use_native_reader_v3=0&enable_lazy_columns_replication=1&allow_special_serialization_kinds_in_output_formats=1&max_bytes_before_external_sort=10737418240&max_bytes_before_external_group_by=29909651&max_bytes_ratio_before_external_sort=0&max_bytes_ratio_before_external_group_by=0&use_skip_indexes_if_final_exact_mode=1&database=test_sobsd9k6&log_comment=03652_memory_usage_headers.sh-test_sobsd9k6&query=SELECT+number%2C+avg%28number%29+FROM+numbers%281e6%29+GROUP+BY+number+FORMAT+Null&send_progress_in_http_headers=1&http_headers_progress_interval_ms=10&max_execution_speed=200000&timeout_before_checking_execution_speed=0 HTTP/1.1
2025-11-17 01:13:42 > Host: localhost:8123
2025-11-17 01:13:42 > User-Agent: curl/7.81.0
2025-11-17 01:13:42 > Accept: */*
2025-11-17 01:13:42 > 
2025-11-17 01:13:42 * Mark bundle as not supporting multiuse
2025-11-17 01:13:42 < HTTP/1.1 200 OK
2025-11-17 01:13:42 < X-ClickHouse-Progress: {"total_rows_to_read":"1000000","elapsed_ns":"127206923","memory_usage":"2097247"}
2025-11-17 01:13:42 < X-ClickHouse-Progress: {"read_rows":"89720","read_bytes":"717760","total_rows_to_read":"1000000","elapsed_ns":"405874959","memory_usage":"7695039"}
2025-11-17 01:13:42 < X-ClickHouse-Progress: {"read_rows":"134580","read_bytes":"1076640","total_rows_to_read":"1000000","elapsed_ns":"578920110","memory_usage":"9102495"}
2025-11-17 01:13:42 < X-ClickHouse-Progress: {"read_rows":"179440","read_bytes":"1435520","total_rows_to_read":"1000000","elapsed_ns":"920579745","memory_usage":"23782559"}
2025-11-17 01:13:42 < X-ClickHouse-Progress: {"read_rows":"224300","read_bytes":"1794400","total_rows_to_read":"1000000","elapsed_ns":"1028789700","memory_usage":"23782559"}
2025-11-17 01:13:42 < X-ClickHouse-Progress: {"read_rows":"269160","read_bytes":"2153280","total_rows_to_read":"1000000","elapsed_ns":"1254710474","memory_usage":"23782559"}
2025-11-17 01:13:42 < X-ClickHouse-Progress: {"read_rows":"314020","read_bytes":"2512160","total_rows_to_read":"1000000","elapsed_ns":"1473221697","memory_usage":"27976863"}
2025-11-17 01:13:42 < X-ClickHouse-Progress: {"read_rows":"358880","read_bytes":"2871040","total_rows_to_read":"1000000","elapsed_ns":"2008918601","memory_usage":"27617983"}
2025-11-17 01:13:42 < X-ClickHouse-Progress: {"read_rows":"448600","read_bytes":"3588800","total_rows_to_read":"1000000","elapsed_ns":"2155359884","memory_usage":"27617983"}
2025-11-17 01:13:42 < X-ClickHouse-Progress: {"read_rows":"493460","read_bytes":"3947680","total_rows_to_read":"1000000","elapsed_ns":"2378616336","memory_usage":"27617983"}
2025-11-17 01:13:42 < X-ClickHouse-Progress: {"read_rows":"538320","read_bytes":"4306560","total_rows_to_read":"1000000","elapsed_ns":"2666129079","memory_usage":"27617983"}
2025-11-17 01:13:42 < X-ClickHouse-Progress: {"read_rows":"583180","read_bytes":"4665440","total_rows_to_read":"1000000","elapsed_ns":"4089585128","memory_usage":"3505087"}
2025-11-17 01:13:42 < X-ClickHouse-Progress: {"read_rows":"672900","read_bytes":"5383200","total_rows_to_read":"1000000","elapsed_ns":"4101482470","memory_usage":"9102751"}
2025-11-17 01:13:42 < X-ClickHouse-Progress: {"read_rows":"762620","read_bytes":"6100960","total_rows_to_read":"1000000","elapsed_ns":"4114737523","memory_usage":"23111071"}
2025-11-17 01:13:42 < X-ClickHouse-Progress: {"read_rows":"807480","read_bytes":"6459840","total_rows_to_read":"1000000","elapsed_ns":"4125955029","memory_usage":"23111071"}
2025-11-17 01:13:42 < X-ClickHouse-Progress: {"read_rows":"852340","read_bytes":"6818720","total_rows_to_read":"1000000","elapsed_ns":"4209624122","memory_usage":"27977119"}
2025-11-17 01:13:42 < X-ClickHouse-Progress: {"read_rows":"897200","read_bytes":"7177600","total_rows_to_read":"1000000","elapsed_ns":"4394775720","memory_usage":"27977119"}
2025-11-17 01:13:42 < X-ClickHouse-Progress: {"read_rows":"942060","read_bytes":"7536480","total_rows_to_read":"1000000","elapsed_ns":"4738143979","memory_usage":"27977119"}
2025-11-17 01:13:42 < X-ClickHouse-Progress: {"read_rows":"986920","read_bytes":"7895360","total_rows_to_read":"1000000","elapsed_ns":"4880917548","memory_usage":"27977119"}
2025-11-17 01:13:42 < X-ClickHouse-Progress: {"read_rows":"1000000","read_bytes":"8000000","total_rows_to_read":"1000000","elapsed_ns":"5065945039","memory_usage":"27977119"}
2025-11-17 01:13:42 < X-ClickHouse-Progress: {"read_rows":"1000000","read_bytes":"8000000","total_rows_to_read":"1000000","result_rows":"1000000","result_bytes":"16423424","elapsed_ns":"9227081165","memory_usage":"88435645"}
2025-11-17 01:13:42 < X-ClickHouse-Summary: {"read_rows":"1000000","read_bytes":"8000000","written_rows":"0","written_bytes":"0","total_rows_to_read":"1000000","result_rows":"1000000","result_bytes":"16423424","elapsed_ns":"9227276603","memory_usage":"88435645"}
2025-11-17 01:13:42 < Date: Sun, 16 Nov 2025 13:11:32 GMT
2025-11-17 01:13:42 < Connection: Keep-Alive
2025-11-17 01:13:42 < Content-Type: text/plain; charset=UTF-8
2025-11-17 01:13:42 < Access-Control-Expose-Headers: X-ClickHouse-Query-Id,X-ClickHouse-Summary,X-ClickHouse-Server-Display-Name,X-ClickHouse-Format,X-ClickHouse-Timezone,X-ClickHouse-Exception-Code,X-ClickHouse-Exception-Tag
2025-11-17 01:13:42 < X-ClickHouse-Server-Display-Name: functional-tests
2025-11-17 01:13:42 < Transfer-Encoding: chunked
2025-11-17 01:13:42 < X-ClickHouse-Query-Id: 76f1133b-2fc5-4729-bfda-e7d692ec0f06
2025-11-17 01:13:42 < X-ClickHouse-Format: Null
2025-11-17 01:13:42 < X-ClickHouse-Timezone: Mexico/BajaSur
2025-11-17 01:13:42 < Keep-Alive: timeout=10, max=9999
2025-11-17 01:13:42 < X-ClickHouse-Exception-Tag: acqqolmmscvbsdki
2025-11-17 01:13:42 < 
2025-11-17 01:13:42 { [5 bytes data]
2025-11-17 01:13:42 * Connection #0 to host localhost left intact'
2025-11-17 01:13:42 + [2025-11-17 01:11:42] [:17] echo '*   Trying 127.0.0.1:8123...
2025-11-17 01:13:42 * Connected to localhost (127.0.0.1) port 8123 (#0)
2025-11-17 01:13:42 > GET /?max_insert_threads=1&group_by_two_level_threshold=293215&group_by_two_level_threshold_bytes=49083116&distributed_aggregation_memory_efficient=1&fsync_metadata=1&output_format_parallel_formatting=0&input_format_parallel_parsing=0&min_chunk_bytes_for_parallel_parsing=11262525&max_read_buffer_size=860339&prefer_localhost_replica=1&max_block_size=44860&max_joined_block_size_rows=50619&joined_block_split_single_row=0&join_output_by_rowlist_perkey_rows_threshold=89&max_threads=2&optimize_append_index=0&use_hedged_requests=1&optimize_if_chain_to_multiif=0&optimize_if_transform_strings_to_enum=1&optimize_read_in_order=0&optimize_or_like_chain=1&optimize_substitute_columns=0&enable_multiple_prewhere_read_steps=1&read_in_order_two_level_merge_threshold=7&optimize_aggregation_in_order=1&aggregation_in_order_max_block_bytes=49949702&use_uncompressed_cache=1&min_bytes_to_use_direct_io=4352738259&min_bytes_to_use_mmap_io=10737418240&local_filesystem_read_method=read&remote_filesystem_read_method=read&local_filesystem_read_prefetch=1&filesystem_cache_segments_batch_size=5&read_from_filesystem_cache_if_exists_otherwise_bypass_cache=0&throw_on_error_from_cache_on_write_operations=1&remote_filesystem_read_prefetch=1&allow_prefetched_read_pool_for_remote_filesystem=1&filesystem_prefetch_max_memory_usage=64Mi&filesystem_prefetches_limit=0&filesystem_prefetch_min_bytes_for_single_read_task=1Mi&filesystem_prefetch_step_marks=50&filesystem_prefetch_step_bytes=0&compile_expressions=0&compile_aggregate_expressions=1&compile_sort_description=1&merge_tree_coarse_index_granularity=30&optimize_distinct_in_order=1&max_bytes_before_remerge_sort=969821950&min_compress_block_size=397107&max_compress_block_size=1187465&merge_tree_compact_parts_min_granules_to_multibuffer_read=116&optimize_sorting_by_input_stream_properties=0&http_response_buffer_size=8029979&http_wait_end_of_query=True&enable_memory_bound_merging_of_aggregation_results=1&min_count_to_compile_expression=3&min_count_to_compile_aggregate_expression=3&min_count_to_compile_sort_description=3&session_timezone=Mexico%2FBajaSur&use_page_cache_for_disks_without_file_cache=True&page_cache_inject_eviction=True&merge_tree_read_split_ranges_into_intersecting_and_non_intersecting_injection_probability=0.81&prefer_external_sort_block_bytes=0&cross_join_min_rows_to_compress=0&cross_join_min_bytes_to_compress=0&min_external_table_block_size_bytes=100000000&max_parsing_threads=1&optimize_functions_to_subcolumns=0&parallel_replicas_local_plan=0&query_plan_join_swap_table=auto&enable_vertical_final=1&optimize_extract_common_expressions=1&use_async_executor_for_materialized_views=0&use_query_condition_cache=1&secondary_indices_enable_bulk_filtering=0&use_skip_indexes_if_final=1&use_skip_indexes_on_data_read=1&optimize_rewrite_like_perfect_affix=0&input_format_parquet_use_native_reader_v3=0&enable_lazy_columns_replication=1&allow_special_serialization_kinds_in_output_formats=1&max_bytes_before_external_sort=10737418240&max_bytes_before_external_group_by=29909651&max_bytes_ratio_before_external_sort=0&max_bytes_ratio_before_external_group_by=0&use_skip_indexes_if_final_exact_mode=1&database=test_sobsd9k6&log_comment=03652_memory_usage_headers.sh-test_sobsd9k6&query=SELECT+number%2C+avg%28number%29+FROM+numbers%281e6%29+GROUP+BY+number+FORMAT+Null&send_progress_in_http_headers=1&http_headers_progress_interval_ms=10&max_execution_speed=200000&timeout_before_checking_execution_speed=0 HTTP/1.1
2025-11-17 01:13:42 ----------------------------------------943 lines are hidden------------------------------------

@cwurm
Copy link
Copy Markdown
Member Author

cwurm commented Nov 16, 2025

Thanks, another Broken pipe. This time while sending progress headers during the execution of the third test that's meant to OOM. Sending progress headers isn't needed for that test, it just checks whether the summary reports memory usage for an OOM query. I'll remove sending progress headers for that one, hope it will fix it.

@nihalzp
Copy link
Copy Markdown
Member

nihalzp commented Nov 17, 2025 •
edited
Loading

Thanks! It seems to be fixed. I will just rerun CI couple more times to just in case there is some 'edge case flakiness' remain.

@nihalzp nihalzp added this pull request to the merge queue Nov 17, 2025
Merged via the queue into ClickHouse:master with commit ad060e7 Nov 17, 2025
132 checks passed
@robot-clickhouse robot-clickhouse added the pr-synced-to-cloud The PR is synced to the cloud repo label Nov 17, 2025
@Felixoid
Copy link
Copy Markdown
Member

Felixoid commented Dec 5, 2025

It looks like it didn't help for 100% https://play.clickhouse.com/play?user=play&run=1#V0lUSAogICAgOTAgQVMgaW50ZXJ2YWxfZGF5cwpTRUxFQ1QKICAgIHRvU3RhcnRPZkRheShjaGVja19zdGFydF90aW1lKSBBUyBkYXksCiAgICBjb3VudCgpIEFTIGZhaWx1cmVzLAogICAgZ3JvdXBVbmlxQXJyYXkocHVsbF9yZXF1ZXN0X251bWJlcikgQVMgcHJzLAogICAgYW55KHJlcG9ydF91cmwpIEFTIHJlcG9ydF91cmwKRlJPTSBjaGVja3MKV0hFUkUgKG5vdygpIC0gdG9JbnRlcnZhbERheShpbnRlcnZhbF9kYXlzKSkgPD0gY2hlY2tfc3RhcnRfdGltZQogICAgQU5EIHRlc3RfbmFtZSA9ICcwMzY1Ml9tZW1vcnlfdXNhZ2VfaGVhZGVycycKICAgIC0tIEFORCBjaGVja19uYW1lID0gJ1N0YXRlbGVzcyB0ZXN0cyAoYW1kX2RlYnVnLCBwYXJhbGxlbCknCiAgICBBTkQgdGVzdF9zdGF0dXMgSU4gKCdGQUlMJywgJ0VSUk9SJykKICAgIEFORCAoKGhlYWRfcmVmID0gJ21hc3RlcicgQU5EIHB1bGxfcmVxdWVzdF9udW1iZXIgPSAwKSBPUiBwdWxsX3JlcXVlc3RfbnVtYmVyICE9IDApCkdST1VQIEJZIGRheQpPUkRFUiBCWSBkYXkgREVTQwo=

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@nihalzp nihalzp nihalzp approved these changes

Assignees

@nihalzp nihalzp

Labels

pr-ci pr-synced-to-cloud The PR is synced to the cloud repo

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Test 03652_memory_usage_headers is flaky

4 participants

@cwurm @nihalzp @Felixoid @robot-clickhouse