Add memory usage to HTTP progress and summary headers by cwurm · Pull Request #88393 · ClickHouse/ClickHouse

cwurm · 2025-10-12T09:49:07Z

Changelog category (leave one):

New Feature

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Adds a memory_usage field to X-ClickHouse-Progress and X-ClickHouse-Summary. This can be used to collect memory usage of queries in real time on the client side.

Details

Adds a memory_usage field to X-ClickHouse-Progress and X-ClickHouse-Summary. This can be used in a few different ways:

Collecting memory usage of queries in real time on the client side without having to wait for the query_log to be flushed (and keeping track of query_ids or log_comments and correlating them). This can be very useful, for example, if a query fails with a 241 MEMORY_LIMIT_EXCEEDED error, it is not immediately clear whether it is this particular query that consumed a lot of memory or if other queries were responsible (and thus this query could be re-run). While it is possible to parse the error message, which has this information, this is cumbersome, and there are no guarantees that the message format won't change in a future version.
Tracking memory usage during query execution. See an example below of visualizing it.

The headers now look like this:

$ curl -s -v -N -G http://localhost:8123/ \
  --data-urlencode "query=select number, avg(number) from numbers(1e6) group by number format Null settings max_threads = 1" \
  --data-urlencode "send_progress_in_http_headers=1" \
  --data-urlencode "http_headers_progress_interval_ms=10"

...

< HTTP/1.1 200 OK
< X-ClickHouse-Progress: {"read_rows":"196227","read_bytes":"1569816","total_rows_to_read":"1000000","elapsed_ns":"20285916","memory_usage":"9544615"}
< X-ClickHouse-Progress: {"read_rows":"981135","read_bytes":"7849080","total_rows_to_read":"1000000","elapsed_ns":"30437999","memory_usage":"83401975"}
< X-ClickHouse-Progress: {"read_rows":"1000000","read_bytes":"8000000","total_rows_to_read":"1000000","elapsed_ns":"40466708","memory_usage":"51408479"}
< X-ClickHouse-Summary: {"read_rows":"1000000","read_bytes":"8000000","written_rows":"0","written_bytes":"0","total_rows_to_read":"1000000","result_rows":"1000000","result_bytes":"16423424","elapsed_ns":"46676833","memory_usage":"85042103"}
< Date: Sun, 12 Oct 2025 09:31:15 GMT
< Connection: Keep-Alive
< Content-Type: text/plain; charset=UTF-8
< Access-Control-Expose-Headers: X-ClickHouse-Query-Id,X-ClickHouse-Summary,X-ClickHouse-Server-Display-Name,X-ClickHouse-Format,X-ClickHouse-Timezone,X-ClickHouse-Exception-Code
< X-ClickHouse-Server-Display-Name: Christophs-MacBook-Pro.local
< Transfer-Encoding: chunked
< X-ClickHouse-Query-Id: 9fa180c4-cb87-4523-b578-939dd116d0ba
< X-ClickHouse-Format: Null
< X-ClickHouse-Timezone: Europe/London

And with a simple Python script like this:

#!/usr/bin/env python3
import sys,json,re,matplotlib.pyplot as plt
x=[];y=[];p=None
for l in sys.stdin:
    l=re.sub(r'^\s*[<>\*]\s+','',l).strip()
    if not ('{' in l and '}' in l): continue
    try: d=json.loads(l.split(':',1)[1] if ':' in l else l)
    except: continue
    if 'Progress' in l:
        e=int(d.get('elapsed_ns',0))/1e9
        m=int(d.get('memory_usage',0))
        x+=[e]; y+=[m]
    elif 'Summary' in l:
        p=int(d.get('memory_usage',0))
plt.plot(x,[v/1048576 for v in y]); plt.xlabel('elapsed (s)'); plt.ylabel('MB')
plt.title(f'Peak ≈ {p/1048576:.1f} MB' if p else 'Memory usage'); plt.show()

we can visualize memory usage during query execution:

$ curl -s -v -N -G http://localhost:8123/ \                                          130 ↵
  --data-urlencode "query=select number, avg(number) from numbers(1e7) group by number format Null" \
  --data-urlencode "send_progress_in_http_headers=1" \
  --data-urlencode "http_headers_progress_interval_ms=10" 2>&1 \
| python3 tiny_ch_mem_plot.py

clickhouse-gh · 2025-10-12T09:50:25Z

Workflow [PR], commit [f9e195a]

Summary: ❌

job_name	test_name	status	info
Stateless tests (amd_binary, old analyzer, s3 storage, DatabaseReplicated, parallel)		failure
	03712_json_advanced_shared_data_bug	FAIL	cidb
BuzzHouse (amd_debug)		failure
	Buzzing result	failure	cidb

cwurm · 2025-10-20T12:01:22Z

CI had some issues (ERROR: Failed to insert data into CI DB), but test failures seem unrelated:

01040_dictionary_invalidate_query_switchover_long

01040_dictionary_invalidate_query_switchover_long is flaky #87587

test_storage_s3_queue/test_parallel_inserts.py::test_parallel_inserts_with_failures

Seems to fail regularly, though I don't see a ticket for it: CI DB

nihalzp

Thank you for working on it!

I have left few comments. Let me know if you have any questions.

src/Interpreters/executeQuery.cpp

src/IO/Progress.h

src/Server/HTTP/WriteBufferFromHTTPServerResponse.cpp

nihalzp

I missed it last time; I think we should update the relevant docs to let user know about this new feature.

Main doc:
https://clickhouse.com/docs/interfaces/http

Example usage can be updated:
https://clickhouse.com/docs/interfaces/formats/JSONEachRowWithProgress

cwurm · 2025-11-11T18:40:53Z

Good idea, adjusted docs for the HTTP interface. It's not implemented for JSONEachRowWithProgress - that one seems to be missing quite a few progress fields, e.g., it doesn't have result_rows/bytes either.

nihalzp · 2025-11-12T03:44:13Z

Stateless tests (amd_binary, old analyzer, s3 storage, DatabaseReplicated, parallel)

03712_json_advanced_shared_data_bug (failing in other places as well)

BuzzHouse (amd_debug) failure

(query: CREATE TABLE d0.`t9` (`c0` Int8, `c1` Int32, `c2` Time64(1) STATISTICS(TDigest, Uniq, CountMin) SETTINGS(min_compress_block_size = 4096), PROJECTION p0 (SELECT -`_table`.5 ORDER BY `c0`, `_table`.`Array(String)`), `c3` Int8 NOT NULL, `c4` JSON(max_dynamic_paths=787, max_dynamic_types=13, max_dynamic_paths=3), PROJECTION p1 (SELECT `_table`.`Int16`[2] ORDER BY `c4`.`Time`, 1, 5696996556135769442::Int8[`c1`] + `c1` IS NOT NULL, `c3`, `c1`), `c5` Array(Map(Int16,Date)) TTL (-bitTest(`c3`, (randomPrintableASCII(904), 208526174753267383, 16067192555130340929, 0.992))) SETTINGS(min_compress_block_size = 1)) ENGINE = ReplicatedSummingMergeTree('/clickhouse/tables/{shard}/{database}/{table}', '{replica}', (`c5`, `c0`)) PRIMARY KEY tuple() SETTINGS vertical_merge_algorithm_min_bytes_to_activate = 0, enable_block_number_column = 1, use_const_adaptive_granularity = 0, allow_floating_point_partition_key = 1, ttl_only_drop_parts = 0, add_minmax_index_for_numeric_columns = 1, allow_experimental_replacing_merge_with_cleanup = 1, enable_block_offset_column = 1, allow_nullable_key = 1, allow_experimental_reverse_key = 1, index_granularity = 1, add_minmax_index_for_string_columns = 1, enable_vertical_merge_algorithm = 1;)
Received exception from server (version 25.11.1):
Code: 170. DB::Exception: Received from localhost:9000. DB::Exception: Bad get: has UInt64, requested String. Stack trace:

Seems to be unrelated.

alexey-milovidov · 2025-11-16T00:56:54Z

Reverted.

nihalzp · 2025-11-16T12:00:38Z

@cwurm

2025-11-14 12:10:50 Reason: result differs with reference:  
2025-11-14 12:10:50 --- /home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/tests/queries/0_stateless/03652_memory_usage_headers.reference	2025-11-14 11:59:39.230008751 +1200
2025-11-14 12:10:50 +++ /home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/tests/queries/0_stateless/03652_memory_usage_headers.stdout	2025-11-14 12:10:50.064091955 +1200
2025-11-14 12:10:50 @@ -1,5 +1,4 @@
2025-11-14 12:10:50  Ok
2025-11-14 12:10:50  Ok
2025-11-14 12:10:50  Ok
2025-11-14 12:10:50  Ok
2025-11-14 12:10:50 -Ok
2025-11-14 12:10:50 
2025-11-14 12:10:50 
2025-11-14 12:10:50 /home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/tests/queries/0_stateless/test_hwlqv0x8/03652_memory_usage_headers.sh.debuglog:
2025-11-14 12:10:50 ++ [2025-11-14 12:08:44] [:15] curl -q -s --max-time 120 -s -S -v -N -G 'http://localhost:8123/?max_insert_threads=1&group_by_two_level_threshold=745246&group_by_two_level_threshold_bytes=2048036&distributed_aggregation_memory_efficient=0&fsync_metadata=0&output_format_parallel_formatting=0&input_format_parallel_parsing=0&min_chunk_bytes_for_parallel_parsing=12558342&max_read_buffer_size=561058&prefer_localhost_replica=1&max_block_size=38246&max_joined_block_size_rows=8297&joined_block_split_single_row=1&join_output_by_rowlist_perkey_rows_threshold=1&max_threads=1&optimize_append_index=0&use_hedged_requests=1&optimize_if_chain_to_multiif=1&optimize_if_transform_strings_to_enum=1&optimize_read_in_order=1&optimize_or_like_chain=1&optimize_substitute_columns=1&enable_multiple_prewhere_read_steps=1&read_in_order_two_level_merge_threshold=64&optimize_aggregation_in_order=1&aggregation_in_order_max_block_bytes=38398000&use_uncompressed_cache=1&min_bytes_to_use_direct_io=167273039&min_bytes_to_use_mmap_io=10737418240&local_filesystem_read_method=mmap&remote_filesystem_read_method=read&local_filesystem_read_prefetch=1&filesystem_cache_segments_batch_size=100&read_from_filesystem_cache_if_exists_otherwise_bypass_cache=1&throw_on_error_from_cache_on_write_operations=1&remote_filesystem_read_prefetch=0&allow_prefetched_read_pool_for_remote_filesystem=1&filesystem_prefetch_max_memory_usage=128Mi&filesystem_prefetches_limit=10&filesystem_prefetch_min_bytes_for_single_read_task=8Mi&filesystem_prefetch_step_marks=0&filesystem_prefetch_step_bytes=100Mi&compile_expressions=0&compile_aggregate_expressions=1&compile_sort_description=1&merge_tree_coarse_index_granularity=13&optimize_distinct_in_order=1&max_bytes_before_remerge_sort=2845981113&min_compress_block_size=1991919&max_compress_block_size=384771&merge_tree_compact_parts_min_granules_to_multibuffer_read=77&optimize_sorting_by_input_stream_properties=0&http_response_buffer_size=6670914&http_wait_end_of_query=False&enable_memory_bound_merging_of_aggregation_results=0&min_count_to_compile_expression=0&min_count_to_compile_aggregate_expression=3&min_count_to_compile_sort_description=3&session_timezone=Africa%2FJuba&use_page_cache_for_disks_without_file_cache=False&page_cache_inject_eviction=True&merge_tree_read_split_ranges_into_intersecting_and_non_intersecting_injection_probability=0.42&prefer_external_sort_block_bytes=0&cross_join_min_rows_to_compress=0&cross_join_min_bytes_to_compress=1&min_external_table_block_size_bytes=100000000&max_parsing_threads=0&optimize_functions_to_subcolumns=1&parallel_replicas_local_plan=1&query_plan_join_swap_table=false&enable_vertical_final=1&optimize_extract_common_expressions=1&use_async_executor_for_materialized_views=0&use_query_condition_cache=1&secondary_indices_enable_bulk_filtering=0&use_skip_indexes_if_final=1&use_skip_indexes_on_data_read=0&optimize_rewrite_like_perfect_affix=1&enable_lazy_columns_replication=0&allow_special_serialization_kinds_in_output_formats=1&max_bytes_before_external_sort=10737418240&max_bytes_before_external_group_by=10737418240&max_bytes_ratio_before_external_sort=0&max_bytes_ratio_before_external_group_by=0&use_skip_indexes_if_final_exact_mode=1&database=test_hwlqv0x8&log_comment=03652_memory_usage_headers.sh-test_hwlqv0x8' --data-urlencode 'query=SELECT number, avg(number) FROM numbers(1e7) GROUP BY number FORMAT Null' --data-urlencode cancel_http_readonly_queries_on_client_close=1 --data-urlencode send_progress_in_http_headers=1 --data-urlencode http_headers_progress_interval_ms=10
2025-11-14 12:10:50 + [2025-11-14 12:10:44] [:14] CURL_OUTPUT='*   Trying 127.0.0.1:8123...
2025-11-14 12:10:50 * Connected to localhost (127.0.0.1) port 8123 (#0)
2025-11-14 12:10:50 > GET /?max_insert_threads=1&group_by_two_level_threshold=745246&group_by_two_level_threshold_bytes=2048036&distributed_aggregation_memory_efficient=0&fsync_metadata=0&output_format_parallel_formatting=0&input_format_parallel_parsing=0&min_chunk_bytes_for_parallel_parsing=12558342&max_read_buffer_size=561058&prefer_localhost_replica=1&max_block_size=38246&max_joined_block_size_rows=8297&joined_block_split_single_row=1&join_output_by_rowlist_perkey_rows_threshold=1&max_threads=1&optimize_append_index=0&use_hedged_requests=1&optimize_if_chain_to_multiif=1&optimize_if_transform_strings_to_enum=1&optimize_read_in_order=1&optimize_or_like_chain=1&optimize_substitute_columns=1&enable_multiple_prewhere_read_steps=1&read_in_order_two_level_merge_threshold=64&optimize_aggregation_in_order=1&aggregation_in_order_max_block_bytes=38398000&use_uncompressed_cache=1&min_bytes_to_use_direct_io=167273039&min_bytes_to_use_mmap_io=10737418240&local_filesystem_read_method=mmap&remote_filesystem_read_method=read&local_filesystem_read_prefetch=1&filesystem_cache_segments_batch_size=100&read_from_filesystem_cache_if_exists_otherwise_bypass_cache=1&throw_on_error_from_cache_on_write_operations=1&remote_filesystem_read_prefetch=0&allow_prefetched_read_pool_for_remote_filesystem=1&filesystem_prefetch_max_memory_usage=128Mi&filesystem_prefetches_limit=10&filesystem_prefetch_min_bytes_for_single_read_task=8Mi&filesystem_prefetch_step_marks=0&filesystem_prefetch_step_bytes=100Mi&compile_expressions=0&compile_aggregate_expressions=1&compile_sort_description=1&merge_tree_coarse_index_granularity=13&optimize_distinct_in_order=1&max_bytes_before_remerge_sort=2845981113&min_compress_block_size=1991919&max_compress_block_size=384771&merge_tree_compact_parts_min_granules_to_multibuffer_read=77&optimize_sorting_by_input_stream_properties=0&http_response_buffer_size=6670914&http_wait_end_of_query=False&enable_memory_bound_merging_of_aggregation_results=0&min_count_to_compile_expression=0&min_count_to_compile_aggregate_expression=3&min_count_to_compile_sort_description=3&session_timezone=Africa%2FJuba&use_page_cache_for_disks_without_file_cache=False&page_cache_inject_eviction=True&merge_tree_read_split_ranges_into_intersecting_and_non_intersecting_injection_probability=0.42&prefer_external_sort_block_bytes=0&cross_join_min_rows_to_compress=0&cross_join_min_bytes_to_compress=1&min_external_table_block_size_bytes=100000000&max_parsing_threads=0&optimize_functions_to_subcolumns=1&parallel_replicas_local_plan=1&query_plan_join_swap_table=false&enable_vertical_final=1&optimize_extract_common_expressions=1&use_async_executor_for_materialized_views=0&use_query_condition_cache=1&secondary_indices_enable_bulk_filtering=0&use_skip_indexes_if_final=1&use_skip_indexes_on_data_read=0&optimize_rewrite_like_perfect_affix=1&enable_lazy_columns_replication=0&allow_special_serialization_kinds_in_output_formats=1&max_bytes_before_external_sort=10737418240&max_bytes_before_external_group_by=10737418240&max_bytes_ratio_before_external_sort=0&max_bytes_ratio_before_external_group_by=0&use_skip_indexes_if_final_exact_mode=1&database=test_hwlqv0x8&log_comment=03652_memory_usage_headers.sh-test_hwlqv0x8&query=SELECT+number%2C+avg%28number%29+FROM+numbers%281e7%29+GROUP+BY+number+FORMAT+Null&cancel_http_readonly_queries_on_client_close=1&send_progress_in_http_headers=1&http_headers_progress_interval_ms=10 HTTP/1.1
2025-11-14 12:10:50 > Host: localhost:8123
2025-11-14 12:10:50 > User-Agent: curl/7.81.0
2025-11-14 12:10:50 > Accept: */*
2025-11-14 12:10:50 > 
2025-11-14 12:10:50 * Mark bundle as not supporting multiuse
2025-11-14 12:10:50 < HTTP/1.1 200 OK
2025-11-14 12:10:50 < X-ClickHouse-Progress: {"total_rows_to_read":"10000000","elapsed_ns":"12510347","memory_usage":"2097247"}
2025-11-14 12:10:50 < X-ClickHouse-Progress: {"read_rows":"76492","read_bytes":"611936","total_rows_to_read":"10000000","elapsed_ns":"87610177","memory_usage":"7642351"}
2025-11-14 12:10:50 < X-ClickHouse-Progress: {"read_rows":"114738","read_bytes":"917904","total_rows_to_read":"10000000","elapsed_ns":"111929488","memory_usage":"8996895"}
2025-11-14 12:10:50 < X-ClickHouse-Progress: {"read_rows":"152984","read_bytes":"1223872","total_rows_to_read":"10000000","elapsed_ns":"140492365","memory_usage":"8996895"}
2025-11-14 12:10:50 < X-ClickHouse-Progress: {"read_rows":"191230","read_bytes":"1529840","total_rows_to_read":"10000000","elapsed_ns":"197598616","memory_usage":"23005215"}
2025-11-14 12:10:50 < X-ClickHouse-Progress: {"read_rows":"229476","read_bytes":"1835808","total_rows_to_read":"10000000","elapsed_ns":"225281837","memory_usage":"23005215"}
2025-11-14 12:10:50 < X-ClickHouse-Progress: {"read_rows":"267722","read_bytes":"2141776","total_rows_to_read":"10000000","elapsed_ns":"421035689","memory_usage":"23005215"}
2025-11-14 12:10:50 < X-ClickHouse-Progress: {"read_rows":"305968","read_bytes":"2447744","total_rows_to_read":"10000000","elapsed_ns":"515802269","memory_usage":"27871263"}
2025-11-14 12:10:50 < X-ClickHouse-Progress: {"read_rows":"344214","read_bytes":"2753712","total_rows_to_read":"10000000","elapsed_ns":"549548285","memory_usage":"27871263"}
2025-11-14 12:10:50 < X-ClickHouse-Progress: {"read_rows":"382460","read_bytes":"3059680","total_rows_to_read":"10000000","elapsed_ns":"578965258","memory_usage":"27871263"}
2025-11-14 12:10:50 < X-ClickHouse-Progress: {"read_rows":"420706","read_bytes":"3365648","total_rows_to_read":"10000000","elapsed_ns":"609271258","memory_usage":"27871263"}
2025-11-14 12:10:50 < X-ClickHouse-Progress: {"read_rows":"458952","read_bytes":"3671616","total_rows_to_read":"10000000","elapsed_ns":"642427444","memory_usage":"27871263"}
2025-11-14 12:10:50 < X-ClickHouse-Progress: {"read_rows":"497198","read_bytes":"3977584","total_rows_to_read":"10000000","elapsed_ns":"674560988","memory_usage":"27871263"}
2025-11-14 12:10:50 < X-ClickHouse-Progress: {"read_rows":"535444","read_bytes":"4283552","total_rows_to_read":"10000000","elapsed_ns":"707738243","memory_usage":"27871263"}
2025-11-14 12:10:50 < X-ClickHouse-Progress: {"read_rows":"573690","read_bytes":"4589520","total_rows_to_read":"10000000","elapsed_ns":"1303789250","memory_usage":"85870623"}
2025-11-14 12:10:50 < X-ClickHouse-Progress: {"read_rows":"611936","read_bytes":"4895488","total_rows_to_read":"10000000","elapsed_ns":"1334663705","memory_usage":"85870623"}
2025-11-14 12:10:50 < X-ClickHouse-Progress: {"read_rows":"650182","read_bytes":"5201456","total_rows_to_read":"10000000","elapsed_ns":"1365836392","memory_usage":"85870623"}
2025-11-14 12:10:50 < X-ClickHouse-Progress: {"read_rows":"688428","read_bytes":"5507424","total_rows_to_read":"10000000","elapsed_ns":"1397145788","memory_usage":"85870623"}
2025-11-14 12:10:50 < X-ClickHouse-Progress: {"read_rows":"726674","read_bytes":"5813392","total_rows_to_read":"10000000","elapsed_ns":"1425409445","memory_usage":"85870623"}
2025-11-14 12:10:50 < X-ClickHouse-Progress: {"read_rows":"764920","read_bytes":"6119360","total_rows_to_read":"10000000","elapsed_ns":"1452220933","memory_usage":"85870623"}
2025-11-14 12:10:50 < X-ClickHouse-Progress: {"read_rows":"803166","read_bytes":"6425328","total_rows_to_read":"10000000","elapsed_ns":"1478605660","memory_usage":"85870623"}
2025-11-14 12:10:50 < X-ClickHouse-Progress: {"read_rows":"841412","read_bytes":"6731296","total_rows_to_read":"10000000","elapsed_ns":"1502493263","memory_usage":"85870623"}
2025-11-14 12:10:50 < X-ClickHouse-Progress: {"read_rows":"879658","read_bytes":"7037264","total_rows_to_read":"10000000","elapsed_ns":"1524796951","memory_usage":"85870623"}
2025-11-14 12:10:50 < X-ClickHouse-Progress: {"read_rows":"917904","read_bytes":"7343232","total_rows_to_read":"10000000","elapsed_ns":"1549844452","memory_usage":"85870623"}
2025-11-14 12:10:50 < X-ClickHouse-Progress: {"read_rows":"956150","read_bytes":"7649200","total_rows_to_read":"10000000","elapsed_ns":"1579275688","memory_usage":"85870623"}
2025-11-14 12:10:50 < X-ClickHouse-Progress: {"read_rows":"994396","read_bytes":"7955168","total_rows_to_read":"10000000","elapsed_ns":"1608929591","memory_usage":"85870623"}
2025-11-14 12:10:50 < X-ClickHouse-Progress: {"read_rows":"1032642","read_bytes":"8261136","total_rows_to_read":"10000000","elapsed_ns":"1639965197","memory_usage":"85870623"}
2025-11-14 12:10:50 < X-ClickHouse-Progress: {"read_rows":"1070888","read_bytes":"8567104","total_rows_to_read":"10000000","elapsed_ns":"1670134392","memory_usage":"85870623"}
2025-11-14 12:10:50 < X-ClickHouse-Progress: {"read_rows":"1109134","read_bytes":"8873072","total_rows_to_read":"10000000","elapsed_ns":"2112374525","memory_usage":"103368735"}
2025-11-14 12:10:50 < X-ClickHouse-Progress: {"read_rows":"1147380","read_bytes":"9179040","total_rows_to_read":"10000000","elapsed_ns":"2279464494","memory_usage":"103368735"}
2025-11-14 12:10:50 < X-ClickHouse-Progress: {"read_rows":"1185626","read_bytes":"9485008","total_rows_to_read":"10000000","elapsed_ns":"2305209877","memory_usage":"103368735"}
2025-11-14 12:10:50 < X-ClickHouse-Progress: {"read_rows":"1223872","read_bytes":"9790976","total_rows_to_read":"10000000","elapsed_ns":"2330681023","memory_usage":"103368735"}
2025-11-14 12:10:50 < X-ClickHouse-Progress: {"read_rows":"1262118","read_bytes":"10096944","total_rows_to_read":"10000000","elapsed_ns":"2353760065","memory_usage":"103368735"}
2025-11-14 12:10:50 < X-ClickHouse-Progress: {"read_rows":"1300364","read_bytes":"10402912","total_rows_to_read":"10000000","elapsed_ns":"2377877224","memory_usage":"103368735"}
2025-11-14 12:10:50 < X-ClickHouse-Progress: {"read_rows":"1338610","read_bytes":"10708880","total_rows_to_read":"10000000","elapsed_ns":"2408652752","memory_usage":"103368735"}
2025-11-14 12:10:50 < X-ClickHouse-Progress: {"read_rows":"1376856","read_bytes":"11014848","total_rows_to_read":"10000000","elapsed_ns":"2439694662","memory_usage":"103368735"}
2025-11-14 12:10:50 < X-ClickHouse-Progress: {"read_rows":"1415102","read_bytes":"11320816","total_rows_to_read":"10000000","elapsed_ns":"2473918658","memory_usage":"103368735"}
2025-11-14 12:10:50 < X-ClickHouse-Progress: {"read_rows":"1453348","read_bytes":"11626784","total_rows_to_read":"10000000","elapsed_ns":"2505194566","memory_usage":"103368735"}
2025-11-14 12:10:50 < X-ClickHouse-Progress: {"read_rows":"1491594","read_bytes":"11932752","total_rows_to_read":"10000000","elapsed_ns":"2533361917","memory_usage":"103368735"}
2025-11-14 12:10:50 ----------------------------------------1205 lines are hidden----------------------------------------
2025-11-14 12:10:50 < X-ClickHouse-Timezone: Africa/Juba
2025-11-14 12:10:50 < Keep-Alive: timeout=10, max=9999
2025-11-14 12:10:50 < X-ClickHouse-Exception-Tag: ekmczubmrbeyjouj
2025-11-14 12:10:50 < 
2025-11-14 12:10:50 { [233 bytes data]
2025-11-14 12:10:50 Code: 241. DB::Exception: Query memory limit exceeded: would use 98.58 MiB (attempt to allocate chunk of 16.69 MiB), maximum: 95.37 MiB: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED) (version 25.11.1.2682)
2025-11-14 12:10:50 * Connection #0 to host localhost left intact'
2025-11-14 12:10:50 + [2025-11-14 12:10:50] [:36] grep -q '"memory_usage":"[1-9][0-9]*"'
2025-11-14 12:10:50 + [2025-11-14 12:10:50] [:36] echo Ok
2025-11-14 12:10:50 + [2025-11-14 12:10:50] [:37] grep -q 'X-ClickHouse-Exception-Code: 241'
2025-11-14 12:10:50 + [2025-11-14 12:10:50] [:37] echo '*   Trying 127.0.0.1:8123...
2025-11-14 12:10:50 * Connected to localhost (127.0.0.1) port 8123 (#0)
2025-11-14 12:10:50 > GET /?max_insert_threads=1&group_by_two_level_threshold=745246&group_by_two_level_threshold_bytes=2048036&distributed_aggregation_memory_efficient=0&fsync_metadata=0&output_format_parallel_formatting=0&input_format_parallel_parsing=0&min_chunk_bytes_for_parallel_parsing=12558342&max_read_buffer_size=561058&prefer_lo

cwurm · 2025-11-16T12:48:10Z

Thanks, test failures are caused by Broken pipe errors that are almost certainly unrelated to the code. Trying to fix the test in #90142.

cwurm added 2 commits October 10, 2025 22:20

Add memory_usage to HTTP progress and summary headers

d4e27db

Small changes and test

4e72d9d

clickhouse-gh bot added the pr-feature Pull request with new product feature label Oct 12, 2025

cwurm added 2 commits October 17, 2025 20:18

Try fix CI

15fa2a6

Try fix CI

7fb4844

cwurm marked this pull request as ready for review October 20, 2025 11:55

nihalzp self-assigned this Oct 30, 2025

nihalzp requested changes Nov 1, 2025

View reviewed changes

src/Interpreters/executeQuery.cpp Outdated Show resolved Hide resolved

src/IO/Progress.h Outdated Show resolved Hide resolved

src/Server/HTTP/WriteBufferFromHTTPServerResponse.cpp Outdated Show resolved Hide resolved

cwurm added 2 commits November 9, 2025 15:15

Address review

386cecd

Merge remote-tracking branch 'origin/master' into progress_memory_usage

c1ff6a3

nihalzp approved these changes Nov 10, 2025

View reviewed changes

nihalzp reviewed Nov 10, 2025

View reviewed changes

Docs

f9e195a

nihalzp added this pull request to the merge queue Nov 12, 2025

Merged via the queue into ClickHouse:master with commit 63abb20 Nov 12, 2025
250 of 256 checks passed

robot-clickhouse-ci-1 added the pr-synced-to-cloud The PR is synced to the cloud repo label Nov 12, 2025

alexey-milovidov mentioned this pull request Nov 16, 2025

Revert "Add memory usage to HTTP progress and summary headers" #90136

Merged

cwurm mentioned this pull request Nov 16, 2025

Improve 03652_memory_usage_headers #90142

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add memory usage to HTTP progress and summary headers#88393

Add memory usage to HTTP progress and summary headers#88393
nihalzp merged 7 commits intoClickHouse:masterfrom
cwurm:progress_memory_usage

cwurm commented Oct 12, 2025 •

edited by clickhouse-gh bot

Loading

Uh oh!

clickhouse-gh bot commented Oct 12, 2025 •

edited

Loading

Uh oh!

cwurm commented Oct 20, 2025

Uh oh!

nihalzp left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nihalzp left a comment

Uh oh!

cwurm commented Nov 11, 2025

Uh oh!

nihalzp commented Nov 12, 2025

Uh oh!

Uh oh!

alexey-milovidov commented Nov 16, 2025

Uh oh!

nihalzp commented Nov 16, 2025

Uh oh!

cwurm commented Nov 16, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

cwurm commented Oct 12, 2025 • edited by clickhouse-gh bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Details

Uh oh!

clickhouse-gh bot commented Oct 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cwurm commented Oct 20, 2025

Uh oh!

nihalzp left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nihalzp left a comment

Choose a reason for hiding this comment

Uh oh!

cwurm commented Nov 11, 2025

Uh oh!

nihalzp commented Nov 12, 2025

Uh oh!

Uh oh!

alexey-milovidov commented Nov 16, 2025

Uh oh!

nihalzp commented Nov 16, 2025

Uh oh!

cwurm commented Nov 16, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

cwurm commented Oct 12, 2025 •

edited by clickhouse-gh bot

Loading

clickhouse-gh bot commented Oct 12, 2025 •

edited

Loading