-
Notifications
You must be signed in to change notification settings - Fork 8.3k
Segmentation fault on system.tables #77177
Copy link
Copy link
Closed
Labels
crashCrash / segfault / abortCrash / segfault / abortjsonIssues and pull-requests related to the new JSON data typeIssues and pull-requests related to the new JSON data typepotential bugTo be reviewed by developers and confirmed/rejected.To be reviewed by developers and confirmed/rejected.
Description
Company or project name
No response
Describe what's wrong
While using the grafana datasource for inspecting the state of my clickhouse cluster (clickhouse-data-analysis dashboard), my server suddenly crashed with the following segfatult error:
2025.03.05 15:28:40.489046 [ 856 ] {} <Fatal> BaseDaemon: ########## Short fault info ############
2025.03.05 15:28:40.489123 [ 856 ] {} <Fatal> BaseDaemon: (version 25.2.1.3085 (official build), build id: 1A3F7CA3FA57AA5910DEB2DA4440126BF05F69CD, git hash: 09989205d6fd661fb2683cbb1390fe4fcedaa548, architecture: x86_64) (from thread 848) Received signal 11
2025.03.05 15:28:40.489138 [ 856 ] {} <Fatal> BaseDaemon: Signal description: Segmentation fault
2025.03.05 15:28:40.489150 [ 856 ] {} <Fatal> BaseDaemon: Address: NULL pointer. Access: read. Address not mapped to object.
2025.03.05 15:28:40.489164 [ 856 ] {} <Fatal> BaseDaemon: Stack trace: 0x000000000f2324b7 0x00007b0730fbe050 0x00007b07310dcef6 0x000000001413599d 0x0000000014135c2e 0x00000000122b86ff 0x00000000122f2c43 0x0000000014134194 0x0000000014134a8c 0x0000000014134403 0x0000000014131ae7 0x0000000013e99b92 0x0000000014036360 0x0000000013e95f13 0x0000000013e865a1 0x0000000013e96c56 0x0000000010e7c857 0x0000000010e867b2 0x0000000014afe5ad 0x0000000014b2de00 0x0000000012f5b89d 0x0000000012f5b17e 0x0000000013309d82 0x0000000013304f44 0x000000001462d8ec 0x000000001464bd79 0x0000000017cfe707 0x0000000017cfeb59 0x0000000017ccaedb 0x0000000017cc93bd 0x00007b073100b1c4 0x00007b073108aac0
2025.03.05 15:28:40.489183 [ 856 ] {} <Fatal> BaseDaemon: ########################################
2025.03.05 15:28:40.489312 [ 856 ] {} <Fatal> BaseDaemon: (version 25.2.1.3085 (official build), build id: 1A3F7CA3FA57AA5910DEB2DA4440126BF05F69CD, git hash: 09989205d6fd661fb2683cbb1390fe4fcedaa548) (from thread 848) (query_id: b1c794b0-ff1a-4d45-8695-64bde2df936a) (query: SELECT concatAssumeInjective(table.database, '.', name) as name,
table_stats.total_rows as total_rows
FROM system.tables table
LEFT JOIN ( SELECT table,
database,
sum(rows) as total_rows
FROM system.parts
WHERE table IN ('CHARACTER_SETS','COLLATIONS','COLUMNS','ENGINES','KEY_COLUMN_USAGE','REFERENTIAL_CONSTRAINTS','SCHEMATA','STATISTICS','TABLES','VIEWS','character_sets','collations','columns','engines','key_column_usage','referential_constraints','schemata','statistics','tables','views','account_spends','campaigns_spend','client_rules','clients','providers','fingerprints','fingerprints_v2','table_name','ws_chat','aggregate_function_combinators','asynchronous_insert_log','asynchronous_inserts','asynchronous_loader','asynchronous_metric_log','asynchronous_metrics','azure_queue','azure_queue_settings','backups','build_options','certificates','clusters','contributors','crash_log','current_roles','dashboards','data_skipping_indices','data_type_families','database_engines','databases','detached_parts','detached_tables','dictionaries','disks','distributed_ddl_queue','distribution_queue','dns_cache','dropped_tables','dropped_tables_parts','enabled_roles','error_log','error_log_0','errors','events','filesystem_cache','filesystem_cache_settings','formats','functions','grants','graphite_retentions','jemalloc_bins','kafka_consumers','keywords','latency_buckets','latency_log','licenses','macros','merge_tree_settings','merges','metric_log','metric_log_0','metric_log_1','metric_log_2','metrics','models','moves','mutations','named_collections','numbers','numbers_mt','one','part_log','part_log_0','part_log_1','part_moves_between_shards','parts','parts_columns','privileges','processes','processors_profile_log','processors_profile_log_0','processors_profile_log_1','projection_parts','projection_parts_columns','projections','query_cache','query_log','query_log_0','query_metric_log','query_metric_log_0','quota_limits','quota_usage','quotas','quotas_usage','remote_data_paths','replicas','replicated_fetches','replicated_merge_tree_settings','replication_queue','resources','rocksdb','role_grants','roles','row_policies','s3_queue_settings','s3queue','scheduler','schema_inference_cache','server_settings','settings','settings_changes','settings_profile_elements','settings_profiles','stack_trace','storage_policies','symbols','table_engines','table_functions','text_log','time_zones','trace_log','trace_log_0','trace_log_1','trace_log_2','user_directories','user_processes','users','view_refreshes','warnings','workloads','zeros','zeros_mt','zookeeper','zookeeper_connection') AND active AND database IN ('INFORMATION_SCHEMA','adszero','default','information_schema','system')
GROUP BY table, database
) AS table_stats ON table.name = table_stats.table AND table.database = table_stats.database
ORDER BY total_rows DESC
LIMIT 10) Received signal Segmentation fault (11)
2025.03.05 15:28:40.489417 [ 856 ] {} <Fatal> BaseDaemon: Address: NULL pointer. Access: read. Address not mapped to object.
2025.03.05 15:28:40.489502 [ 856 ] {} <Fatal> BaseDaemon: Stack trace: 0x000000000f2324b7 0x00007b0730fbe050 0x00007b07310dcef6 0x000000001413599d 0x0000000014135c2e 0x00000000122b86ff 0x00000000122f2c43 0x0000000014134194 0x0000000014134a8c 0x0000000014134403 0x0000000014131ae7 0x0000000013e99b92 0x0000000014036360 0x0000000013e95f13 0x0000000013e865a1 0x0000000013e96c56 0x0000000010e7c857 0x0000000010e867b2 0x0000000014afe5ad 0x0000000014b2de00 0x0000000012f5b89d 0x0000000012f5b17e 0x0000000013309d82 0x0000000013304f44 0x000000001462d8ec 0x000000001464bd79 0x0000000017cfe707 0x0000000017cfeb59 0x0000000017ccaedb 0x0000000017cc93bd 0x00007b073100b1c4 0x00007b073108aac0
2025.03.05 15:28:40.489634 [ 856 ] {} <Fatal> BaseDaemon: 0. signalHandler(int, siginfo_t*, void*) @ 0x000000000f2324b7
2025.03.05 15:28:40.489683 [ 856 ] {} <Fatal> BaseDaemon: 1. ? @ 0x00007b0730fbe050
2025.03.05 15:28:40.489722 [ 856 ] {} <Fatal> BaseDaemon: 2. ? @ 0x00007b07310dcef6
2025.03.05 15:28:40.489807 [ 856 ] {} <Fatal> BaseDaemon: 3. std::map<String, std::unique_ptr<DB::MergeTreeReaderStream, std::default_delete<DB::MergeTreeReaderStream>>, std::less<String>, std::allocator<std::pair<String const, std::unique_ptr<DB::MergeTreeReaderStream, std::default_delete<DB::MergeTreeReaderStream>>>>>::contains[abi:ne190107](String const&) const @ 0x000000001413599d
2025.03.05 15:28:40.489923 [ 856 ] {} <Fatal> BaseDaemon: 4. void std::__function::__policy_invoker<void (DB::ISerialization::SubstreamPath const&)>::__call_impl[abi:ne190107]<std::__function::__default_alloc_func<DB::MergeTreeReaderWide::deserializePrefix(std::shared_ptr<DB::ISerialization const> const&, DB::NameAndTypePair const&, unsigned long, std::unordered_map<String, std::shared_ptr<DB::ISerialization::DeserializeBinaryBulkState>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, std::shared_ptr<DB::ISerialization::DeserializeBinaryBulkState>>>>&, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&, std::unordered_map<String, std::shared_ptr<DB::ISerialization::DeserializeBinaryBulkState>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, std::shared_ptr<DB::ISerialization::DeserializeBinaryBulkState>>>>&, std::function<void (DB::ISerialization::SubstreamPath const&)>)::$_1, void (DB::ISerialization::SubstreamPath const&)>>(std::__function::__policy_storage const*, DB::ISerialization::SubstreamPath const&) @ 0x0000000014135c2e
2025.03.05 15:28:40.489995 [ 856 ] {} <Fatal> BaseDaemon: 5. DB::SerializationDynamic::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization::SubstreamData const&) const @ 0x00000000122b86ff
2025.03.05 15:28:40.490078 [ 856 ] {} <Fatal> BaseDaemon: 6. DB::SerializationObject::deserializeBinaryBulkStatePrefix(DB::ISerialization::DeserializeBinaryBulkSettings&, std::shared_ptr<DB::ISerialization::DeserializeBinaryBulkState>&, std::unordered_map<String, std::shared_ptr<DB::ISerialization::DeserializeBinaryBulkState>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, std::shared_ptr<DB::ISerialization::DeserializeBinaryBulkState>>>>*) const @ 0x00000000122f2c43
2025.03.05 15:28:40.490234 [ 856 ] {} <Fatal> BaseDaemon: 7. DB::MergeTreeReaderWide::deserializePrefix(std::shared_ptr<DB::ISerialization const> const&, DB::NameAndTypePair const&, unsigned long, std::unordered_map<String, std::shared_ptr<DB::ISerialization::DeserializeBinaryBulkState>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, std::shared_ptr<DB::ISerialization::DeserializeBinaryBulkState>>>>&, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&, std::unordered_map<String, std::shared_ptr<DB::ISerialization::DeserializeBinaryBulkState>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, std::shared_ptr<DB::ISerialization::DeserializeBinaryBulkState>>>>&, std::function<void (DB::ISerialization::SubstreamPath const&)>) @ 0x0000000014134194
2025.03.05 15:28:40.490312 [ 856 ] {} <Fatal> BaseDaemon: 8. DB::MergeTreeReaderWide::deserializePrefixForAllColumnsImpl(unsigned long, unsigned long, std::function<std::function<void (DB::ISerialization::SubstreamPath const&)> (DB::NameAndTypePair const&)>)::$_0::operator()() const @ 0x0000000014134a8c
2025.03.05 15:28:40.490383 [ 856 ] {} <Fatal> BaseDaemon: 9. DB::MergeTreeReaderWide::deserializePrefixForAllColumnsImpl(unsigned long, unsigned long, std::function<std::function<void (DB::ISerialization::SubstreamPath const&)> (DB::NameAndTypePair const&)>) @ 0x0000000014134403
2025.03.05 15:28:40.490450 [ 856 ] {} <Fatal> BaseDaemon: 10. DB::MergeTreeReaderWide::readRows(unsigned long, unsigned long, bool, unsigned long, std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>&) @ 0x0000000014131ae7
2025.03.05 15:28:40.490520 [ 856 ] {} <Fatal> BaseDaemon: 11. DB::IMergeTreeDataPart::getColumnSample(DB::NameAndTypePair const&) const @ 0x0000000013e99b92
2025.03.05 15:28:40.490582 [ 856 ] {} <Fatal> BaseDaemon: 12. DB::MergeTreeDataPartWide::calculateEachColumnSizes(std::unordered_map<String, DB::ColumnSize, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, DB::ColumnSize>>>&, DB::ColumnSize&, std::optional<DB::Block>) const @ 0x0000000014036360
2025.03.05 15:28:40.490644 [ 856 ] {} <Fatal> BaseDaemon: 13. DB::IMergeTreeDataPart::calculateColumnsSizesOnDisk(std::optional<DB::Block>) const @ 0x0000000013e95f13
2025.03.05 15:28:40.490708 [ 856 ] {} <Fatal> BaseDaemon: 14. DB::IMergeTreeDataPart::calculateColumnsAndSecondaryIndicesSizesOnDisk(std::optional<DB::Block>) const @ 0x0000000013e865a1
2025.03.05 15:28:40.490770 [ 856 ] {} <Fatal> BaseDaemon: 15. DB::IMergeTreeDataPart::getTotalColumnsSize() const @ 0x0000000013e96c56
2025.03.05 15:28:40.490844 [ 856 ] {} <Fatal> BaseDaemon: 16. DB::StorageSystemParts::processNextStorage(std::shared_ptr<DB::Context const>, std::vector<COW<DB::IColumn>::mutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::mutable_ptr<DB::IColumn>>>&, std::vector<char8_t, std::allocator<char8_t>>&, DB::StoragesInfo const&, bool) @ 0x0000000010e7c857
2025.03.05 15:28:40.490904 [ 856 ] {} <Fatal> BaseDaemon: 17. DB::ReadFromSystemPartsBase::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x0000000010e867b2
2025.03.05 15:28:40.490965 [ 856 ] {} <Fatal> BaseDaemon: 18. DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x0000000014afe5ad
2025.03.05 15:28:40.491033 [ 856 ] {} <Fatal> BaseDaemon: 19. DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x0000000014b2de00
2025.03.05 15:28:40.491128 [ 856 ] {} <Fatal> BaseDaemon: 20. DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() @ 0x0000000012f5b89d
2025.03.05 15:28:40.491193 [ 856 ] {} <Fatal> BaseDaemon: 21. DB::InterpreterSelectQueryAnalyzer::execute() @ 0x0000000012f5b17e
2025.03.05 15:28:40.491261 [ 856 ] {} <Fatal> BaseDaemon: 22. DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*, std::shared_ptr<DB::IAST>&) @ 0x0000000013309d82
2025.03.05 15:28:40.491329 [ 856 ] {} <Fatal> BaseDaemon: 23. DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x0000000013304f44
2025.03.05 15:28:40.491397 [ 856 ] {} <Fatal> BaseDaemon: 24. DB::TCPHandler::runImpl() @ 0x000000001462d8ec
2025.03.05 15:28:40.491452 [ 856 ] {} <Fatal> BaseDaemon: 25. DB::TCPHandler::run() @ 0x000000001464bd79
2025.03.05 15:28:40.491512 [ 856 ] {} <Fatal> BaseDaemon: 26. Poco::Net::TCPServerConnection::start() @ 0x0000000017cfe707
2025.03.05 15:28:40.491558 [ 856 ] {} <Fatal> BaseDaemon: 27. Poco::Net::TCPServerDispatcher::run() @ 0x0000000017cfeb59
2025.03.05 15:28:40.491616 [ 856 ] {} <Fatal> BaseDaemon: 28. Poco::PooledThread::run() @ 0x0000000017ccaedb
2025.03.05 15:28:40.491675 [ 856 ] {} <Fatal> BaseDaemon: 29. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000017cc93bd
2025.03.05 15:28:40.491725 [ 856 ] {} <Fatal> BaseDaemon: 30. ? @ 0x00007b073100b1c4
2025.03.05 15:28:40.491778 [ 856 ] {} <Fatal> BaseDaemon: 31. ? @ 0x00007b073108aac0
2025.03.05 15:28:40.808409 [ 856 ] {} <Fatal> BaseDaemon: Integrity check of the executable successfully passed (checksum: 22606E44C93DF5D27026808E5D6560BA)
2025.03.05 15:28:40.808470 [ 856 ] {} <Fatal> BaseDaemon: Report this error to https://github.com/ClickHouse/ClickHouse/issues
2025.03.05 15:28:40.808645 [ 856 ] {} <Fatal> BaseDaemon: Changed settings: max_execution_time = 124.
2025.03.05 15:28:44.762213 [ 1 ] {} <Fatal> Application: Child process was terminated by signal 11.
Does it reproduce on the most recent release?
Yes
How to reproduce
execute the following query:
SELECT concatAssumeInjective(table.database, '.', name) as name,
table_stats.total_rows as total_rows
FROM system.tables table
LEFT JOIN ( SELECT table,
database,
sum(rows) as total_rows
FROM system.parts
WHERE table IN ('CHARACTER_SETS','COLLATIONS','COLUMNS','ENGINES','KEY_COLUMN_USAGE','REFERENTIAL_CONSTRAINTS','SCHEMATA','STATISTICS','TABLES','VIEWS','character_sets','collations','columns','engines','key_column_usage','referential_constraints','schemata','statistics','tables','views','account_spends','campaigns_spend','client_rules','clients','providers','fingerprints','fingerprints_v2','table_name','ws_chat','aggregate_function_combinators','asynchronous_insert_log','asynchronous_inserts','asynchronous_loader','asynchronous_metric_log','asynchronous_metrics','azure_queue','azure_queue_settings','backups','build_options','certificates','clusters','contributors','crash_log','current_roles','dashboards','data_skipping_indices','data_type_families','database_engines','databases','detached_parts','detached_tables','dictionaries','disks','distributed_ddl_queue','distribution_queue','dns_cache','dropped_tables','dropped_tables_parts','enabled_roles','error_log','error_log_0','errors','events','filesystem_cache','filesystem_cache_settings','formats','functions','grants','graphite_retentions','jemalloc_bins','kafka_consumers','keywords','latency_buckets','latency_log','licenses','macros','merge_tree_settings','merges','metric_log','metric_log_0','metric_log_1','metric_log_2','metrics','models','moves','mutations','named_collections','numbers','numbers_mt','one','part_log','part_log_0','part_log_1','part_moves_between_shards','parts','parts_columns','privileges','processes','processors_profile_log','processors_profile_log_0','processors_profile_log_1','projection_parts','projection_parts_columns','projections','query_cache','query_log','query_log_0','query_metric_log','query_metric_log_0','quota_limits','quota_usage','quotas','quotas_usage','remote_data_paths','replicas','replicated_fetches','replicated_merge_tree_settings','replication_queue','resources','rocksdb','role_grants','roles','row_policies','s3_queue_settings','s3queue','scheduler','schema_inference_cache','server_settings','settings','settings_changes','settings_profile_elements','settings_profiles','stack_trace','storage_policies','symbols','table_engines','table_functions','text_log','time_zones','trace_log','trace_log_0','trace_log_1','trace_log_2','user_directories','user_processes','users','view_refreshes','warnings','workloads','zeros','zeros_mt','zookeeper','zookeeper_connection') AND active AND database IN ('INFORMATION_SCHEMA','adszero','default','information_schema','system')
GROUP BY table, database
) AS table_stats ON table.name = table_stats.table AND table.database = table_stats.database
ORDER BY total_rows DESC
LIMIT 10
Expected behavior
No segmentation fault
Error message and/or stacktrace
No response
Additional context
clickhouse version: 25.2.1.3085
additional settings:
<asynchronous_metric_log remove="1"/>
<backup_log remove="1"/>
<metric_log remove="1"/>
<query_thread_log remove="1" />
<session_log remove="1"/>
<text_log remove="1" />
<trace_log remove="1"/>
<crash_log remove="1"/>
<opentelemetry_span_log remove="1"/>
<processors_profile_log remove="1"/>
cluster of 1 shard 1 replica (testing env) deployed trough helm chart
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
crashCrash / segfault / abortCrash / segfault / abortjsonIssues and pull-requests related to the new JSON data typeIssues and pull-requests related to the new JSON data typepotential bugTo be reviewed by developers and confirmed/rejected.To be reviewed by developers and confirmed/rejected.