Skip to content

min_bytes_to_use_direct_io > 0 can lead to "Cannot read all array values" or even segfault #65690

Closed
#79840
Closed
min_bytes_to_use_direct_io > 0 can lead to "Cannot read all array values" or even segfault#65690
#79840
Assignees
alexey-milovidov
Labels
bugConfirmed user-visible misbehaviour in official releasecrashCrash / segfault / abort
@al13n321

Description

@al13n321
al13n321
opened on Jun 25, 2024

https://s3.amazonaws.com/clickhouse-test-reports/64951/ea403eec0a8251c92f35f12a704108f3138dca1c/stateless_tests__release_.html

2024-06-21 19:58:31 03038_nested_dynamic_merges:                                            [ FAIL ] 8.68 sec. - having stderror: 
2024-06-21 19:58:31 [68ddc39586b9] 2024.06.21 16:58:30.177091 [ 8084 ] {4a074eed-f525-49b5-9a12-9ed3445f18f4} <Error> executeQuery: Code: 33. DB::Exception: Cannot read all array values: read just 1797 of 1935: (while reading column d): (while reading from part /var/lib/clickhouse/store/ff7/ff761762-88a0-4170-907f-f5766f50f7ba/all_1_3_1/ in table test_z5oi1ytj.test (ff761762-88a0-4170-907f-f5766f50f7ba) located on disk default of type local, from mark 88 with max_rows_to_read = 1940): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_READ_ALL_DATA) (version 24.7.1.80) (from [::1]:47102) (in query: select count(), dynamicType(d) || ':' || dynamicType(d.`Tuple(a Dynamic(max_types=3))`.a) as type from test group by type order by count(), type), Stack trace (when copying this message, always include the lines below):
2024-06-21 19:58:31 
2024-06-21 19:58:31 0. ./build_docker/./src/Common/Exception.cpp:101: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000cfc8d5b
2024-06-21 19:58:31 1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x00000000078c7e2c
2024-06-21 19:58:31 2. DB::Exception::Exception<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x00000000078c9ccb
2024-06-21 19:58:31 3. ./build_docker/./src/DataTypes/Serializations/SerializationArray.cpp:0: DB::SerializationArray::deserializeBinaryBulkWithMultipleStreams(COW<DB::IColumn>::immutable_ptr<DB::IColumn>&, unsigned long, DB::ISerialization::DeserializeBinaryBulkSettings&, std::shared_ptr<DB::ISerialization::DeserializeBinaryBulkState>&, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>*) const @ 0x000000001096c1f9
2024-06-21 19:58:31 4. ./build_docker/./src/DataTypes/Serializations/SerializationVariant.cpp:0: DB::SerializationVariant::deserializeBinaryBulkWithMultipleStreams(COW<DB::IColumn>::immutable_ptr<DB::IColumn>&, unsigned long, DB::ISerialization::DeserializeBinaryBulkSettings&, std::shared_ptr<DB::ISerialization::DeserializeBinaryBulkState>&, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>*) const @ 0x00000000109c835f
2024-06-21 19:58:31 5. ./build_docker/./src/DataTypes/Serializations/SerializationDynamic.cpp:0: DB::SerializationDynamic::deserializeBinaryBulkWithMultipleStreams(COW<DB::IColumn>::immutable_ptr<DB::IColumn>&, unsigned long, DB::ISerialization::DeserializeBinaryBulkSettings&, std::shared_ptr<DB::ISerialization::DeserializeBinaryBulkState>&, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>*) const @ 0x000000001097fdf0
2024-06-21 19:58:31 6. ./src/DataTypes/Serializations/SerializationNamed.cpp:0: DB::SerializationTuple::deserializeBinaryBulkWithMultipleStreams(COW<DB::IColumn>::immutable_ptr<DB::IColumn>&, unsigned long, DB::ISerialization::DeserializeBinaryBulkSettings&, std::shared_ptr<DB::ISerialization::DeserializeBinaryBulkState>&, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>*) const @ 0x00000000109c1227
2024-06-21 19:58:31 7. ./build_docker/./src/DataTypes/Serializations/SerializationVariant.cpp:0: DB::SerializationVariant::deserializeBinaryBulkWithMultipleStreams(COW<DB::IColumn>::immutable_ptr<DB::IColumn>&, unsigned long, DB::ISerialization::DeserializeBinaryBulkSettings&, std::shared_ptr<DB::ISerialization::DeserializeBinaryBulkState>&, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>*) const @ 0x00000000109c835f
2024-06-21 19:58:31 8. ./build_docker/./src/DataTypes/Serializations/SerializationDynamic.cpp:0: DB::SerializationDynamic::deserializeBinaryBulkWithMultipleStreams(COW<DB::IColumn>::immutable_ptr<DB::IColumn>&, unsigned long, DB::ISerialization::DeserializeBinaryBulkSettings&, std::shared_ptr<DB::ISerialization::DeserializeBinaryBulkState>&, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>*) const @ 0x000000001097fdf0
2024-06-21 19:58:31 9. ./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:196: DB::MergeTreeReaderWide::readRows(unsigned long, unsigned long, bool, unsigned long, std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>&) @ 0x0000000012666dd3
2024-06-21 19:58:31 10. ./build_docker/./src/Storages/MergeTree/MergeTreeRangeReader.cpp:118: DB::MergeTreeRangeReader::DelayedStream::finalize(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>&) @ 0x000000001264eac2
2024-06-21 19:58:31 11. ./build_docker/./src/Storages/MergeTree/MergeTreeRangeReader.cpp:0: DB::MergeTreeRangeReader::startReadingChain(unsigned long, DB::MarkRanges&) @ 0x0000000012659b3d
2024-06-21 19:58:31 12. ./build_docker/./src/Storages/MergeTree/MergeTreeRangeReader.cpp:0: DB::MergeTreeRangeReader::read(unsigned long, DB::MarkRanges&) @ 0x0000000012656eb5
2024-06-21 19:58:31 13. ./build_docker/./src/Storages/MergeTree/MergeTreeReadTask.cpp:165: DB::MergeTreeReadTask::read(DB::MergeTreeReadTask::BlockSizeParams const&) @ 0x0000000012678748
2024-06-21 19:58:31 14. ./src/Storages/MergeTree/MergeTreeSelectAlgorithms.h:38: DB::MergeTreeThreadSelectAlgorithm::readFromTask(DB::MergeTreeReadTask&, DB::MergeTreeReadTask::BlockSizeParams const&) @ 0x0000000012f30a0f
2024-06-21 19:58:31 15. ./build_docker/./src/Storages/MergeTree/MergeTreeSelectProcessor.cpp:137: DB::MergeTreeSelectProcessor::read() @ 0x000000001267572e
2024-06-21 19:58:31 16. ./build_docker/./src/Storages/MergeTree/MergeTreeSource.cpp:181: DB::MergeTreeSource::tryGenerate() @ 0x0000000012f67378
2024-06-21 19:58:31 17. ./contrib/llvm-project/libcxx/include/optional:344: DB::ISource::work() @ 0x0000000012af80e2
2024-06-21 19:58:31 18. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000012b11f67
2024-06-21 19:58:31 19. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:272: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000012b06850
2024-06-21 19:58:31 20. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:380: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000012b07ece
2024-06-21 19:58:31 21. ./base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false, true>, void*>) @ 0x000000000d07eabb
2024-06-21 19:58:31 22. ./build_docker/./src/Common/ThreadPool.cpp:0: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false, true>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000000d082b51
2024-06-21 19:58:31 23. ./base/base/../base/wide_integer_impl.h:817: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000d081903
2024-06-21 19:58:31 24. ? @ 0x00007f19761ceac3
2024-06-21 19:58:31 25. ? @ 0x00007f1976260850
2024-06-21 19:58:31 
2024-06-21 19:58:31 Received exception from server (version 24.7.1):
2024-06-21 19:58:31 Code: 33. DB::Exception: Received from localhost:9000. DB::Exception: Cannot read all array values: read just 1797 of 1935: (while reading column d): (while reading from part /var/lib/clickhouse/store/ff7/ff761762-88a0-4170-907f-f5766f50f7ba/all_1_3_1/ in table test_z5oi1ytj.test (ff761762-88a0-4170-907f-f5766f50f7ba) located on disk default of type local, from mark 88 with max_rows_to_read = 1940): While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (CANNOT_READ_ALL_DATA)
2024-06-21 19:58:31 (query: select count(), dynamicType(d) || ':' || dynamicType(d.`Tuple(a Dynamic(max_types=3))`.a) as type from test group by type order by count(), type)
2024-06-21 19:58:31 
2024-06-21 19:58:31 stdout:
2024-06-21 19:58:31 MergeTree compact + horizontal merge
2024-06-21 19:58:31 test
2024-06-21 19:58:31 16667	Tuple(a Dynamic(max_types=3)):Date
2024-06-21 19:58:31 33333	Tuple(a Dynamic(max_types=3)):Array(UInt8)
2024-06-21 19:58:31 50000	Tuple(a Dynamic(max_types=3)):String
2024-06-21 19:58:31 50000	Tuple(a Dynamic(max_types=3)):UInt64
2024-06-21 19:58:31 100000	UInt64:None
2024-06-21 19:58:31 33333	Tuple(a Dynamic(max_types=3)):Array(UInt8)
2024-06-21 19:58:31 50000	Tuple(a Dynamic(max_types=3)):UInt64
2024-06-21 19:58:31 66667	Tuple(a Dynamic(max_types=3)):String
2024-06-21 19:58:31 100000	UInt64:None
2024-06-21 19:58:31 16667	Tuple(a Dynamic(max_types=3)):DateTime
2024-06-21 19:58:31 33333	Tuple(a Dynamic(max_types=3)):Array(UInt8)
2024-06-21 19:58:31 50000	Tuple(a Dynamic(max_types=3)):UInt64
2024-06-21 19:58:31 66667	Tuple(a Dynamic(max_types=3)):String
2024-06-21 19:58:31 100000	Tuple(a Dynamic(max_types=3)):Tuple(UInt64)
2024-06-21 19:58:31 100000	UInt64:None
2024-06-21 19:58:31 133333	Tuple(a Dynamic(max_types=3)):None
2024-06-21 19:58:31 50000	Tuple(a Dynamic(max_types=3)):UInt64
2024-06-21 19:58:31 100000	Tuple(a Dynamic(max_types=3)):Tuple(UInt64)
2024-06-21 19:58:31 100000	UInt64:None
2024-06-21 19:58:31 116667	Tuple(a Dynamic(max_types=3)):String
2024-06-21 19:58:31 133333	Tuple(a Dynamic(max_types=3)):None
2024-06-21 19:58:31 MergeTree wide + horizontal merge
2024-06-21 19:58:31 test
2024-06-21 19:58:31 16667	Tuple(a Dynamic(max_types=3)):Date
2024-06-21 19:58:31 33333	Tuple(a Dynamic(max_types=3)):Array(UInt8)
2024-06-21 19:58:31 50000	Tuple(a Dynamic(max_types=3)):String
2024-06-21 19:58:31 50000	Tuple(a Dynamic(max_types=3)):UInt64
2024-06-21 19:58:31 100000	UInt64:None
2024-06-21 19:58:31 33333	Tuple(a Dynamic(max_types=3)):Array(UInt8)
2024-06-21 19:58:31 50000	Tuple(a Dynamic(max_types=3)):UInt64
2024-06-21 19:58:31 66667	Tuple(a Dynamic(max_types=3)):String
2024-06-21 19:58:31 100000	UInt64:None
2024-06-21 19:58:31 16667	Tuple(a Dynamic(max_types=3)):DateTime
2024-06-21 19:58:31 33333	Tuple(a Dynamic(max_types=3)):Array(UInt8)
2024-06-21 19:58:31 50000	Tuple(a Dynamic(max_types=3)):UInt64
2024-06-21 19:58:31 66667	Tuple(a Dynamic(max_types=3)):String
2024-06-21 19:58:31 100000	Tuple(a Dynamic(max_types=3)):Tuple(UInt64)
2024-06-21 19:58:31 100000	UInt64:None
2024-06-21 19:58:31 133333	Tuple(a Dynamic(max_types=3)):None
2024-06-21 19:58:31 50000	Tuple(a Dynamic(max_types=3)):UInt64
2024-06-21 19:58:31 100000	Tuple(a Dynamic(max_types=3)):Tuple(UInt64)
2024-06-21 19:58:31 100000	UInt64:None
2024-06-21 19:58:31 116667	Tuple(a Dynamic(max_types=3)):String
2024-06-21 19:58:31 133333	Tuple(a Dynamic(max_types=3)):None
2024-06-21 19:58:31 MergeTree compact + vertical merge
2024-06-21 19:58:31 test
2024-06-21 19:58:31 16667	Tuple(a Dynamic(max_types=3)):Date
2024-06-21 19:58:31 33333	Tuple(a Dynamic(max_types=3)):Array(UInt8)
2024-06-21 19:58:31 50000	Tuple(a Dynamic(max_types=3)):String
2024-06-21 19:58:31 50000	Tuple(a Dynamic(max_types=3)):UInt64
2024-06-21 19:58:31 100000	UInt64:None
2024-06-21 19:58:31 33333	Tuple(a Dynamic(max_types=3)):Array(UInt8)
2024-06-21 19:58:31 50000	Tuple(a Dynamic(max_types=3)):UInt64
2024-06-21 19:58:31 66667	Tuple(a Dynamic(max_types=3)):String
2024-06-21 19:58:31 100000	UInt64:None
2024-06-21 19:58:31 16667	Tuple(a Dynamic(max_types=3)):DateTime
2024-06-21 19:58:31 33333	Tuple(a Dynamic(max_types=3)):Array(UInt8)
2024-06-21 19:58:31 50000	Tuple(a Dynamic(max_types=3)):UInt64
2024-06-21 19:58:31 66667	Tuple(a Dynamic(max_types=3)):String
2024-06-21 19:58:31 100000	Tuple(a Dynamic(max_types=3)):Tuple(UInt64)
2024-06-21 19:58:31 100000	UInt64:None
2024-06-21 19:58:31 133333	Tuple(a Dynamic(max_types=3)):None
2024-06-21 19:58:31 50000	Tuple(a Dynamic(max_types=3)):UInt64
2024-06-21 19:58:31 100000	Tuple(a Dynamic(max_types=3)):Tuple(UInt64)
2024-06-21 19:58:31 100000	UInt64:None
2024-06-21 19:58:31 116667	Tuple(a Dynamic(max_types=3)):String
2024-06-21 19:58:31 133333	Tuple(a Dynamic(max_types=3)):None
2024-06-21 19:58:31 MergeTree wide + vertical merge
2024-06-21 19:58:31 test
2024-06-21 19:58:31 16667	Tuple(a Dynamic(max_types=3)):Date
2024-06-21 19:58:31 33333	Tuple(a Dynamic(max_types=3)):Array(UInt8)
2024-06-21 19:58:31 50000	Tuple(a Dynamic(max_types=3)):String
2024-06-21 19:58:31 50000	Tuple(a Dynamic(max_types=3)):UInt64
2024-06-21 19:58:31 100000	UInt64:None
2024-06-21 19:58:31 33333	Tuple(a Dynamic(max_types=3)):Array(UInt8)
2024-06-21 19:58:31 50000	Tuple(a Dynamic(max_types=3)):UInt64
2024-06-21 19:58:31 66667	Tuple(a Dynamic(max_types=3)):String
2024-06-21 19:58:31 100000	UInt64:None
2024-06-21 19:58:31 50000	Tuple(a Dynamic(max_types=3)):UInt64
2024-06-21 19:58:31 100000	Tuple(a Dynamic(max_types=3)):Tuple(UInt64)
2024-06-21 19:58:31 100000	UInt64:None
2024-06-21 19:58:31 116667	Tuple(a Dynamic(max_types=3)):String
2024-06-21 19:58:31 133333	Tuple(a Dynamic(max_types=3)):None
2024-06-21 19:58:31 
2024-06-21 19:58:31 Settings used in the test: --max_insert_threads 0 --group_by_two_level_threshold 147831 --group_by_two_level_threshold_bytes 33389058 --distributed_aggregation_memory_efficient 0 --fsync_metadata 0 --output_format_parallel_formatting 1 --input_format_parallel_parsing 0 --min_chunk_bytes_for_parallel_parsing 13336238 --max_read_buffer_size 519563 --prefer_localhost_replica 0 --max_block_size 46058 --max_joined_block_size_rows 78498 --max_threads 64 --optimize_append_index 1 --optimize_if_chain_to_multiif 1 --optimize_if_transform_strings_to_enum 0 --optimize_read_in_order 1 --optimize_or_like_chain 0 --optimize_substitute_columns 0 --enable_multiple_prewhere_read_steps 0 --read_in_order_two_level_merge_threshold 63 --optimize_aggregation_in_order 0 --aggregation_in_order_max_block_bytes 43721071 --use_uncompressed_cache 1 --min_bytes_to_use_direct_io 1 --min_bytes_to_use_mmap_io 10737418240 --local_filesystem_read_method io_uring --remote_filesystem_read_method threadpool --local_filesystem_read_prefetch 0 --filesystem_cache_segments_batch_size 0 --read_from_filesystem_cache_if_exists_otherwise_bypass_cache 0 --throw_on_error_from_cache_on_write_operations 0 --remote_filesystem_read_prefetch 1 --allow_prefetched_read_pool_for_remote_filesystem 0 --filesystem_prefetch_max_memory_usage 128Mi --filesystem_prefetches_limit 10 --filesystem_prefetch_min_bytes_for_single_read_task 1Mi --filesystem_prefetch_step_marks 50 --filesystem_prefetch_step_bytes 100Mi --compile_aggregate_expressions 1 --compile_sort_description 0 --merge_tree_coarse_index_granularity 3 --optimize_distinct_in_order 0 --max_bytes_before_external_sort 10737418240 --max_bytes_before_external_group_by 0 --max_bytes_before_remerge_sort 908956056 --min_compress_block_size 2869657 --max_compress_block_size 2204584 --merge_tree_compact_parts_min_granules_to_multibuffer_read 68 --optimize_sorting_by_input_stream_properties 0 --http_response_buffer_size 4160034 --http_wait_end_of_query False --enable_memory_bound_merging_of_aggregation_results 1 --min_count_to_compile_expression 3 --min_count_to_compile_aggregate_expression 3 --min_count_to_compile_sort_description 0 --session_timezone America/Hermosillo --prefer_warmed_unmerged_parts_seconds 8 --use_page_cache_for_disks_without_file_cache True --page_cache_inject_eviction False --merge_tree_read_split_ranges_into_intersecting_and_non_intersecting_injection_probability 0.3 --prefer_external_sort_block_bytes 100000000 --cross_join_min_rows_to_compress 0 --cross_join_min_bytes_to_compress 1 --min_external_table_block_size_bytes 0 --max_parsing_threads 0
2024-06-21 19:58:31 
2024-06-21 19:58:31 MergeTree settings used in test: --ratio_of_defaults_for_sparse_serialization 1.0 --prefer_fetch_merged_part_size_threshold 10737418240 --vertical_merge_algorithm_min_rows_to_activate 238948 --vertical_merge_algorithm_min_columns_to_activate 100 --allow_vertical_merges_from_compact_to_wide_parts 1 --min_merge_bytes_to_use_direct_io 2211552964 --index_granularity_bytes 23232156 --merge_max_block_size 194 --index_granularity 40050 --min_bytes_for_wide_part 1073741824 --marks_compress_block_size 73083 --primary_key_compress_block_size 12526 --replace_long_file_name_to_hash 1 --max_file_name_length 0 --min_bytes_for_full_part_storage 0 --compact_parts_max_bytes_to_buffer 498931603 --compact_parts_max_granules_to_buffer 22 --compact_parts_merge_max_bytes_to_prefetch_part 460715 --cache_populated_by_fetch 0 --concurrent_part_removal_threshold 84 --old_parts_lifetime 480
2024-06-21 19:58:31 
2024-06-21 19:58:31 Database: test_z5oi1ytj

Metadata

Metadata

Assignees

Labels

bugConfirmed user-visible misbehaviour in official releasecrashCrash / segfault / abort

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions