@filimonov:

Mutation

3 [ 13 ] {} <Trace> Aggregator: Aggregated. 0 to 1 rows (from 0.000 MiB) in 0.000 sec. (0.000 rows/sec., 0.000 MiB/sec.)
2019.10.21 21:41:35.339432 [ 13 ] {} <Trace> Aggregator: Merging aggregated data
2019.10.21 21:41:35.339472 [ 13 ] {} <Trace> virtual DB::MergingAndConvertingBlockInputStream::~MergingAndConvertingBlockInputStream(): Waiting for threads to finish
2019.10.21 21:41:35.339528 [ 13 ] {} <Trace> default.test (MergerMutator): Part all_601_792_3_955 doesn't change up to mutation version 956
2019.10.21 21:41:35.339564 [ 13 ] {} <Debug> DiskSpaceMonitor: Reserving 2.68 GiB on disk `ephemeral`, having unreserved 55.21 GiB.
2019.10.21 21:41:35.339607 [ 13 ] {} <Debug> default.test: Cloning part /var/lib/clickhouse/data/default/test/all_601_792_3_955/ to /mnt/ephemeral/data/default/test/tmp_clone_all_601_792_3_956
2019.10.21 21:41:35.340081 [ 20 ] {} <Trace> system.part_log: Renaming temporary part tmp_insert_201910_4_4_0 to 201910_31_31_0.
2019.10.21 21:41:35.340393 [ 11 ] {} <Debug> DiskSpaceMonitor: Reserving 2.48 GiB on disk `default`, having unreserved 27.12 GiB.
2019.10.21 21:41:35.340597 [ 11 ] {} <Debug> default.test (SelectExecutor): Key condition: (column 0 in [999000001, +inf))
2019.10.21 21:41:35.340705 [ 11 ] {} <Debug> default.test (SelectExecutor): Selected 1 parts by date, 1 parts by key, 123 marks to read from 1 ranges
2019.10.21 21:41:35.340727 [ 11 ] {} <Trace> MergeTreeSelectBlockInputStream: Reading 1 ranges from part all_793_954_3_955, approx. 1007616 rows starting from 168525824
2019.10.21 21:41:35.340751 [ 11 ] {} <Trace> InterpreterSelectQuery: FetchColumns -> Complete
2019.10.21 21:41:35.340813 [ 11 ] {} <Trace> Aggregator: Aggregating
2019.10.21 21:41:35.349846 [ 11 ] {} <Trace> Aggregator: Aggregation method: without_key
2019.10.21 21:41:35.380235 [ 11 ] {} <Trace> Aggregator: Aggregated. 999999 to 1 rows (from 0.000 MiB) in 0.039 sec. (25387821.577 rows/sec., 0.003 MiB/sec.)
2019.10.21 21:41:35.380269 [ 11 ] {} <Trace> Aggregator: Merging aggregated data
2019.10.21 21:41:35.380569 [ 11 ] {} <Trace> virtual DB::MergingAndConvertingBlockInputStream::~MergingAndConvertingBlockInputStream(): Waiting for threads to finish
2019.10.21 21:41:35.380659 [ 11 ] {} <Trace> default.test (MergerMutator): Mutating part all_793_954_3_955 to mutation version 956
2019.10.21 21:41:35.381434 [ 11 ] {} <Debug> default.test (SelectExecutor): Key condition: unknown
2019.10.21 21:41:35.381462 [ 11 ] {} <Debug> default.test (SelectExecutor): Selected 1 parts by date, 1 parts by key, 20695 marks to read from 1 ranges
2019.10.21 21:41:35.381533 [ 11 ] {} <Trace> MergeTreeSelectBlockInputStream: Reading 1 ranges from part all_793_954_3_955, approx. 169533440 rows starting from 0
2019.10.21 21:41:35.381592 [ 11 ] {} <Trace> InterpreterSelectQuery: FetchColumns -> Complete
2019.10.21 21:41:35.388070 [ 13 ] {} <Error> void DB::BackgroundProcessingPool::threadFunction(): Code: 424, e.displayText() = DB::ErrnoException: Cannot link /var/lib/clickhouse/data/default/test/all_601_792_3_955/checksums.txt to /mnt/ephemeral/data/default/test/tmp_clone_all_601_792_3_956/checksums.txt, errno: 18, strerror: Invalid cross-device link, Stack trace:

0. 0x556fbb3ec640 StackTrace::StackTrace() /usr/bin/clickhouse
1. 0x556fbb3fdbd3 DB::ErrnoException::ErrnoException(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, int, std::optional<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&) /usr/bin/clickhouse
2. 0x556fbaf9e82a DB::throwFromErrnoWithPath(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, int) /usr/bin/clickhouse
3. 0x556fbf295049 DB::createHardLink(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) /usr/bin/clickhouse
4. 0x556fbf29701e ? /usr/bin/clickhouse
5. 0x556fbf2971b9 DB::localBackup(Poco::Path const&, Poco::Path const&, std::optional<unsigned long>) /usr/bin/clickhouse
6. 0x556fbea559d4 DB::MergeTreeData::cloneAndLoadDataPart(std::shared_ptr<DB::MergeTreeDataPart const> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::MergeTreePartInfo const&) /usr/bin/clickhouse
7. 0x556fbea74ee3 DB::MergeTreeDataMergerMutator::mutatePartToTemporaryPart(DB::FutureMergedMutatedPart const&, std::vector<DB::MutationCommand, std::allocator<DB::MutationCommand> > const&, DB::MergeListEntry&, DB::Context const&, DB::DiskSpace::Reservation*, DB::TableStructureReadLockHolder&) /usr/bin/clickhouse
8. 0x556fbe992deb DB::StorageMergeTree::tryMutatePart() /usr/bin/clickhouse
9. 0x556fbe99325b DB::StorageMergeTree::mergeMutateTask() /usr/bin/clickhouse
10. 0x556fbea20c94 DB::BackgroundProcessingPool::threadFunction() /usr/bin/clickhouse
11. 0x556fbea2160a ? /usr/bin/clickhouse
12. 0x556fbb43561c ThreadPoolImpl<std::thread>::worker(std::_List_iterator<std::thread>) /usr/bin/clickhouse
13. 0x556fc10fceb0 ? /usr/bin/clickhouse
14. 0x7f7ed9db56db start_thread /lib/x86_64-linux-gnu/libpthread-2.27.so
15. 0x7f7ed96d288f clone /lib/x86_64-linux-gnu/libc-2.27.so
 (version 19.15.3.6 (official build))

and after

[ 11 ] {} <Debug> DiskSpaceMonitor: Reserving 2.68 GiB on disk `ephemeral`, having unreserved 55.21 GiB.
2019.10.21 21:41:40.180100 [ 11 ] {} <Error> void DB::BackgroundProcessingPool::threadFunction(): Code: 84, e.displayText() = DB::Exception: Part in /mnt/ephemeral/data/default/test/tmp_clone_all_601_792_3_956 already exists, Stack trace:

0. 0x556fbb3ec640 StackTrace::StackTrace() /usr/bin/clickhouse
1. 0x556fbb3ec415 DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int) /usr/bin/clickhouse
2. 0x556fbb1d979b ? /usr/bin/clickhouse
3. 0x556fbea74ee3 DB::MergeTreeDataMergerMutator::mutatePartToTemporaryPart(DB::FutureMergedMutatedPart const&, std::vector<DB::MutationCommand, std::allocator<DB::MutationCommand> > const&, DB::MergeListEntry&, DB::Context const&, DB::DiskSpace::Reservation*, DB::TableStructureReadLockHolder&) /usr/bin/clickhouse
4. 0x556fbe992deb DB::StorageMergeTree::tryMutatePart() /usr/bin/clickhouse
5. 0x556fbe99325b DB::StorageMergeTree::mergeMutateTask() /usr/bin/clickhouse
6. 0x556fbea20c94 DB::BackgroundProcessingPool::threadFunction() /usr/bin/clickhouse
7. 0x556fbea2160a ? /usr/bin/clickhouse
8. 0x556fbb43561c ThreadPoolImpl<std::thread>::worker(std::_List_iterator<std::thread>) /usr/bin/clickhouse
9. 0x556fc10fceb0 ? /usr/bin/clickhouse
10. 0x7f7ed9db56db start_thread /lib/x86_64-linux-gnu/libpthread-2.27.so
11. 0x7f7ed96d288f clone /lib/x86_64-linux-gnu/libc-2.27.so
 (version 19.15.3.6 (official build))