E           helpers.client.QueryRuntimeException: Client failed! Return code: 243, stderr: Received exception from server (version 22.4.1):
E           Code: 499. DB::Exception: Received from 172.16.19.7:9000. DB::Exception: The specified multipart upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed. Tags:"5118f8a583c5ff1af9f98da14fd029fe" "839f64c2f2de04ee4da659f13b042efa" "6c7383f2f57ce35892f120c75ebc24db" "9d6207c61e084dde895a8347a7564c7d" "cdf4e2dd729a8bb62b545bf0e6f3e8e8" "308130384072acaee77e3146bede5d7d" "08e4a70dc42565ad60485e2ba3b25586" "5c1cab94eacfa9fcc70714424caddc6c" "7b4de238304f49ec622fa7a8f2eee31e" "47ce59d0ed4095854ed81f18a9cca290" "a727298167cbc201792e76db2662d28a" "cc8fcdc2d2b399325c43841bc6b7d758" "fbd863fe64186635114fbf1180362e23" "6c6a98c3c38888ec4a5a5b34e188f683" "f7fcf4a3397d627c287bdfe47b8a2d4c" "f4092c000cb6288bab3d921505265232" "db1422861a99f47ea74ae537d9d01a14" "7e68a87245ac0e580b4343b9bec10ead" "30b5976c1558af6fc9de31f2cd23d05c" "39f25cb17d5f43f5317f1e6136ba442d" "7c0baff4c1b2ec298e760a0c24a14584" "d44b4edb8eb3ed68c4634748d1e5cec7" "ee551aceec032ef6d1d00acaa5638d53" "dae34f0f45817fa68f5c2a37288c15b9" "70d51bfd462f0c7b7890d91577ce2b28" "d4712e56c51963a7c9e192439fe56ffc" "335da02334a106dd1ee82a18853f50e2" "984ee1766043a837f9e6eaeacbaf0426" "97ff4fc6f9c8c2ac98e94a8e51ce179e" "e42183209a79c81585b3b5a15351b181". Stack trace:
E
E           0. ./build_docker/../contrib/libcxx/include/exception:133: Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x24a02592 in /usr/bin/clickhouse
E           1. ./build_docker/../src/Common/Exception.cpp:58: DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xf64754f in /usr/bin/clickhouse
E           2. ./build_docker/../contrib/libcxx/include/string:1445: DB::Exception::Exception<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, fmt::v8::join_view<std::__1::__wrap_iter<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*>, std::__1::__wrap_iter<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*>, char> >(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, fmt::v8::join_view<std::__1::__wrap_iter<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*>, std::__1::__wrap_iter<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*>, char>&&) @ 0x21c0a371 in /usr/bin/clickhouse
E           3. ./build_docker/../src/IO/WriteBufferFromS3.cpp:0: DB::WriteBufferFromS3::completeMultipartUpload() @ 0x21c05ed3 in /usr/bin/clickhouse
E           4. ./build_docker/../src/IO/WriteBuffer.h:127: DB::WriteBuffer::finalize() @ 0xf650849 in /usr/bin/clickhouse
E           5. ./build_docker/../src/Storages/StorageS3.cpp:438: DB::StorageS3Sink::onFinish() @ 0x23098078 in /usr/bin/clickhouse
E           6. ./build_docker/../src/Processors/Transforms/ExceptionKeepingTransform.cpp:103: DB::runStep(std::__1::function<void ()>, DB::ThreadStatus*, std::__1::atomic<unsigned long>*) @ 0x23c3259c in /usr/bin/clickhouse
E           7. ./build_docker/../src/Processors/Transforms/ExceptionKeepingTransform.cpp:0: DB::ExceptionKeepingTransform::work() @ 0x23c31f3a in /usr/bin/clickhouse
E           8. ./build_docker/../src/Processors/Executors/ExecutionThreadContext.cpp:53: DB::executeJob(DB::IProcessor*) @ 0x238ddf99 in /usr/bin/clickhouse
E           9. ./build_docker/../src/Processors/Executors/ExecutionThreadContext.cpp:65: DB::ExecutionThreadContext::executeTask() @ 0x238dde28 in /usr/bin/clickhouse
E           10. ./build_docker/../src/Processors/Executors/PipelineExecutor.cpp:213: DB::PipelineExecutor::executeStepImpl(unsigned long, std::__1::atomic<bool>*) @ 0x238d0461 in /usr/bin/clickhouse
E           11. ./build_docker/../src/Processors/Executors/PipelineExecutor.cpp:326: DB::PipelineExecutor::executeImpl(unsigned long) @ 0x238cfde7 in /usr/bin/clickhouse
E           12. ./build_docker/../contrib/libcxx/include/__memory/unique_ptr.h:284: DB::PipelineExecutor::execute(unsigned long) @ 0x238cf94d in /usr/bin/clickhouse
E           13. ./build_docker/../src/Processors/Executors/CompletedPipelineExecutor.cpp:0: DB::threadFunction(DB::CompletedPipelineExecutor::Data&, std::__1::shared_ptr<DB::ThreadGroupStatus>, unsigned long) @ 0x238cf0d7 in /usr/bin/clickhouse
E           14. ./build_docker/../contrib/libcxx/include/type_traits:3648: decltype(static_cast<DB::CompletedPipelineExecutor::execute()::$_0&>(fp)()) std::__1::__invoke_constexpr<DB::CompletedPipelineExecutor::execute()::$_0&>(DB::CompletedPipelineExecutor::execute()::$_0&) @ 0x238cf042 in /usr/bin/clickhouse
E           15. ./build_docker/../contrib/libcxx/include/tuple:0: decltype(static_cast<DB::CompletedPipelineExecutor::execute()::$_0>(fp)()) std::__1::__invoke<ThreadFromGlobalPool::ThreadFromGlobalPool<DB::CompletedPipelineExecutor::execute()::$_0>(DB::CompletedPipelineExecutor::execute()::$_0&&)::'lambda'()&>(DB::CompletedPipelineExecutor::execute()::$_0&&) @ 0x238ceefb in /usr/bin/clickhouse
E           16. ./build_docker/../contrib/libcxx/include/__functional/function.h:1157: ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0xf71735a in /usr/bin/clickhouse
E           17. ./build_docker/../src/Common/ThreadPool.cpp:0: void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()> >(void*) @ 0xf719e82 in /usr/bin/clickhouse
E           18. ? @ 0x7fbc17bcb609 in ?
E           19. clone @ 0x7fbc17af0163 in ?
E           . (S3_ERROR)
E           (query: insert into table function s3(s3_parquet, structure='a Int32, b String', format='Parquet') select number, randomString(100) from numbers(5000000) settings s3_truncate_on_insert=1)

grep multipart test_storage_s3/_instances/docker.log
minio1_1            | Error: open /data1-1/.minio.sys/multipart/c6bc7d7763d4c5cda2278674e011c3b20f4d15970fea969ac3c7d8d189207386/b91da2f8-1823-48cb-a784-090e379ace5f/00001.5118f8a583c5ff1af9f98da14fd029fe.34603008: no such file or directory (*fs.PathError)
minio1_1            |        uploadID=b91da2f8-1823-48cb-a784-090e379ace5f, partPath=/data1-1/.minio.sys/multipart/c6bc7d7763d4c5cda2278674e011c3b20f4d15970fea969ac3c7d8d189207386/b91da2f8-1823-48cb-a784-090e379ace5f/00001.5118f8a583c5ff1af9f98da14fd029fe.34603008, filepath=/data1-1/.minio.sys/tmp/4334d55b-4ca4-4f73-bb9b-a46a3d7d090c/b91da2f8-1823-48cb-a784-090e379ace5f.5ddf13b4-2fab-4e53-b36b-c3912a7834b4
minio1_1            |        4: cmd/fs-v1-multipart.go:121:cmd.(*FSObjects).backgroundAppend()
minio1_1            |        3: cmd/fs-v1-multipart.go:654:cmd.(*FSObjects).CompleteMultipartUpload()
minio1_1            | Error: open /data1-1/.minio.sys/multipart/c6bc7d7763d4c5cda2278674e011c3b20f4d15970fea969ac3c7d8d189207386/b91da2f8-1823-48cb-a784-090e379ace5f/00001.5118f8a583c5ff1af9f98da14fd029fe.34603008: no such file or directory (*fs.PathError)
minio1_1            |        uploadID=b91da2f8-1823-48cb-a784-090e379ace5f, partPath=/data1-1/.minio.sys/multipart/c6bc7d7763d4c5cda2278674e011c3b20f4d15970fea969ac3c7d8d189207386/b91da2f8-1823-48cb-a784-090e379ace5f/00001.5118f8a583c5ff1af9f98da14fd029fe.34603008, filepath=/data1-1/.minio.sys/tmp/4334d55b-4ca4-4f73-bb9b-a46a3d7d090c/b91da2f8-1823-48cb-a784-090e379ace5f.5ddf13b4-2fab-4e53-b36b-c3912a7834b4
minio1_1            |        3: cmd/fs-v1-multipart.go:696:cmd.(*FSObjects).CompleteMultipartUpload()
minio1_1            | Error: open /data1-1/.minio.sys/multipart/c6bc7d7763d4c5cda2278674e011c3b20f4d15970fea969ac3c7d8d189207386/b91da2f8-1823-48cb-a784-090e379ace5f/00001.5118f8a583c5ff1af9f98da14fd029fe.34603008: no such file or directory (*fs.PathError)
minio1_1            |        uploadID=b91da2f8-1823-48cb-a784-090e379ace5f, partPath=/data1-1/.minio.sys/multipart/c6bc7d7763d4c5cda2278674e011c3b20f4d15970fea969ac3c7d8d189207386/b91da2f8-1823-48cb-a784-090e379ace5f/00001.5118f8a583c5ff1af9f98da14fd029fe.34603008, filepath=/data1-1/.minio.sys/tmp/4334d55b-4ca4-4f73-bb9b-a46a3d7d090c/b91da2f8-1823-48cb-a784-090e379ace5f.5ddf13b4-2fab-4e53-b36b-c3912a7834b4
proxy1_1            | 172.16.19.7 - - [30/Mar/2022:14:05:28 +0000] "GET /root?list-type=2&prefix=test_multipart.csv HTTP/1.1" 307 171 "-" "aws-sdk-cpp/1.8.121/S3/Linux/5.13.0-1019-aws x86_64 Clang/13.0.1"
proxy1_1            | 172.16.19.7 - - [30/Mar/2022:14:05:30 +0000] "POST /root/test_multipart.csv?uploads HTTP/1.1" 307 171 "-" "aws-sdk-cpp/1.8.121/S3/Linux/5.13.0-1019-aws x86_64 Clang/13.0.1"
proxy1_1            | 172.16.19.7 - - [30/Mar/2022:14:05:30 +0000] "PUT /root/test_multipart.csv?partNumber=2&uploadId=6db5da74-ee45-4391-acff-25655630aeac HTTP/1.1" 307 171 "-" "aws-sdk-cpp/1.8.121/S3/Linux/5.13.0-1019-aws x86_64 Clang/13.0.1"
proxy1_1            | 172.16.19.7 - - [30/Mar/2022:14:05:30 +0000] "PUT /root/test_multipart.csv?partNumber=1&uploadId=6db5da74-ee45-4391-acff-25655630aeac HTTP/1.1" 307 171 "-" "aws-sdk-cpp/1.8.121/S3/Linux/5.13.0-1019-aws x86_64 Clang/13.0.1"
proxy1_1            | 172.16.19.7 - - [30/Mar/2022:14:05:30 +0000] "POST /root/test_multipart.csv?uploadId=6db5da74-ee45-4391-acff-25655630aeac HTTP/1.1" 307 171 "-" "aws-sdk-cpp/1.8.121/S3/Linux/5.13.0-1019-aws x86_64 Clang/13.0.1"