-
Notifications
You must be signed in to change notification settings - Fork 8.3k
The specified key does not exist (S3_ERROR) on server startup in functional tests with Replicated database #37791
Copy link
Copy link
Closed
Labels
comp-object-storageObject storage connectivity (S3/GCS/Azure) including credentials, retries, multipart, etc.Object storage connectivity (S3/GCS/Azure) including credentials, retries, multipart, etc.testingSpecial issue with list of bugs found by CISpecial issue with list of bugs found by CI
Description
Sometimes server fails to start because of the following error:
2022.06.01 20:57:07.656220 [ 480 ] {} <Information> DiskS3: Starting up disk s3_cache
2022.06.01 20:57:07.656230 [ 480 ] {} <Information> DiskS3: Disk s3_cache started up
2022.06.01 20:57:07.656412 [ 480 ] {} <Information> Disk(s3_cache_2): Disk registered with cache path: /var/lib/clickhouse1/disks/s3_cache_2/data_cache/. Cache size: 22548578304, max cache elements size: 1048576, max_file_segment_size: 104857600
2022.06.01 20:57:07.656483 [ 480 ] {} <Information> AWSClient: Aws::Config::AWSConfigFileProfileConfigLoader: Initializing config loader against fileName /root/.aws/credentials and using profilePrefix = 0
2022.06.01 20:57:07.656490 [ 480 ] {} <Information> AWSClient: ProfileConfigFileAWSCredentialsProvider: Setting provider to read credentials from /root/.aws/credentials for credentials file and /root/.aws/config for the config file , for use with profile default
2022.06.01 20:57:07.656555 [ 480 ] {} <Trace> WriteBufferFromS3: Making single part upload. Bucket: test, Key: 00170_test/ceruvazksouixzeherhiuqazepedihzw, Size: 4, WithPool: true
2022.06.01 20:57:07.657724 [ 1148 ] {} <Warning> AWSClient: AWSErrorMarshaller: Encountered AWSError 'NoSuchKey': The specified key does not exist.
2022.06.01 20:57:07.657770 [ 1148 ] {} <Error> AWSClient: HTTP response code: 404
Resolved remote host IP address:
Request ID: 16F4A47ECC949983
Exception name: NoSuchKey
Error message: The specified key does not exist.
12 response headers:
accept-ranges : bytes
connection : close
content-length : 373
content-security-policy : block-all-mixed-content
content-type : application/xml
date : Wed, 01 Jun 2022 23:57:07 GMT
server : MinIO
strict-transport-security : max-age=31536000; includeSubDomains
vary : Accept-Encoding
x-amz-request-id : 16F4A47ECC949983
x-content-type-options : nosniff
x-xss-protection : 1; mode=block
2022.06.01 20:57:07.657856 [ 1148 ] {} <Warning> AWSClient: If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
2022.06.01 20:57:07.658223 [ 480 ] {} <Error> virtual DB::WriteIndirectBufferFromRemoteFS::~WriteIndirectBufferFromRemoteFS(): Code: 499. DB::Exception: The specified key does not exist. (S3_ERROR), Stack trace (when copying this message, always include the lines below):
0. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xb6f337a in /usr/bin/clickhouse
1. DB::WriteBufferFromS3::processPutRequest(DB::WriteBufferFromS3::PutObjectTask&) @ 0x15c984df in /usr/bin/clickhouse
2. ? @ 0x15c9a121 in /usr/bin/clickhouse
3. ? @ 0x168cb080 in /usr/bin/clickhouse
4. ThreadPoolImpl<ThreadFromGlobalPool>::worker(std::__1::__list_iterator<ThreadFromGlobalPool, void*>) @ 0xb7a72ba in /usr/bin/clickhouse
5. ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0xb7a92a4 in /usr/bin/clickhouse
6. ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0xb7a46e7 in /usr/bin/clickhouse
7. ? @ 0xb7a811d in /usr/bin/clickhouse
8. ? @ 0x7fc8d7345609 in ?
9. __clone @ 0x7fc8d726a163 in ?
(version 22.6.1.1015)
2022.06.01 20:57:07.658682 [ 480 ] {} <Information> Application: Shutting down storages.
cc: @kssenii
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
comp-object-storageObject storage connectivity (S3/GCS/Azure) including credentials, retries, multipart, etc.Object storage connectivity (S3/GCS/Azure) including credentials, retries, multipart, etc.testingSpecial issue with list of bugs found by CISpecial issue with list of bugs found by CI