Disable S3 requests processing during context shutdown to speed up termination process#14496
Conversation
…rmination process.
| for (auto & [disk_name, disk] : getDisksMap()) | ||
| { | ||
| LOG_INFO(shared->log, "Shutdown disk {}", disk_name); | ||
| disk->shutdown(); |
There was a problem hiding this comment.
But tables shutdown will be called only after this line in shared->shutdown(). Can it lead to unexpected errors in log on server shutdown when tables still try to use their s3 disks?
There was a problem hiding this comment.
client->DisableRequestProcessing stops any next retry attempt for ongoing S3 request.
If an S3 request is failed and the code above is executed S3 client immediately returns the last failed S3 request outcome.
If S3 is healthy nothing wrong will be happened and S3 requests are processed in a regular way without errors.
When table is shutdown it waits for currently running merges or moves. If a table uses S3 and S3 is unhealthy we will just fail merge or move the process faster with the same result as we fail request after all unsuccessful retry attempts. If S3 is a healthy merge or move process will finish in a regular way without errors.
There was a problem hiding this comment.
Awesome, can you add the same explanation to the shutdown method of DiskS3.
|
Only comment updated, merge without tests |
Backport to 20.8 - Disable S3 requests processing during context shutdown to speed up termination process #14496
I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Speed up server shutdown process if there are ongoing S3 requests
Detailed description / Documentation draft:
The shutdown process can be delayed if there are ongoing S3 requests, especially when S3 is unavailable (request is repeated again and again due to retry strategy).
When shutdown is called we can forcibly break all S3 connections and speed up the shutdown process.