S3 reads and writes issued from WriteBufferFromS3 and ReadBufferFromS3 are done under ResourceGuard. But the number of bytes written (or read) in a single PutObject (or GetObject) AWS request can be high: tens or hundreds of MBs. Huge costs lead to larger unfairness and bandwidth_limit throttling violation in case if max_burst is comparable or less than cost of a single resource request.

Describe the solution you'd like
Probably, integration with IO Scheduler should be moved from S3 Buffers to S3 Client. Consider reimplementing ResourceGuard as RateLimiterInterface implementation.