Skip to content

IO: Fix infinity retries when meet network partition with S3#10447

Merged
ti-chi-bot[bot] merged 8 commits intopingcap:masterfrom
JaySon-Huang:perf
Sep 19, 2025
Merged

IO: Fix infinity retries when meet network partition with S3#10447
ti-chi-bot[bot] merged 8 commits intopingcap:masterfrom
JaySon-Huang:perf

Conversation

@JaySon-Huang
Copy link
Contributor

@JaySon-Huang JaySon-Huang commented Sep 18, 2025 •
edited
Loading

What problem does this PR solve?

Issue Number: close #10424

Problem Summary:

S3RandomAccessFile may return S3StreamError(-2) as return value in read / seek. However, ReadBufferFromRandomAccessFile only check whether the return value of read is -1. This will lead to infinity retries when we happen to create a S3RandomAccessFile successfully, but later can not read from S3.

tiflash/dbms/src/IO/Buffer/ReadBufferFromRandomAccessFile.cpp

Lines 53 to 73 in f455495

size_t bytes_read = 0;
while (!bytes_read)
{
ProfileEvents::increment(ProfileEvents::ReadBufferFromFileDescriptorRead);
ssize_t res = 0;
{
res = file->read(internal_buffer.begin(), internal_buffer.size());
}
if (!res)
break;
if (-1 == res && errno != EINTR)
{
ProfileEvents::increment(ProfileEvents::ReadBufferFromFileDescriptorReadFailed);
throwFromErrno("Cannot read from file " + getFileName(), ErrorCodes::CANNOT_READ_FROM_FILE_DESCRIPTOR);
}
if (res > 0)
bytes_read += res;
}

What is changed and how it works?

* `S3RandomAccessFile::initialize` now directly throw an exception when exceed the max retries time
* Change the return value check of seek, read from `!= -1` to `< 0`
* Add attribute `[[nodiscard]]` for seek, read, write, pread, pwrite
* Tune the AWS error level logging to warning level

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

Fix the issue that TiFlash may retries infinity when meet network partition with S3

@JaySon-Huang
Save
2598e9c 
Signed-off-by: JaySon-Huang <tshent@qq.com>
@JaySon-Huang
Add tests
5fee59d 
Signed-off-by: JaySon-Huang <tshent@qq.com>
@JaySon-Huang
Add check for return value of seek/read
0b1d1db 
Signed-off-by: JaySon-Huang <tshent@qq.com>
@ti-chi-bot ti-chi-bot bot added do-not-merge/needs-linked-issue release-note-none Denotes a PR that doesn't merit a release note. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Sep 18, 2025
JaySon-Huang
JaySon-Huang commented Sep 18, 2025
View reviewed changes
dbms/src/IO/Buffer/ReadBufferFromRandomAccessFile.cpp
if (res == 0)
break;

if (-1 == res && errno != EINTR)
Copy link
Contributor Author

@JaySon-Huang JaySon-Huang Sep 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This cause #10424

@JaySon-Huang
Add test about inf retries
e1d7169 
Signed-off-by: JaySon-Huang <tshent@qq.com>
@JaySon-Huang
Throw exception when initialize exceed max retries
731ef7e 
Signed-off-by: JaySon-Huang <tshent@qq.com>
@JaySon-Huang
Revert the changes in TableScanImpl
5abaeb1 
Signed-off-by: JaySon-Huang <tshent@qq.com>
@JaySon-Huang JaySon-Huang changed the title [WIP] IO: Fix infinity retries when meet network partition with S3 Sep 19, 2025
@ti-chi-bot ti-chi-bot bot added do-not-merge/needs-triage-completed release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. do-not-merge/needs-linked-issue do-not-merge/needs-triage-completed release-note-none Denotes a PR that doesn't merit a release note. labels Sep 19, 2025
@JaySon-Huang
Cherry-pick the fix on local index size under disagg arch
2ab11fd 
Signed-off-by: JaySon-Huang <tshent@qq.com>
@ti-chi-bot ti-chi-bot bot added needs-1-more-lgtm Indicates a PR needs 1 more LGTM. approved labels Sep 19, 2025
CalvinNeo
CalvinNeo approved these changes Sep 19, 2025
View reviewed changes
Copy link
Member

@CalvinNeo CalvinNeo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Sep 19, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: CalvinNeo, Lloyd-Pottiger

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:
  • OWNERS [CalvinNeo,Lloyd-Pottiger]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Sep 19, 2025
@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Sep 19, 2025

[LGTM Timeline notifier]

Timeline:

  • 2025-09-19 03:05:36.792441982 +0000 UTC m=+155498.271022639: ☑️ agreed by Lloyd-Pottiger.
  • 2025-09-19 03:08:12.140369684 +0000 UTC m=+155653.618950351: ☑️ agreed by CalvinNeo.

@JaySon-Huang
Fix lint
c7021e9 
Signed-off-by: JaySon-Huang <tshent@qq.com>
@ti-chi-bot ti-chi-bot bot merged commit 1254924 into pingcap:master Sep 19, 2025
7 checks passed
@JaySon-Huang JaySon-Huang deleted the perf branch September 19, 2025 06:08
@ti-chi-bot ti-chi-bot bot added the needs-cherry-pick-release-8.5 Should cherry pick this PR to release-8.5 branch. label Sep 19, 2025
ti-chi-bot pushed a commit to ti-chi-bot/tiflash that referenced this pull request Sep 19, 2025
@JaySon-Huang @ti-chi-bot
This is an automated cherry-pick of pingcap#10447
43d62bb 
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot ti-chi-bot mentioned this pull request Sep 19, 2025
Merged
12 tasks
@ti-chi-bot
Copy link
Member

ti-chi-bot commented Sep 19, 2025

In response to a cherrypick label: new pull request created to branch release-8.5: #10449.
But this PR has conflicts, please resolve them!

ti-chi-bot bot pushed a commit that referenced this pull request Sep 22, 2025
@ti-chi-bot @JaySon-Huang
IO: Fix infinity retries when meet network partition with S3 (#10447) (…
496ffee 
…#10449)

close #10424

* `S3RandomAccessFile::initialize` now directly throw an exception when exceed the max retries time
* Change the return value check of seek, read from `!= -1` to `< 0`
* Add attribute `[[nodiscard]]` for seek, read, write, pread, pwrite
* Tune the AWS error level logging to warning level

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
Signed-off-by: JaySon-Huang <tshent@qq.com>

Co-authored-by: JaySon <tshent@qq.com>
Co-authored-by: JaySon-Huang <tshent@qq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@CalvinNeo CalvinNeo CalvinNeo approved these changes

@Lloyd-Pottiger Lloyd-Pottiger Lloyd-Pottiger approved these changes

@JinheLin JinheLin Awaiting requested review from JinheLin

Assignees

No one assigned

Labels

approved lgtm needs-cherry-pick-release-8.5 Should cherry pick this PR to release-8.5 branch. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[next gen] write node repeatedly retry reading S3 object after inject network partition between write node and S3

4 participants

@JaySon-Huang @ti-chi-bot @CalvinNeo @Lloyd-Pottiger