[next gen] write node repeatedly retry reading S3 object after inject network partition between write node and S3 #10424
Description
Bug Report
Please answer these questions before submitting your issue. Thanks!
1. Minimal reproduce step (Required)
2. What did you expect to see? (Required)
3. What did you see instead (Required)
[2025/09/07 13:10:12.812 +00:00] [WARN] [PageDirectory.cpp:2177] ["Meet a stale snapshot, create_thread=1706 tracing_id=write seq=25559957174 alive_time=2489.249"] [source=uni_write] [thread_id=1695]Show context
[2025/09/07 12:49:07.421 +00:00] [WARN] [PageDirectory.cpp:2177] ["Meet a stale snapshot, create_thread=1706 tracing_id=write seq=25559957174 alive_time=1223.859"] [source=uni_write] [thread_id=1690]
[2025/09/07 12:42:10.510 +00:00] [WARN] [PageDirectory.cpp:2177] ["Meet a stale snapshot, create_thread=1706 tracing_id=write seq=25559957174 alive_time=806.947"] [source=uni_write] [thread_id=1686]
[2025/09/07 12:40:06.596 +00:00] [WARN] [PageDirectory.cpp:2177] ["Meet a stale snapshot, create_thread=1706 tracing_id=write seq=25559957174 alive_time=683.033"] [source=uni_write] [thread_id=1689]
[2025/09/07 13:12:24.839 +00:00] [ERROR] [S3RandomAccessFile.cpp:118] ["Cannot read from istream, size=2688 gcount=0 state=0x05 cur_offset=0 content_length=1767099 errno=2 errmsg=No such file or directory cost=413ns"] [source=s389908/data/ks_1_t_521/dmf_22530948/0.merged] [thread_id=1706]
[2025/09/07 13:12:24.839 +00:00] [INFO] [S3RandomAccessFile.cpp:269] ["S3 GetObject timeout: max_retry=3, key=tiflash-root/s389908/data/ks_1_t_521/dmf_22530948/0.merged, elapsed=0.000s"] [source=s389908/data/ks_1_t_521/dmf_22530948/0.merged] [thread_id=1706]
[2025/09/07 13:12:24.839 +00:00] [ERROR] [S3RandomAccessFile.cpp:118] ["Cannot read from istream, size=2688 gcount=0 state=0x05 cur_offset=0 content_length=1767099 errno=2 errmsg=No such file or directory cost=442ns"] [source=s389908/data/ks_1_t_521/dmf_22530948/0.merged] [thread_id=1706]
[2025/09/07 13:12:24.839 +00:00] [INFO] [S3RandomAccessFile.cpp:269] ["S3 GetObject timeout: max_retry=3, key=tiflash-root/s389908/data/ks_1_t_521/dmf_22530948/0.merged, elapsed=0.000s"] [source=s389908/data/ks_1_t_521/dmf_22530948/0.merged] [thread_id=1706]
[2025/09/07 13:12:24.839 +00:00] [ERROR] [S3RandomAccessFile.cpp:118] ["Cannot read from istream, size=2688 gcount=0 state=0x05 cur_offset=0 content_length=1767099 errno=2 errmsg=No such file or directory cost=408ns"] [source=s389908/data/ks_1_t_521/dmf_22530948/0.merged] [thread_id=1706]
[2025/09/07 13:12:24.839 +00:00] [INFO] [S3RandomAccessFile.cpp:269] ["S3 GetObject timeout: max_retry=3, key=tiflash-root/s389908/data/ks_1_t_521/dmf_22530948/0.merged, elapsed=0.000s"] [source=s389908/data/ks_1_t_521/dmf_22530948/0.merged] [thread_id=1706]
[2025/09/07 13:12:24.839 +00:00] [ERROR] [S3RandomAccessFile.cpp:118] ["Cannot read from istream, size=2688 gcount=0 state=0x05 cur_offset=0 content_length=1767099 errno=2 errmsg=No such file or directory cost=407ns"] [source=s389908/data/ks_1_t_521/dmf_22530948/0.merged] [thread_id=1706]
[2025/09/07 13:12:24.839 +00:00] [INFO] [S3RandomAccessFile.cpp:269] ["S3 GetObject timeout: max_retry=3, key=tiflash-root/s389908/data/ks_1_t_521/dmf_22530948/0.merged, elapsed=0.000s"] [source=s389908/data/ks_1_t_521/dmf_22530948/0.merged] [thread_id=1706]
[2025/09/07 13:12:24.839 +00:00] [ERROR] [S3RandomAccessFile.cpp:118] ["Cannot read from istream, size=2688 gcount=0 state=0x05 cur_offset=0 content_length=1767099 errno=2 errmsg=No such file or directory cost=414ns"] [source=s389908/data/ks_1_t_521/dmf_22530948/0.merged] [thread_id=1706]
[2025/09/07 13:12:24.839 +00:00] [INFO] [S3RandomAccessFile.cpp:269] ["S3 GetObject timeout: max_retry=3, key=tiflash-root/s389908/data/ks_1_t_521/dmf_22530948/0.merged, elapsed=0.000s"] [source=s389908/data/ks_1_t_521/dmf_22530948/0.merged] [thread_id=1706]
4. What is your TiFlash version? (Required)
master