Skip to content

Raft: Fix coprocessor ignore new added RegionReadStatus thus cause inconsistent result#10543

Merged
ti-chi-bot[bot] merged 2 commits intopingcap:masterfrom
JaySon-Huang:fix_inconsist
Nov 11, 2025
Merged

Raft: Fix coprocessor ignore new added RegionReadStatus thus cause inconsistent result#10543
ti-chi-bot[bot] merged 2 commits intopingcap:masterfrom
JaySon-Huang:fix_inconsist

Conversation

@JaySon-Huang
Copy link
Contributor

@JaySon-Huang JaySon-Huang commented Nov 10, 2025 •
edited
Loading

What problem does this PR solve?

Issue Number: close #10510

Problem Summary:

From the failure test logging, we can see that all of those failure read request meet an RegionException: ... message: Region error STALE_COMMAND(). #10494 adds new enum values including STALE_COMMAND in RegionException:: RegionReadStatus.

https://github.com/JaySon-Huang/tiflash/blob/eb407fdfdeb9865794868009245bb0e1adb20c0e/dbms/src/Storages/KVStore/Read/RegionException.h#L42-L44

endless-htap-consistency-chaos-tps-7973167-1-292

-- read index return stale_command
server.log:[2025/11/08 23:31:38.634 +08:00] [WARN] [LearnerReadWorker.cpp:332] ["meet abnormal region error region_error { message: \"stale command\" stale_command { } }, region_id=18056"] [source="Coprocessor, is_remote_read: true, start_ts: 462059181271678978, region_info: {18056, 7, 109}"] [thread_id=678]
server.log:[2025/11/08 23:31:38.634 +08:00] [INFO] [LearnerReadWorker.cpp:432] ["[Learner Read] Batch read index, num_regions=1 num_requests=1 num_stale_read=0 num_cached_index=0 n_unavailable=1 read_cost=1016ms start_ts=462059181271678978 unavailable_regions={ids=[{region_id:18056,s:STALE_COMMAND,msg:}] locks=[]}"] [source="Coprocessor, is_remote_read: true, start_ts: 462059181271678978, region_info: {18056, 7, 109}"] [thread_id=678]
server.log:[2025/11/08 23:31:38.634 +08:00] [INFO] [LearnerReadWorker.cpp:534] ["[Learner Read] Finish wait index and resolve locks, wait_cost=0ms n_regions=1 n_unavailable=1 start_ts=462059181271678978 unavailable_regions={ids=[{region_id:18056,s:STALE_COMMAND,msg:}] locks=[]}"] [source="Coprocessor, is_remote_read: true, start_ts: 462059181271678978, region_info: {18056, 7, 109}"] [thread_id=678]
server.log:[2025/11/08 23:31:38.634 +08:00] [DEBUG] [LearnerReadWorker.cpp:573] ["[Learner Read] Learner Read Summary, regions_info=(region_id=18056 to_wait=0 applied_index=18534 bypass_locks=[]) unavailable_regions={ids=[{region_id:18056,s:STALE_COMMAND,msg:}] locks=[]} start_ts=462059181271678978"] [source="Coprocessor, is_remote_read: true, start_ts: 462059181271678978, region_info: {18056, 7, 109}"] [thread_id=678]
server.log:[2025/11/08 23:31:38.634 +08:00] [WARN] [CoprocessorHandler.cpp:239] ["RegionException: region_id=18056, message: Region error STALE_COMMAND()"] [source="Coprocessor, is_remote_read: true, start_ts: 462059181271678978, region_info: {18056, 7, 109}"] [thread_id=678]

endless-htap-consistency-chaos-tps-7972869-1-937

server.log:[2025/11/08 03:40:24.408 +08:00] [DEBUG] [LearnerReadWorker.cpp:405] ["[Learner Read] Start read index, start_ts=462040445410344962 num_regions=1"] [source="Coprocessor, is_remote_read: true, start_ts: 462040445410344962, region_info: {17930, 7, 84}"] [thread_id=993]
server.log:[2025/11/08 03:40:25.885 +08:00] [WARN] [LearnerReadWorker.cpp:332] ["meet abnormal region error region_error { message: \"stale command\" stale_command { } }, region_id=17930"] [source="Coprocessor, is_remote_read: true, start_ts: 462040445410344962, region_info: {17930, 7, 84}"] [thread_id=993]
server.log:[2025/11/08 03:40:25.885 +08:00] [INFO] [LearnerReadWorker.cpp:432] ["[Learner Read] Batch read index, num_regions=1 num_requests=1 num_stale_read=0 num_cached_index=0 n_unavailable=1 read_cost=1477ms start_ts=462040445410344962 unavailable_regions={ids=[{region_id:17930,s:STALE_COMMAND,msg:}] locks=[]}"] [source="Coprocessor, is_remote_read: true, start_ts: 462040445410344962, region_info: {17930, 7, 84}"] [thread_id=993]
server.log:[2025/11/08 03:40:25.885 +08:00] [INFO] [LearnerReadWorker.cpp:534] ["[Learner Read] Finish wait index and resolve locks, wait_cost=0ms n_regions=1 n_unavailable=1 start_ts=462040445410344962 unavailable_regions={ids=[{region_id:17930,s:STALE_COMMAND,msg:}] locks=[]}"] [source="Coprocessor, is_remote_read: true, start_ts: 462040445410344962, region_info: {17930, 7, 84}"] [thread_id=993]
server.log:[2025/11/08 03:40:25.885 +08:00] [DEBUG] [LearnerReadWorker.cpp:573] ["[Learner Read] Learner Read Summary, regions_info=(region_id=17930 to_wait=0 applied_index=12790 bypass_locks=[462040445318594567]) unavailable_regions={ids=[{region_id:17930,s:STALE_COMMAND,msg:}] locks=[]} start_ts=462040445410344962"] [source="Coprocessor, is_remote_read: true, start_ts: 462040445410344962, region_info: {17930, 7, 84}"] [thread_id=993]
server.log:[2025/11/08 03:40:25.885 +08:00] [WARN] [CoprocessorHandler.cpp:239] ["RegionException: region_id=17930, message: Region error STALE_COMMAND()"] [source="Coprocessor, is_remote_read: true, start_ts: 462040445410344962, region_info: {17930, 7, 84}"] [thread_id=993]

endless-htap-consistency-chaos-tps-7911630-1-667

server.log:[2025/11/08 02:34:11.944 +08:00] [DEBUG] [LearnerReadWorker.cpp:405] ["[Learner Read] Start read index, start_ts=462039392040517633 num_regions=1"] [source="Coprocessor, is_remote_read: true, start_ts: 462039392040517633, region_info: {17945, 7, 103}"] [thread_id=1000]
server.log:[2025/11/08 02:34:11.946 +08:00] [WARN] [LearnerReadWorker.cpp:332] ["meet abnormal region error region_error { message: \"stale command\" stale_command { } }, region_id=17945"] [source="Coprocessor, is_remote_read: true, start_ts: 462039392040517633, region_info: {17945, 7, 103}"] [thread_id=1000]
server.log:[2025/11/08 02:34:11.946 +08:00] [INFO] [LearnerReadWorker.cpp:432] ["[Learner Read] Batch read index, num_regions=1 num_requests=1 num_stale_read=0 num_cached_index=0 n_unavailable=1 read_cost=1ms start_ts=462039392040517633 unavailable_regions={ids=[{region_id:17945,s:STALE_COMMAND,msg:}] locks=[]}"] [source="Coprocessor, is_remote_read: true, start_ts: 462039392040517633, region_info: {17945, 7, 103}"] [thread_id=1000]
server.log:[2025/11/08 02:34:11.946 +08:00] [INFO] [LearnerReadWorker.cpp:534] ["[Learner Read] Finish wait index and resolve locks, wait_cost=0ms n_regions=1 n_unavailable=1 start_ts=462039392040517633 unavailable_regions={ids=[{region_id:17945,s:STALE_COMMAND,msg:}] locks=[]}"] [source="Coprocessor, is_remote_read: true, start_ts: 462039392040517633, region_info: {17945, 7, 103}"] [thread_id=1000]
server.log:[2025/11/08 02:34:11.946 +08:00] [DEBUG] [LearnerReadWorker.cpp:573] ["[Learner Read] Learner Read Summary, regions_info=(region_id=17945 to_wait=0 applied_index=5 bypass_locks=[]) unavailable_regions={ids=[{region_id:17945,s:STALE_COMMAND,msg:}] locks=[]} start_ts=462039392040517633"] [source="Coprocessor, is_remote_read: true, start_ts: 462039392040517633, region_info: {17945, 7, 103}"] [thread_id=1000]
server.log:[2025/11/08 02:34:11.946 +08:00] [WARN] [CoprocessorHandler.cpp:239] ["RegionException: region_id=17945, message: Region error STALE_COMMAND()"] [source="Coprocessor, is_remote_read: true, start_ts: 462039392040517633, region_info: {17945, 7, 103}"] [thread_id=1000]

What is changed and how it works?

* Handling new added RegionReadStatus enum values for error response of coprocessor
  * Do not set the `default` branch for switch (e.status) so that the compiler will throw an error if there is not covered new enum values

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

None

@JaySon-Huang
Add new enum handler for handling region error of coprocessor
1b272df 
Signed-off-by: JaySon-Huang <tshent@qq.com>
@JaySon-Huang
Add unit test
8f5481f 
Signed-off-by: JaySon-Huang <tshent@qq.com>
@ti-chi-bot ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. needs-cherry-pick-release-8.5 Should cherry pick this PR to release-8.5 branch. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Nov 10, 2025
@JaySon-Huang
Copy link
Contributor Author

JaySon-Huang commented Nov 10, 2025 •
edited
Loading

/hold
wait for internal testing

@ti-chi-bot ti-chi-bot bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 10, 2025
JaySon-Huang
JaySon-Huang commented Nov 10, 2025
View reviewed changes
dbms/src/Storages/KVStore/Read/RegionException.cpp
Comment on lines +47 to +48
case RegionException::RegionReadStatus::STALE_COMMAND:
case RegionException::RegionReadStatus::STORE_NOT_MATCH:
Copy link
Contributor Author

@JaySon-Huang JaySon-Huang Nov 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The main changes is adding handling these two enum values

@JaySon-Huang
Copy link
Contributor Author

JaySon-Huang commented Nov 11, 2025

/unhold

internal testing passed

image

@ti-chi-bot ti-chi-bot bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 11, 2025
CalvinNeo
CalvinNeo approved these changes Nov 11, 2025
View reviewed changes
Copy link
Member

@CalvinNeo CalvinNeo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@ti-chi-bot ti-chi-bot bot added needs-1-more-lgtm Indicates a PR needs 1 more LGTM. approved labels Nov 11, 2025
@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Nov 11, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: CalvinNeo, Lloyd-Pottiger

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:
  • OWNERS [CalvinNeo,Lloyd-Pottiger]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Nov 11, 2025
@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Nov 11, 2025

[LGTM Timeline notifier]

Timeline:

  • 2025-11-11 08:26:19.363741874 +0000 UTC m=+777028.806771753: ☑️ agreed by CalvinNeo.
  • 2025-11-11 08:27:29.1934352 +0000 UTC m=+777098.636465079: ☑️ agreed by Lloyd-Pottiger.

@ti-chi-bot ti-chi-bot bot merged commit e84ee46 into pingcap:master Nov 11, 2025
7 checks passed
@ti-chi-bot
Copy link
Member

ti-chi-bot commented Nov 11, 2025

In response to a cherrypick label: new pull request created to branch release-8.5: #10544.
But this PR has conflicts, please resolve them!

ti-chi-bot pushed a commit to ti-chi-bot/tiflash that referenced this pull request Nov 11, 2025
@JaySon-Huang @ti-chi-bot
This is an automated cherry-pick of pingcap#10543
b106b93 
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot ti-chi-bot mentioned this pull request Nov 11, 2025
Merged
12 tasks
@JaySon-Huang JaySon-Huang deleted the fix_inconsist branch November 11, 2025 13:52
@JaySon-Huang
Copy link
Contributor Author

JaySon-Huang commented Nov 11, 2025

/cherry-pick release-nextgen-20251011

@ti-chi-bot ti-chi-bot mentioned this pull request Nov 11, 2025
Merged
12 tasks
@ti-chi-bot
Copy link
Member

ti-chi-bot commented Nov 11, 2025

@JaySon-Huang: new pull request created to branch release-nextgen-20251011: #10545.

Details

In response to this:

/cherry-pick release-nextgen-20251011

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot bot pushed a commit that referenced this pull request Nov 12, 2025
@ti-chi-bot @JaySon-Huang
Raft: Fix coprocessor ignore new added RegionReadStatus thus cause in…
7c971f7 
…consistent result (#10543) (#10545)

close #10510

* Handling new added RegionReadStatus enum values for error response of coprocessor
  * Do not set the `default` branch for switch (e.status) so that the compiler will throw an error if there is not covered new enum values

Signed-off-by: JaySon-Huang <tshent@qq.com>

Co-authored-by: JaySon-Huang <tshent@qq.com>
ti-chi-bot bot pushed a commit that referenced this pull request Nov 28, 2025
@ti-chi-bot @JaySon-Huang
Raft: Fix coprocessor ignore new added RegionReadStatus thus cause in…
b50ff41 
…consistent result (#10543) (#10544)

close #10510

* Handling new added RegionReadStatus enum values for error response of coprocessor
  * Do not set the `default` branch for switch (e.status) so that the compiler will throw an error if there is not covered new enum values

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
Signed-off-by: JaySon-Huang <tshent@qq.com>

Co-authored-by: JaySon <tshent@qq.com>
Co-authored-by: JaySon-Huang <tshent@qq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@CalvinNeo CalvinNeo CalvinNeo approved these changes

@Lloyd-Pottiger Lloyd-Pottiger Lloyd-Pottiger approved these changes

@JinheLin JinheLin Awaiting requested review from JinheLin

@gengliqi gengliqi Awaiting requested review from gengliqi

Assignees

No one assigned

Labels

approved lgtm needs-cherry-pick-release-8.5 Should cherry pick this PR to release-8.5 branch. release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Consistency test failed

4 participants

@JaySon-Huang @ti-chi-bot @CalvinNeo @Lloyd-Pottiger