Skip to content

add retry info to request source#953

Merged
cfzjywxk merged 5 commits intotikv:masterfrom
you06:req-source-detail
Aug 29, 2023
Merged

add retry info to request source#953
cfzjywxk merged 5 commits intotikv:masterfrom
you06:req-source-detail

Conversation

@you06
Copy link
Contributor

@you06 you06 commented Aug 24, 2023

image

As the image shows, this PR will change the request srouce when there is retry.

The first request's request source will be set to {request source}-{retry}_{stale}_{replica type}.

The retried request's request soruce will be set to:

  • {request source}-{replica type} for the first non-stale request.
  • {request source}-stale_{replica type} for the first stale request.
  • {request source}-retry_{first replica type}_{replica type} for the retried non-stale request, the first tried replica type will be recorded.
  • {request source}-retry_stale_{first replica type} for the retried stale request, the first tried replica type will be recorded. Even its stale flag is removed, the retry info still contain it.

e.g.

  • external_Select-stale_leader stands for first request with stale read sent to leader.
  • external_Select-stale_follower stands for first request with stale read sent to follower.
  • external_Select-retry_stale_follower_leader stands for retried stale request to leader read, and it's fallback from follower. In the figure, because leader's safe ts is fresh enough to server the stale read, so the retried requests are fallback from followe.

When meeting key error, it the retry information is also recorded, the get and batch get test with lock error injection:

image

Signed-off-by: you06 <you1474600@gmail.com>
@cfzjywxk
Copy link
Contributor

@crazycs520 PTAL

return nil, nil, retryTimes, err
}
rpcCtx.contextPatcher.applyTo(&req.Context)
patchRequestSource(req, requestSource, readType, retryTimes)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If finally "nil, nil" is returned, the next round RegionRequestSender.SendReqCtx retry, stale flag would not be considered again, in this case it seems the readType may not represent the original read type.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I change the implemention and it now handles the retry from upper layer.

Signed-off-by: you06 <you1474600@gmail.com>
Signed-off-by: you06 <you1474600@gmail.com>
// The initial read type.
ReadType string
// InputRequestSource is the input source of the request, if it's not empty, the final RequestSource sent to store will be attached with the retry info.
InputRequestSource string
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would the InputRequestSource of coprocessor related requests be set accordingly in tidb?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Coprocessor code also need to be updated.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, let's merge the client-go part first

Copy link
Contributor

@cfzjywxk cfzjywxk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Signed-off-by: you06 <you1474600@gmail.com>
@cfzjywxk
Copy link
Contributor

The retry flag should also be handled when key error is returned to the snapshot or cop send request layer, setting req.IsRetryRequest is not enough for such cases.

Signed-off-by: you06 <you1474600@gmail.com>
@you06
Copy link
Contributor Author

you06 commented Aug 28, 2023

The retry flag should also be handled when key error is returned to the snapshot or cop send request layer, setting req.IsRetryRequest is not enough for such cases.

Updated and added the test result in the end of description.

@cfzjywxk
Copy link
Contributor

@you06
If there's no "stale" word in the description, does it mean this is not a stale request but it's retried for some reason?

@you06
Copy link
Contributor Author

you06 commented Aug 29, 2023

If there's no "stale" word in the description, does it mean this is not a stale request but it's retried for some reason?

Yes, if there is no "stale" word, it's not a stale read request.

There are two figures in the description, the first one is tested with stale read and the second one is not.

@cfzjywxk cfzjywxk merged commit 295094e into tikv:master Aug 29, 2023
you06 added a commit to you06/client-go that referenced this pull request Sep 4, 2023
* add retry info to request source

Signed-off-by: you06 <you1474600@gmail.com>

* handle upper layer retry

Signed-off-by: you06 <you1474600@gmail.com>

* stabilize test

Signed-off-by: you06 <you1474600@gmail.com>

* retry in 3 dimension

Signed-off-by: you06 <you1474600@gmail.com>

* record and restore req.ReadType

Signed-off-by: you06 <you1474600@gmail.com>

---------

Signed-off-by: you06 <you1474600@gmail.com>
disksing pushed a commit that referenced this pull request Sep 5, 2023
Signed-off-by: you06 <you1474600@gmail.com>
@nolouch
Copy link
Contributor

nolouch commented Sep 6, 2023

{request source}-{retry}_{stale}_{replica type}, breaks the dependency for the background control:

https://github.com/tikv/pd/blob/41eae1a0a306d763c505f2cadaaec40795c120bc/client/resource_group/controller/controller.go#L490-L511

can we adjust to like : {retry}_{stale}_{replica type}_{request source}?

you06 added a commit to you06/client-go that referenced this pull request Sep 15, 2023
Signed-off-by: you06 <you1474600@gmail.com>
you06 added a commit that referenced this pull request Sep 15, 2023
add retry info to request source (#953) (#959)
you06 added a commit to you06/client-go that referenced this pull request Oct 12, 2023
Signed-off-by: you06 <you1474600@gmail.com>
cfzjywxk pushed a commit that referenced this pull request Oct 12, 2023
* add retry info to request source (#953)

Signed-off-by: you06 <you1474600@gmail.com>

* set the request source at the last section (#960)

Signed-off-by: you06 <you1474600@gmail.com>

---------

Signed-off-by: you06 <you1474600@gmail.com>
}

if e := tikvrpc.SetContext(req, rpcCtx.Meta, rpcCtx.Peer); e != nil {
return nil, nil, retryTimes, err
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should it return e rather than err?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants