kv: include conflicting request information in latch manager traces/logs#115511
kv: include conflicting request information in latch manager traces/logs#115511craig[bot] merged 1 commit intocockroachdb:masterfrom
Conversation
|
Thank you for contributing to CockroachDB. Please ensure you have followed the guidelines for creating a PR. Before a member of our team reviews your PR, I have some potential action items for you:
I have added a few people who may be able to assist in reviewing:
🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
338ebdf to
3bdb2d1
Compare
|
Thank you for updating your pull request. Before a member of our team reviews your PR, I have some potential action items for you:
🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
lyang24
left a comment
There was a problem hiding this comment.
Reviewable status:
complete! 0 of 0 LGTMs obtained (waiting on @nvanbenschoten)
pkg/kv/kvserver/concurrency/concurrency_control.go line 438 at r1 (raw file):
// We store a reference to kvpb.BatchRequest. This is used to enrich logging // when latches conflict. BatchRequest *kvpb.BatchRequest
probably not good to have duplicates (i.e. two mutable reference of transaction from req.Txn or req.BatchRequest.Txn). However, I think this is a safer change just in case BatchRequest changed downstream.
pkg/kv/kvserver/spanlatch/manager.go line 208 at r1 (raw file):
latch := &latches[i] latch.span = ss[i].Span latch.guard = guard
want to careful about potential npe here.
The old model is both latch and guard helds the same location to the signal in memory. In the new model this process become latch -> guard -> signal which requires guards lifetime >= latch's lifetime to avoid nil pointers - I think this statement is true because when latchGuard is releasing the latch also get deleted from the btree first?
3bdb2d1 to
638cac7
Compare
|
Thank you for updating your pull request. Before a member of our team reviews your PR, I have some potential action items for you:
🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
638cac7 to
b13ec82
Compare
|
Thank you for updating your pull request. Before a member of our team reviews your PR, I have some potential action items for you:
🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
b13ec82 to
f65ed24
Compare
|
Thank you for updating your pull request. Before a member of our team reviews your PR, I have some potential action items for you:
🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
f65ed24 to
bca4dd0
Compare
|
Thank you for updating your pull request. Before a member of our team reviews your PR, I have some potential action items for you:
🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
lyang24
left a comment
There was a problem hiding this comment.
Reviewable status:
complete! 0 of 0 LGTMs obtained (waiting on @nvanbenschoten)
pkg/kv/kvserver/concurrency/testdata/concurrency_manager/shared_locks_latches line 172 at r2 (raw file):
# exclusive_lock(ts) < shared_lock(ts) new-request name=req10 txn=txn3 ts=9,1 get key=a str=exclusive
get request with exclusive strength is interesting
|
Hi @nvanbenschoten , I rebased this pr to the recent master that reflects the refactor of request safeformat. Do you mind take a look at this pr when you have time? |
nvb
left a comment
There was a problem hiding this comment.
Reviewed 2 of 19 files at r1, 18 of 18 files at r2, all commit messages.
Reviewable status:complete! 0 of 0 LGTMs obtained (waiting on @lyang24)
pkg/kv/kvserver/concurrency/concurrency_control.go line 438 at r1 (raw file):
Previously, lyang24 (Lanqing Yang) wrote…
probably not good to have duplicates (i.e. two mutable reference of transaction from req.Txn or req.BatchRequest.Txn). However, I think this is a safer change just in case BatchRequest changed downstream.
I think this is ok as long as we change this to not be a direct reference to BatchRequest. Instead, we could make this a more narrow interface. For instance, it could just be RequestFormatter redact.SafeFormatter.
#116696 will be helpful for this.
pkg/kv/kvserver/spanlatch/manager.go line 208 at r1 (raw file):
Previously, lyang24 (Lanqing Yang) wrote…
want to careful about potential npe here.
The old model is both latch and guard helds the same location to the signal in memory. In the new model this process become latch -> guard -> signal which requires guards lifetime >= latch's lifetime to avoid nil pointers - I think this statement is true because when latchGuard is releasing the latch also get deleted from the btree first?
signals is embedded in Guard, so the models are actually equivalent in that latch is pointing into memory in the Guard. The primary difference is that now it's pointing at the entire Guard struct instead of just field in it.
pkg/kv/kvserver/spanlatch/manager.go line 88 at r2 (raw file):
// of a single key span. type latch struct { guard *Guard
nit: let's just call this g.
pkg/kv/kvserver/spanlatch/manager.go line 108 at r2 (raw file):
w.Printf("%s@%s", la.span, la.ts) if la.guard != nil && la.guard.batchRequest != nil { w.Printf("[")
Instead of bracketing this, how about:
w.Printf(" for request ")
la.guard.requestFormatter.SafeFormat(w, verb)pkg/kv/kvserver/concurrency/testdata/concurrency_manager/shared_locks_latches line 172 at r2 (raw file):
Previously, lyang24 (Lanqing Yang) wrote…
get request with exclusive strength is interesting
It is! This is the request type that powers SELECT FOR UPDATE.
pkg/kv/kvserver/spanlatch/manager_test.go line 755 at r2 (raw file):
} // Unit test the output of latch SafeFormat
Nice test!
nit: // TestLatchStringAndSafeformat tests the output of latch.SafeFormat.
bca4dd0 to
1121ff3
Compare
|
Thank you for updating your pull request. Before a member of our team reviews your PR, I have some potential action items for you:
🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
lyang24
left a comment
There was a problem hiding this comment.
thanks for reviewing
Reviewable status:
complete! 0 of 0 LGTMs obtained (waiting on @nvanbenschoten)
pkg/kv/kvserver/concurrency/concurrency_control.go line 438 at r1 (raw file):
Previously, nvanbenschoten (Nathan VanBenschoten) wrote…
I think this is ok as long as we change this to not be a direct reference to
BatchRequest. Instead, we could make this a more narrow interface. For instance, it could just beRequestFormatter redact.SafeFormatter.#116696 will be helpful for this.
Yes redact.SafeFormatter is a much better fit will wait for #116696 to merge and update
pkg/kv/kvserver/spanlatch/manager.go line 208 at r1 (raw file):
Previously, nvanbenschoten (Nathan VanBenschoten) wrote…
signalsis embedded inGuard, so the models are actually equivalent in thatlatchis pointing into memory in theGuard. The primary difference is that now it's pointing at the entireGuardstruct instead of just field in it.
Great I learned about Struct Embedding in golang, very cool!
1121ff3 to
41c1f29
Compare
|
Thank you for updating your pull request. Before a member of our team reviews your PR, I have some potential action items for you:
🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
41c1f29 to
2eb7a7b
Compare
|
Thank you for updating your pull request. Before a member of our team reviews your PR, I have some potential action items for you:
🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
2eb7a7b to
0264b7f
Compare
|
Thank you for updating your pull request. Before a member of our team reviews your PR, I have some potential action items for you:
🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
|
Thank you for updating your pull request. Before a member of our team reviews your PR, I have some potential action items for you:
🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
|
seems like this pr had a small regression on benchmarks :(. |
8bf4b84 to
510bbad
Compare
|
Thank you for updating your pull request. Before a member of our team reviews your PR, I have some potential action items for you:
🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
nvb
left a comment
There was a problem hiding this comment.
Thanks for running these benchmarks! A regression of that magnitude is expected and acceptable, given the improvement this has on observability. We're adding 16 bytes (an interface header) to the Guard struct's size.
Reviewed 1 of 13 files at r3, 8 of 9 files at r5, 1 of 1 files at r6, all commit messages.
Reviewable status:complete! 0 of 0 LGTMs obtained (and 1 stale) (waiting on @lyang24)
pkg/kv/kvserver/concurrency/concurrency_control.go line 438 at r1 (raw file):
Previously, lyang24 (Lanqing Yang) wrote…
Yes
redact.SafeFormatteris a much better fit will wait for #116696 to merge and update
This turned out nicely.
pkg/kv/kvserver/concurrency/concurrency_control.go line 437 at r6 (raw file):
LockSpans *lockspanset.LockSpanSet // The SafeFormatter from batchRequest pointer. This is used to enrich logging
"The SafeFormatter capable of formatting the request."
pkg/kv/kvserver/concurrency/concurrency_control.go line 439 at r6 (raw file):
// The SafeFormatter from batchRequest pointer. This is used to enrich logging // with request level information when latches conflict. BaFormatter redact.SafeFormatter
nit: consider switching the name of this field from BaFormatter to BaFmt. And then do the same in the functions where this is passed.
pkg/kv/kvserver/spanlatch/manager.go line 140 at r6 (raw file):
// checking of conflicts, and waiting. snap *snapshot baFormatter redact.SafeFormatter
nit: move this up to right below pp.
pkg/kv/kvserver/concurrency/concurrency_manager_test.go line 223 at r6 (raw file):
LockSpans: lockSpans, PoisonPolicy: pp, BaFormatter: &ba,
small nit: let's just make ba := &kvpb.BatchRequest{} above.
510bbad to
a9aa79c
Compare
|
Thank you for updating your pull request. Before a member of our team reviews your PR, I have some potential action items for you:
🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
a9aa79c to
34fa348
Compare
|
Thank you for updating your pull request. Before a member of our team reviews your PR, I have some potential action items for you:
🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
lyang24
left a comment
There was a problem hiding this comment.
Thank you for reviewing!
Reviewable status:
complete! 0 of 0 LGTMs obtained (and 1 stale) (waiting on @nvanbenschoten)
pkg/kv/kvserver/concurrency/concurrency_manager_test.go line 223 at r6 (raw file):
Previously, nvanbenschoten (Nathan VanBenschoten) wrote…
small nit: let's just make
ba := &kvpb.BatchRequest{}above.
wow thanks for finding this, I applied to all the related test touched by this commit
nvb
left a comment
There was a problem hiding this comment.
Reviewed 5 of 9 files at r7, all commit messages.
Reviewable status:complete! 0 of 0 LGTMs obtained (and 1 stale) (waiting on @lyang24)
pkg/kv/kvserver/concurrency/concurrency_control.go line 439 at r6 (raw file):
Previously, nvanbenschoten (Nathan VanBenschoten) wrote…
nit: consider switching the name of this field from
BaFormattertoBaFmt. And then do the same in the functions where this is passed.
Sorry, for parameters and unexported fields (in Guard), we'd want baFmt.
pkg/kv/kvserver/spanlatch/manager_test.go line 749 at r8 (raw file):
// TestSizeOfLatch tests the size of the latch struct. func TestSizeOfLatch(t *testing.T) {
While we're here, mind adding one of these tests for Guard as well?
34fa348 to
db13a02
Compare
|
Thank you for updating your pull request. Before a member of our team reviews your PR, I have some potential action items for you:
🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
lyang24
left a comment
There was a problem hiding this comment.
Thank you for speedy review i addressed the comments.
Reviewable status:
complete! 0 of 0 LGTMs obtained (and 1 stale) (waiting on @nvanbenschoten)
nvb
left a comment
There was a problem hiding this comment.
Reviewed 3 of 9 files at r7, 2 of 2 files at r9, all commit messages.
Reviewable status:complete! 0 of 0 LGTMs obtained (and 1 stale) (waiting on @lyang24)
pkg/kv/kvserver/concurrency/concurrency_control.go line 439 at r6 (raw file):
Previously, nvanbenschoten (Nathan VanBenschoten) wrote…
Sorry, for parameters and unexported fields (in
Guard), we'd wantbaFmt.
For function parameters as well (e.g. in newGuard). Variable names are rarely capitalized in Go.
db13a02 to
d11f713
Compare
|
Thank you for updating your pull request. Before a member of our team reviews your PR, I have some potential action items for you:
🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
lyang24
left a comment
There was a problem hiding this comment.
Reviewable status:
complete! 0 of 0 LGTMs obtained (and 1 stale) (waiting on @nvanbenschoten)
pkg/kv/kvserver/concurrency/concurrency_control.go line 439 at r6 (raw file):
Previously, nvanbenschoten (Nathan VanBenschoten) wrote…
For function parameters as well (e.g. in
newGuard). Variable names are rarely capitalized in Go.
my bad
nvb
left a comment
There was a problem hiding this comment.
Thanks @lyang24! I'll merge this once CI goes green.
Reviewed 3 of 3 files at r10, all commit messages.
Reviewable status:complete! 1 of 0 LGTMs obtained (waiting on @lyang24)
d11f713 to
0bee58b
Compare
|
Thank you for updating your pull request. Before a member of our team reviews your PR, I have some potential action items for you:
🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
This commit logs latch waiter and latch holder's request information when latch conflicts. This is achieved by adding redact.SafeFormatter from batch request into concurrency.Request and pass it down to latch Guard. Since the Guard struct embedded the signal struct, we will store store a pointer to the latch guard in latch struct instead. Fixes: cockroachdb#114601 Relase note: None
0bee58b to
ba13697
Compare
|
Thank you for updating your pull request. Before a member of our team reviews your PR, I have some potential action items for you:
🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
|
bors r+ |
|
Build succeeded: |
This commit logs latch waiter and latch holder's request information when latch
conflicts. This is achieved by adding batch request derived redact.SafeFormatter
into concurrency.Request and pass it down to latch Guard. Since the Guard struct
embedded the signal struct, we will store store a pointer to the latch guard
in latch struct instead.
Fixes: #114601
Relase note: None