kv: introduce new "max outstanding size" setting to txnPipeliner#35014
Conversation
|
This change is |
tbg
left a comment
There was a problem hiding this comment.
Reviewed 4 of 4 files at r1.
Reviewable status:complete! 0 of 0 LGTMs obtained (waiting on @andreimatei and @nvanbenschoten)
pkg/kv/txn_interceptor_pipeliner.go, line 533 at r1 (raw file):
} tp.owSizeBytes -= int64(len(key)) }
Is there a convenient places for an assertion somewhere? The silly example I can think of is EndTransaction where Key is set kind of late, though that isn't something the pipeliner would track.
andreimatei
left a comment
There was a problem hiding this comment.
LGTM
Reviewable status:
pkg/kv/txn_interceptor_pipeliner.go, line 36 at r1 (raw file):
) var pipelinedWritesMaxOutstandingSize = settings.RegisterIntSetting( // TODO(nvanbenschoten): The need for this extra setting alongside
I'd say the existence of this setting indicates the need for background proving of intents, not necessarily anything to do with the other setting... no?
pkg/kv/txn_interceptor_pipeliner.go, line 152 at r1 (raw file):
outstandingWrites *btree.BTree owSizeBytes int64
pls comment the field
pkg/kv/txn_interceptor_pipeliner.go, line 212 at r1 (raw file):
asyncConsensus := pipelinedWritesEnabled.Get(&tp.st.SV) && !tp.disabled // We provide a setting to bound the size of outstanding writes that the
why declare these variable with such a broad scope? They're only needed in a deeply nested place.
65a8fff to
1c667ff
Compare
nvb
left a comment
There was a problem hiding this comment.
Reviewable status:
pkg/kv/txn_interceptor_pipeliner.go, line 36 at r1 (raw file):
Previously, andreimatei (Andrei Matei) wrote…
I'd say the existence of this setting indicates the need for background proving of intents, not necessarily anything to do with the other setting... no?
Probably both, but the point I'm making here is that the need to bound the memory footprint of two separate in-memory structures that track very similar things demonstrates a clear area for simplification.
pkg/kv/txn_interceptor_pipeliner.go, line 152 at r1 (raw file):
Previously, andreimatei (Andrei Matei) wrote…
pls comment the field
Done.
pkg/kv/txn_interceptor_pipeliner.go, line 212 at r1 (raw file):
Previously, andreimatei (Andrei Matei) wrote…
why declare these variable with such a broad scope? They're only needed in a deeply nested place.
We modify owSizeBytes in each iteration though, and I don't want the atomic access in the loop.
pkg/kv/txn_interceptor_pipeliner.go, line 533 at r1 (raw file):
Previously, tbg (Tobias Grieger) wrote…
Is there a convenient places for an assertion somewhere? The silly example I can think of is
EndTransactionwhereKeyis set kind of late, though that isn't something the pipeliner would track.
Done.
1c667ff to
d9f370b
Compare
Fixes cockroachdb#32522. This change creates a new cluster setting called: ``` kv.transaction.write_pipelining_max_outstanding_size ``` It limits the size in bytes that can be dedicated to tracking in-flight writes that have been pipelined. Once this limit is hit, not more writes will be pipelined by a transaction until some of the writes are proven and removed from the outstanding write set. This change once again illustrates the need for periodic proving of outstanding writes. We touch upon that in the type definition's comment and in cockroachdb#35009. Release note: None
d9f370b to
0dc8b84
Compare
|
bors r+ |
35014: kv: introduce new "max outstanding size" setting to txnPipeliner r=nvanbenschoten a=nvanbenschoten Fixes #32522. This change creates a new cluster setting called: ``` kv.transaction.write_pipelining_max_outstanding_size ``` It limits the size in bytes that can be dedicated to tracking in-flight writes that have been pipelined. Once this limit is hit, not more writes will be pipelined by a transaction until some of the writes are proven and removed from the outstanding write set. This change once again illustrates the need for periodic proving of outstanding writes. We touch upon that in the type definition's comment and in #35009. Release note: None 35199: log: fix remaining misuse of runtime.Callers/runtime.FuncForPC r=nvanbenschoten a=nvanbenschoten Fixes #17770. This commit fixes the last user of `runtime.Callers` that misused the stdlib function by translating the PC values it returned directly into symbolic information (see https://golang.org/pkg/runtime/#Callers) [1]. Go's documentation warns that this is a recipe for disaster when mixed with mid-stack inlining. The other concern in #17770 was this comment: #17770 (comment). This was discussed in golang/go#29582 and addressed in golang/go@956879d. An alternative would be to use `runtime.Caller` here, but that would force an allocation that was hard-earned in #29017. Instead, this commit avoids any performance hit. ``` name old time/op new time/op delta Header-4 267ns ± 1% 268ns ± 0% ~ (p=0.584 n=10+10) VDepthWithVModule-4 260ns ± 3% 255ns ± 1% -1.87% (p=0.018 n=10+9) name old alloc/op new alloc/op delta Header-4 0.00B 0.00B ~ (all equal) VDepthWithVModule-4 0.00B 0.00B ~ (all equal) name old allocs/op new allocs/op delta Header-4 0.00 0.00 ~ (all equal) VDepthWithVModule-4 0.00 0.00 ~ (all equal) ``` [1] I went through and verified that this was still correct. Release note: None 35203: closedts: react faster to config changes r=danhhz a=tbg This approximately halves the duration of `make test PKG=./pkg/ccl/changefeedccl TESTS=/sinkless`, from ~60s to ~30s. Touches #34455. Release note: None Co-authored-by: Nathan VanBenschoten <nvanbenschoten@gmail.com> Co-authored-by: Tobias Schottdorf <tobias.schottdorf@gmail.com>
Build succeeded |
Fixes #32522.
This change creates a new cluster setting called:
It limits the size in bytes that can be dedicated to tracking in-flight
writes that have been pipelined. Once this limit is hit, not more writes
will be pipelined by a transaction until some of the writes are proven
and removed from the outstanding write set.
This change once again illustrates the need for periodic proving of
outstanding writes. We touch upon that in the type definition's comment
and in #35009.
Release note: None