[DNM] add actual keys read to command queue by tbg · Pull Request #10084 · cockroachdb/cockroach

tbg · 2016-10-19T14:56:39Z

The code is a mess and not meant to go in; don't bother leaving style
comments. Look only at the second commit.

This is an exploration for #6290. Briefly put, evaluation of commands
before proposing their result to Raft exercises the command queue in ways
the current code did not (which executes everything serially because it's
on the Raft goroutine).

In particular, the command queue now has to live up to its role of allowing
only commands which can be parallelized to be evaluated (and subsequently
proposed and applied).

What the command queue tracked so far are simply the main key ranges of the
requests in the batch, but that is not nearly enough.

To err on the side of being conservative, this PR presents an (unmergeably
hacky) attempt to classify what keys really need to be protected, as
measured by our testing coverage. As such, this list is not complete and in
particular it does not take into account reads of Replica state where the
in-memory cache is hit (though that could be caught by not accessing state
directly but having it go to disk every time; not done yet).

Major takeaways are:

Splits need to lock the key range of the RHS descriptor for writes (we
can
get a little tricky with reads), but also touches all of the abort cache
for
reading. It also accesses the Replica state for the RHS, though that's
a lesser concern.
Every command needs to read FrozenStatus, RangeLastGCKey and writes
MVCCStats. Unless we are smart about it, that already completely kills off
any proposal parallelism. We will have to be smart about it because that's
pretty bad. Especially the stats update is interesting because that is
both
frequent and a write operation. The obvious solution here is to make use
of
the commutativity of stats delta updates (though ContainsEstimates needs
to
be taken into account conservatively).
Every transaction reads its AbortCacheKey and reads or writes its
transaction
record (see discussion in storage: implement proposer-evaluated kv #6290 on why the latter makes sense).

The proposal's MaxLeaseIndex already makes sure that when we submit command
A to Raft before command B, then they can only apply in that order (mod
refurbishments, which can reorder and cause problems to this arguments,
though I think they can be overcome). So we could imagine a world in which
the command queue wouldn't commonly block commands on top of each other:

a command arrives.
checks the command queue; finds that it must wait for some prior commands
to
apply.
instead of waiting, it evaluates these commands in the correct order and
applies itself on top of that, entering itself in the command queue but
immediately proposing to Raft a command whose successful application is
contingent upon the previous commands applying as planned.
once the command applies, leave the command queue (as usual)
if the command does not apply (i.e. it or its dependencies got
reordered),
leave the command queue and enter it anew (instead of the previous
internal
refurbishment cycle).

The above is fairly half baked and challenging to implement, but so is
tracking every command's implicit writes; the above idea would in principle
remove any latency that isn't Raft or doing actual work, which is a great
benefit.

This change is

petermattis · 2016-10-19T15:15:34Z

Every command needs to read FrozenStatus, RangeLastGCKey and writes MVCCStats.

For the MVCCStats writes, I was assuming that was why the proposal was containing a delta.

Why does every write op have to read RangeLastGCKey?

petermattis · 2016-10-19T15:19:38Z

instead of waiting, it evaluates these commands in the correct order and applies itself on top of that, entering itself in the command queue but immediately proposing to Raft a command whose successful application is contingent upon the previous commands applying as planned.

This is tricky because in order to evaluate a command on top of a previous command we would need to have the previous command's batch (or we would have to reevaluate the previous command which feels like a non-starter). I suppose I can imagine a hierarchy of batches. This would require a bit of worker in storage/engine to accomplish. Perhaps I'm misunderstanding what you are thinking of here.

tbg · 2016-10-19T15:23:57Z

This is tricky because in order to evaluate a command on top of a previous command we would need to have the previous command's batch (or we would have to reevaluate the previous command which feels like a non-starter). I suppose I can imagine a hierarchy of batches. This would require a bit of worker in storage/engine to accomplish. Perhaps I'm misunderstanding what you are thinking of here.

The simpler one would be the former, but of course the reevaluation cost seems prohibitive. The tree of engine.Batch is what I pictured, though that does sound kind of spooky.

Why does every write op have to read RangeLastGCKey?

Sorry, not RangeLastGCKey, but RangeGCThresholdKey (and the requirement could likely be relaxed, but definitely a CPut would have to).

petermattis · 2016-10-19T15:40:30Z

Sorry, not RangeLastGCKey, but RangeGCThresholdKey

RangeTxnSpanGCThresholdKey? Seems like that value only ratchets forward.

The above is fairly half baked and challenging to implement, but so is
tracking every command's implicit writes.

Can we move the implicit reads and writes into ProposalData? What is the scope here? Is it too onerous/fragile to identify all of them?

tbg · 2016-10-19T15:43:16Z

RangeTxnSpanGCThresholdKey? Seems like that value only ratchets forward.

Yes, but you can't issue a CPut at a timestamp that is lower than the threshold at the time you propose that CPut but higher when that CPut applies.

tbg · 2016-10-19T15:45:42Z

Can we move the implicit reads and writes into ProposalData? What is the scope here? Is it too onerous/fragile to identify all of them?

What do you mean by moving them?
Yeah, it's pretty onerous and fragile. The evaluation gets a Batch and gets to do whatever it pleases, plus there's access to the in-memory state that should also count towards accessing keys.

The need to special-case and be clever about a lot of this stuff because the trivial solution is wildly unperformant in the common case is pretty frightening.

petermattis · 2016-10-19T15:55:41Z

What do you mean by moving them?

Any key which is being touched (read or written) on every command would not be added to the WriteBatch but would instead have some flag and/or special handling in ProposalData. Onerous to find all these locations, but we could add a wrapper around engine.Batch which prohibits writing these keys.

I'm probably speculating a bit too much here without having thought through the problem thoroughly. I expect massive holes to be poked in the above.

tbg · 2016-10-19T16:36:59Z

Yeah, that amounts to moving more logic downstream of Raft again, and all of these problems have a solution, but I'd rather they have a solution with a solid abstraction so that it's manageable. Regardless, even the batch-tree needs to know what is going to be touched before evaluation happens, so it doesn't really change the game in that regard. I wish there were a better solution than looking hard at each command before it goes into the command queue, but it can probably be made workable.

tbg · 2016-10-19T16:55:10Z

@nvanbenschoten going in the direction this PR is headed, we would add a bunch of highly contended keys to the command queue which however are most often only read. Also, there are always range-local keys in there - I believe we do some shady things in the command queue that may now backfire by creating unnecessarily large spans that always conflict with each other:

    // Create the covering entry. Note that this may have an "illegal" key range
    // spanning from range-local to range-global, but that's acceptable here as
    // long as we're careful in the future.

Do you have input on how to deal with both of those effects? Would having a range-local component with tighter semantics to the command queue make sense? The rough idea would be that state access would act like a RWMutex, i.e. in normal operation everyone just grabs read locks and nothing really happens. A split would grab the exclusive lock, which of course blocks everyone after, so splits would be more expensive. Or we could "honestly" track the reads and writes of state of each command, but then we need to revisit the optimizations.

petermattis · 2016-10-19T17:15:24Z

pkg/storage/replica.go

+		// the heuristic we are using in the CommandQueue to make a large
+		// span first is going to be horrible as we are always including
+		// range-local and range-global keys, and so that mutant span includes
+		// the whole world.


Probably need to pass in separate range-local and range-global spans to the CommandQueue.

Agreed. We'll want to be much more fine grained about what's read-only and what's read-write (much of the system spans are almost always accessed read-only).

petermattis · 2016-10-19T17:15:56Z

pkg/storage/replica.go

+				spans = append(spans, roachpb.Span{
+					Key: keys.RangeLastGCKey(r.RangeID),
+				}, roachpb.Span{
+					Key: keys.RangeLastGCKey(r.RangeID),


I assume the duplication of RangeLastGCKey was an accident.

petermattis · 2016-10-19T17:25:53Z

Yeah, that amounts to moving more logic downstream of Raft again, and all of these problems have a solution, but I'd rather they have a solution with a solid abstraction so that it's manageable.

"For every complex problem there is an answer that is clear, simple, and wrong." H. L. Mencken

Good point about moving logic downstream of Raft again. I'm still thinking about this.

bdarnell

I don't see the stats key or the GC threshold key for every command. Are those just left out of this PR?

bdarnell · 2016-10-20T05:30:34Z

pkg/storage/replica.go

+			spans = append(spans, roachpb.Span{
+				Key: keys.AbortCacheKey(r.RangeID, ba.Txn.ID),
+			}, roachpb.Span{
+				Key: keys.TransactionKey(ba.Txn.Key, ba.Txn.ID),


I understand now why we need to read the abort cache key, but we shouldn't be reading the transaction record on every command - it might not be present on this range.

You're right, will address this.

tbg · 2016-10-20T11:45:52Z

re @bdarnell in #6290:

We have two different ordering mechanisms: the command queue and the lease index. Since the lease index gives a total order to all commands, maybe we don't need a tree of batches but a linear chain: every command is proposed on top of the batch produced by the previous one (and whenever we reset our lease index we rewind to whatever is on disk and start over). Would this free us from having to figure out which commands could interfere with each other for the purposes of proposer-evaluated KV? (this wouldn't let us get rid of the command queue, of course, but it would mean we wouldn't have to expand its role or add additional complications)

The list vs. tree thing is a tradeoff, but I think the list is too simplistic (?).

You would presumably queue all the writes back-to-back, i.e. there's a single list of batches. The downside here is that one write has to wait for a completely unrelated write to create its Batch, i.e. while a large DeleteRange is evaluated, a lot of writes to a different part of the Range has to just sit there.

Reads still have to know fairly fine-grained information from the command queue. For example, reads need to know if something is in-flight that mutates, say, the GC threshold. Naively, that means that an inflight GCRequest blocks all reads until it applies, which is, of course, unnecessary in all but contrived practical situations and a high tax for those. Ideally an inflight GCRequest blocks only the reads at a timestamp that violate the GCThreshold that this request would set, but this is a very fine-grained condition. Easy to do, but it would be done through a custom data structure. I think similar considerations for the other fields will still let us gravitate towards maintaining fine-grained control over which key ranges are accessed by which command.

Back to proposals: The list version above is fairly bad when an expensive operation blocks non-overlapping operations. For example, picture a huge DeleteRange which comes in before a lot of individual non-overlapping single key writes. In that situation, we would put DeleteRange in the command queue, and then start evaluating it. All the following writes would have to sit out that evaluation, which isn't necessary. They could look at the command queue and realize that they don't overlap, and could evaluate themselves in parallel (since they also all mutually don't overlap), and propose themselves in the order in which they get there. Perhaps at some point the DeleteRange would finish and propose itself; all legit, because the command queue guarantees that all of these proposals commute. That corresponds to the direction in this PR (assuming the system keys are suitably special-cased) and requires all the fine-grained tracking of key ranges.

Adding the batch hierarchy to that results in the tree of batches. The case in which this shines is when the DeleteRange has evaluated and now more writes are piling up on top of that. We save the consensus latency that way, which is of course huge. But we absolutely need to keep tracking the "real" set of keys touched by commands for it to be correct.

But now that I've considered all of these options above, it seems that

the list of batches obviates perhaps a little of the tracking complexity (but not all) but is probably bad for the high percentile latencies (or are we afraid of too much parallelism at proposal time to hurt us? We could of course limit the number of concurrent evaluations)
the command tracking approach here might be the most beneficial to set up now: We won't be able to avoid it completely, and it precisely avoids the pitfalls of 1.
the tree of batches is pretty advanced and there seems no good reason to pursue this now.

My vote at this point is to make this PR work, that is to suitably special-case those keys which are touched by almost every command (FrozenStatus, {TxnSpan,}GCThreshold). Ideally at some point we would not give a raw Batch to the commands but some tighter interface (think of a Batch that has a more high-level interface on what keys are being touched, and which can prevent access to keys it wasn't set up for), but for the inital version it would be based on trust and assertions as presented in this PR.

I'll go and annotate all of the necessary spans found in the PR with possible strategies and their remaining performance impact. Then I'll spend some time looking at the mostly orthogonal raftMu question again to give folks some time to take a look at this post and the update to the PR (will ping when that has happened).

@petermattis Creating a hierarchy of Batch wouldn't be more difficult than

b1 := e.NewBatch()
doStuff(b1)
b2 := e.NewBatch()
b2.ApplyBatchRepr(b1.Repr())
doStuff(b2)

With some sugar added to populate b2 directly from b1 mod the round-trip through Go? Maintaining the right "base" to create the batches from sounds like an interesting problem, though.

petermattis · 2016-10-20T12:34:07Z

b2 := e.NewBatch()
b2.ApplyBatchRepr(b1.Repr())
doStuff(b2)

At this point, b2.Repr() contains all of the b1 operations as well as the b2 operations. Don't you want it to only contain the b2 operations? Since batches are append-only we could probably do some hackery to extract out the b2 mutations.

Also note that b2.ApplyBatchRepr(b1.Repr()) is moderately expensive as it has to re-index all of the keys in b1. Certainly much less expensive than re-performing the operation for b1, but definitely not free.

I was imaging we'd do something like:

b1 := e.NewBatch()
doStuff(b1)
b2 := b1.NewBatch()
doStuff(b2)

This wouldn't work at the moment because you can't create a batch on top of another batch. In particular, iterators in such a list of batches would need to be fixed (see db.cc:DBBatch::NewIter). This approach would also make iterators progressively more expensive as the depth of the list increased.

tbg · 2016-10-20T12:43:55Z

I was imaging we'd do something like:

Yes, that's what it would look like in code (though really the Batch should be unusable once it's branched off of, so perhaps that API isn't great), but I was trying to understand what would go on under the hood. Doesn't seem worth poking this too much now as I don't expect us to actually try to do batch compounding.

tbg · 2016-10-20T12:57:12Z

Re @bdarnell:

I don't see the stats key or the GC threshold key for every command. Are those just left out of this PR?

Distinct() writes weren't tracked. They are now. We'll need a better API to make it harder for Raft commands to access random parts of keyspace.

petermattis · 2016-10-20T20:14:43Z

Review status: 0 of 10 files reviewed at latest revision, 6 unresolved discussions, some commit checks failed.

pkg/storage/replica.go, line 1214 at r3 (raw file):

              })
          case *roachpb.HeartbeatTxnRequest:
              if ba.Txn == nil {

When does this occur?

pkg/storage/replica.go, line 1235 at r3 (raw file):

              }, roachpb.Span{
                  // This one isn't asserted but serves as a reminder that
                  // something odd might be going on when we the commit

Some grammar is off here: when we the commit.

pkg/storage/replica.go, line 1303 at r3 (raw file):

              // This blocks all other commands, but that's fine.
              spans = append(spans, roachpb.Span{ // rw
                  Key: keys.RangeFrozenStatusKey(r.RangeID),

You already added this above. Before making this change, I think you should do another PR which changes CommandQueue.{getWait,add} to take separate slices of read and write spans. And there is another question as to whether the API should separate out range-local vs global keys or if CommandQueue should make that determination internally. I'd lean towards the latter.

pkg/storage/replica.go, line 1373 at r3 (raw file):

  return func(br *roachpb.BatchResponse, pErr *roachpb.Error, debug interface{}) *roachpb.Error {
      // TODO(tschottdorf): check that our spans added above cover everything
      // we touched. In particilar, each individual write must be covered

s/particilar/particular/g

Comments from Reviewable

nvb · 2016-10-21T19:09:05Z

Just talked to @tschottdorf offline about this and one of the major takeaways was that splitting up the command queue for range-local and range-global keys seems to be the way to go. This will allow the standard range-global command queue to behave as it does now without having to reconsider previous optimizations.

Neither of us are confident that an interval tree is the best structure for the new range-local cache. The workload will be almost entirely read-only, which means that the overhead of searching an interval tree for all overlaps, only to throw most of them away, would be incredibly inefficient. On top of this, most (all?) requests will be working on a single key, not a key range. An initial idea is to have a map from keys to lists of pending kv operations for that key, with reads being bucketed. Like @tschottdorf mentioned above, each map element would act a lot like a RWMutex, with the added benefit of deterministic ordering.

bdarnell · 2016-10-23T15:22:34Z

You would presumably queue all the writes back-to-back, i.e. there's a single list of batches. The downside here is that one write has to wait for a completely unrelated write to create its Batch, i.e. while a large DeleteRange is evaluated, a lot of writes to a different part of the Range has to just sit there.

That's the same as today, right (with all that work happening in raft command application)? So it's not a regression from today, just a case where we're not getting as much parallelism as we could.

Review status: 0 of 10 files reviewed at latest revision, 7 unresolved discussions, some commit checks failed.

Comments from Reviewable

tbg · 2016-10-23T15:27:30Z

Yeah, you're right, that amounts to the same but we're presumably doing at least a little more work by piling multiple batches upon each other.

Review status: 0 of 10 files reviewed at latest revision, 7 unresolved discussions, some commit checks failed.

Comments from Reviewable

The main objective of this change is to (later) enable proposer-evaluated KV to compute the `WriteBatch` when evaluating a proposal (i.e. when turning a `BatchRequest` into a `*ProposalData`). The previous code assumes making a proposal is cheap, and thus it is done under a large critical sections. MaxLeaseIndex is assigned at the latest possible moment before submitting to Raft. Multiple inbound requests can pass the command queue without interacting, and it's important to assign the Lease index only right before when the proposal is submitted to Raft. Note that the command queue currently doesn't cover all the data accessed by Raft commands (cockroachdb#10084) and that a discussion of how precisely command evaluation will work is still being discussed (cockroachdb#6290). Nevertheless, this change should be safe and a step in the right direction.

The code is a mess and not meant to go in; don't bother leaving style comments. This is an exploration for cockroachdb#6290. Briefly put, evaluation of commands before proposing their result to Raft exercises the command queue in ways the current code did not (which executes everything serially because it's on the Raft goroutine). In particular, the command queue now has to live up to its role of allowing only commands which can be parallelized to be evaluated (and subsequently proposed and applied). What the command queue tracked so far are simply the main key ranges of the requests in the batch, but that is not nearly enough. To err on the side of being conservative, this PR presents an (unmergeably hacky) attempt to classify what keys really need to be protected, as measured by our testing coverage. As such, this list is not complete and in particular it does not take into account reads of Replica state where the in-memory cache is hit (though that could be caught by not accessing state directly but having it go to disk every time; not done yet). Major takeaways are: - Splits need to lock the key range of the RHS descriptor for writes (we can get a little tricky with reads), but also touches all of the abort cache for reading. It also accesses the Replica state for the RHS, though that's a lesser concern. - Every command needs to read FrozenStatus, RangeLastGCKey and writes MVCCStats. Unless we are smart about it, that already completely kills off any proposal parallelism. We will have to be smart about it because that's pretty bad. Especially the stats update is interesting because that is both frequent and a write operation. The obvious solution here is to make use of the commutativity of stats delta updates (though `ContainsEstimates` needs to be taken into account conservatively). - Every transaction reads its AbortCacheKey and reads or writes its transaction record (see discussion in cockroachdb#6290 on why the latter makes sense). The proposal's MaxLeaseIndex already makes sure that when we submit command A to Raft before command B, then they can only apply in that order (mod refurbishments, which can reorder and cause problems to this arguments, though I think they can be overcome). So we could imagine a world in which the command queue wouldn't commonly block commands on top of each other: - a command arrives. - checks the command queue; finds that it must wait for some prior commands to apply. - instead of waiting, it evaluates these commands in the correct order and applies itself on top of that, entering itself in the command queue but immediately proposing to Raft a command whose successful application is contingent upon the previous commands applying as planned. - once the command applies, leave the command queue (as usual) - if the command does not apply (i.e. it or its dependencies got reordered), leave the command queue and enter it anew (instead of the previous internal refurbishment cycle). The above is fairly half baked and challenging to implement, but so is tracking every command's implicit writes; the above idea would in principle remove any latency that isn't Raft or doing actual work, which is a great benefit.

tbg · 2016-10-24T12:52:38Z

@nvanbenschoten I'll send a PR splitting the command queue into two and base my work for cleaning this PR up on top of that, at which point I'd imagine addressing the most egregious performance penalties with your help.

Fixes #6290. Add experimental proposer-evaluated KV gated behind the environment variable `COCKROACH_PROPOSER_EVALUATED_KV`. When set to a truthy value, Raft proposals are evaluated and the resulting RocksDB `WriteBatch` submitted to Raft along with some auxiliary metadata. The result of the evaluation is only stored in the pending command on the proposer, and returned to the waiting client after the `WriteBatch` has been applied. Introduce a natural failfast path for (most) proposals returning an error. Instead of proposing, waiting for Raft, and only then receiving an error, proposals which do not lead to a state change receive their error directly when the proposal is evaluated, upstream of Raft. Only errors which still want to persist data (for instance, `*TransactionRetryError` when intents were laid down) go through the whole proposal, with the client receiving the error after the associated `Batch` commits. While proposer-evaluated KV is now ready for review, preliminary testing and benchmarking, the current implementation is incomplete and incorrect: - `Lease` acquisition is not special-cased, meaning that lease state may be clobbered freely when non-leaseholders propose a lease request based on stale data. This needs to be fixed but it also shows that we don't stress that scenario sufficiently in testing yet. - Similarly, we don't check that commands can only apply under the lease that they were proposed (which is necessary). - `CommandQueue` does not account for internal keys accessed by overlapping commands correctly (this is tracked in #10084), which in principle also lead to anomalies which should be exposed by testing and addressed. Instead, **every** command inserts a span that covers everything. Horrible for performance, great for correctness; #10084 needs to address this. - `TestingCommandFilter` needs to be refactored to be an explicit interceptor for the pre-Raft stage of commands. Tests were fixed up enough to pass with proposer-evaluated KV as well, but it's possible that some tests don't test what they used to. Benchmark results for non-proposer-evaluated-KV against `master` below. **Note that all of them have the "correctness hack" in `(*Replica).beginCmds` disabled; not doing that should give much worse results but we don't care about those in practice** We pay a small fee in allocations, but nothing appears outrageous. So from a performance perspective, this PR seems safe to merge: ``` $ for br in exp-prop-kv masterdo git checkout $br && make bench COCKROACH_PROPOSER_EVALUATED_KV=false PKG=./sql TESTS='(Select|Insert|Update|Delete)1[0-9]*_Cockroach' TESTFLAGS='-benchmem -count 10' BENCHTIMEOUT=1h > $(git rev-parse --abbrev-ref HEAD); done $ benchstat master exp-prop-kv name old time/op new time/op delta Select1_Cockroach-24 79.8µs ± 2% 79.7µs ± 3% ~ (p=0.912 n=10+10) Insert1_Cockroach-24 500µs ± 3% 520µs ± 5% +4.12% (p=0.001 n=10+10) Insert10_Cockroach-24 665µs ± 3% 667µs ± 3% ~ (p=0.631 n=10+10) Insert100_Cockroach-24 1.77ms ± 5% 1.78ms ± 4% ~ (p=0.529 n=10+10) Insert1000_Cockroach-24 11.9ms ± 3% 11.9ms ± 2% ~ (p=1.000 n=10+10) Update1_Cockroach-24 727µs ± 2% 732µs ± 6% ~ (p=0.971 n=10+10) Update10_Cockroach-24 1.14ms ± 4% 1.15ms ± 3% ~ (p=0.211 n=9+10) Update100_Cockroach-24 4.53ms ±12% 4.46ms ± 6% ~ (p=0.796 n=10+10) Update1000_Cockroach-24 33.8ms ± 4% 32.9ms ± 3% -2.60% (p=0.019 n=10+10) Delete1_Cockroach-24 674µs ± 2% 671µs ± 2% ~ (p=0.796 n=10+10) Delete10_Cockroach-24 812µs ± 2% 828µs ± 2% +2.01% (p=0.003 n=9+10) Delete100_Cockroach-24 2.35ms ± 1% 2.39ms ± 5% ~ (p=0.094 n=9+9) Delete1000_Cockroach-24 17.0ms ± 4% 17.0ms ± 2% ~ (p=0.853 n=10+10) InterleavedSelect1000_Cockroach-24 52.7ms ± 4% 52.8ms ± 5% ~ (p=0.549 n=10+9) name old alloc/op new alloc/op delta Select1_Cockroach-24 7.31kB ± 0% 7.31kB ± 0% +0.03% (p=0.000 n=10+9) Insert1_Cockroach-24 36.6kB ± 0% 40.0kB ± 0% +9.25% (p=0.000 n=10+10) Insert10_Cockroach-24 85.2kB ± 0% 90.5kB ± 0% +6.14% (p=0.000 n=10+10) Insert100_Cockroach-24 538kB ± 0% 543kB ± 0% +0.96% (p=0.000 n=10+9) Insert1000_Cockroach-24 4.63MB ± 0% 4.64MB ± 0% +0.09% (p=0.002 n=10+10) Update1_Cockroach-24 62.2kB ± 0% 65.5kB ± 0% +5.37% (p=0.000 n=8+10) Update10_Cockroach-24 136kB ± 0% 142kB ± 0% +3.95% (p=0.000 n=10+9) Update100_Cockroach-24 863kB ± 0% 865kB ± 0% +0.29% (p=0.000 n=9+9) Update1000_Cockroach-24 7.15MB ± 0% 7.15MB ± 0% ~ (p=0.065 n=9+10) Delete1_Cockroach-24 54.0kB ± 0% 57.3kB ± 0% +6.15% (p=0.000 n=10+10) Delete10_Cockroach-24 64.4kB ± 0% 67.9kB ± 0% +5.49% (p=0.000 n=10+10) Delete100_Cockroach-24 173kB ± 0% 178kB ± 0% +2.87% (p=0.000 n=10+9) Delete1000_Cockroach-24 1.19MB ± 0% 1.14MB ± 0% -3.72% (p=0.000 n=9+10) InterleavedSelect1000_Cockroach-24 797kB ± 0% 797kB ± 0% ~ (p=0.811 n=10+10) name old allocs/op new allocs/op delta Select1_Cockroach-24 111 ± 0% 111 ± 0% ~ (all equal) Insert1_Cockroach-24 386 ± 0% 395 ± 0% +2.33% (p=0.000 n=9+8) Insert10_Cockroach-24 612 ± 0% 622 ± 0% +1.63% (p=0.000 n=8+8) Insert100_Cockroach-24 2.63k ± 0% 2.64k ± 0% +0.36% (p=0.000 n=10+7) Insert1000_Cockroach-24 22.6k ± 0% 22.6k ± 0% +0.05% (p=0.006 n=10+10) Update1_Cockroach-24 682 ± 0% 691 ± 0% +1.32% (p=0.000 n=10+10) Update10_Cockroach-24 1.01k ± 0% 1.02k ± 0% +0.96% (p=0.000 n=10+9) Update100_Cockroach-24 3.99k ± 0% 4.00k ± 0% +0.22% (p=0.000 n=9+9) Update1000_Cockroach-24 31.6k ± 0% 31.6k ± 0% +0.03% (p=0.000 n=9+9) Delete1_Cockroach-24 568 ± 0% 577 ± 0% +1.58% (p=0.000 n=10+8) Delete10_Cockroach-24 693 ± 0% 703 ± 0% +1.46% (p=0.000 n=10+10) Delete100_Cockroach-24 1.83k ± 0% 1.84k ± 0% +0.55% (p=0.000 n=9+10) Delete1000_Cockroach-24 12.7k ± 0% 12.7k ± 0% ~ (p=0.954 n=10+10) InterleavedSelect1000_Cockroach-24 3.11k ± 0% 3.11k ± 0% ~ (p=0.586 n=10+7) ``` `master` vs proposer-evaluated KV. Takes a hit, as expected (prop-eval KV does more work as it first preps a batch, serializes and unserializes it, then commits (this is optimizable): ``` name old time/op new time/op delta Select1_Cockroach-24 78.1µs ± 2% 80.7µs ± 3% +3.26% (p=0.000 n=10+10) Insert1_Cockroach-24 507µs ± 5% 525µs ± 5% +3.41% (p=0.019 n=10+10) Insert10_Cockroach-24 662µs ± 2% 674µs ± 4% ~ (p=0.075 n=10+10) Insert100_Cockroach-24 1.74ms ± 5% 1.80ms ± 4% +3.69% (p=0.013 n=9+10) Insert1000_Cockroach-24 11.7ms ± 3% 11.6ms ± 3% ~ (p=0.436 n=10+10) Update1_Cockroach-24 727µs ± 4% 693µs ± 1% -4.67% (p=0.000 n=10+8) Update10_Cockroach-24 1.15ms ± 5% 1.14ms ± 6% ~ (p=0.579 n=10+10) Update100_Cockroach-24 4.42ms ± 6% 4.52ms ± 7% ~ (p=0.190 n=10+10) Update1000_Cockroach-24 32.9ms ± 3% 34.3ms ± 5% +4.04% (p=0.000 n=10+10) Delete1_Cockroach-24 675µs ± 3% 672µs ± 2% ~ (p=0.579 n=10+10) Delete10_Cockroach-24 799µs ± 2% 827µs ± 3% +3.46% (p=0.000 n=8+10) Delete100_Cockroach-24 2.38ms ± 3% 2.52ms ± 4% +6.16% (p=0.000 n=10+10) Delete1000_Cockroach-24 16.7ms ± 4% 17.7ms ± 3% +5.82% (p=0.000 n=10+10) InterleavedSelect1000_Cockroach-24 52.0ms ± 1% 51.9ms ± 2% ~ (p=0.274 n=10+8) name old alloc/op new alloc/op delta Select1_Cockroach-24 7.31kB ± 0% 7.31kB ± 0% +0.03% (p=0.000 n=8+9) Insert1_Cockroach-24 36.6kB ± 0% 39.5kB ± 0% +7.89% (p=0.000 n=9+10) Insert10_Cockroach-24 85.2kB ± 0% 91.2kB ± 0% +7.04% (p=0.000 n=10+10) Insert100_Cockroach-24 538kB ± 0% 559kB ± 0% +3.91% (p=0.000 n=10+10) Insert1000_Cockroach-24 4.63MB ± 0% 4.80MB ± 0% +3.54% (p=0.000 n=10+9) Update1_Cockroach-24 62.2kB ± 0% 65.4kB ± 0% +5.12% (p=0.000 n=10+10) Update10_Cockroach-24 136kB ± 0% 143kB ± 0% +5.09% (p=0.000 n=9+8) Update100_Cockroach-24 863kB ± 0% 883kB ± 0% +2.31% (p=0.000 n=8+10) Update1000_Cockroach-24 7.15MB ± 0% 7.33MB ± 0% +2.53% (p=0.000 n=10+10) Delete1_Cockroach-24 54.0kB ± 0% 56.8kB ± 0% +5.15% (p=0.000 n=10+10) Delete10_Cockroach-24 64.3kB ± 0% 68.7kB ± 0% +6.74% (p=0.000 n=10+10) Delete100_Cockroach-24 173kB ± 0% 194kB ± 0% +11.72% (p=0.000 n=10+10) Delete1000_Cockroach-24 1.19MB ± 0% 1.26MB ± 0% +5.98% (p=0.000 n=9+10) InterleavedSelect1000_Cockroach-24 797kB ± 0% 796kB ± 0% ~ (p=0.095 n=10+9) name old allocs/op new allocs/op delta Select1_Cockroach-24 111 ± 0% 111 ± 0% ~ (all equal) Insert1_Cockroach-24 386 ± 0% 389 ± 0% +0.78% (p=0.000 n=8+9) Insert10_Cockroach-24 612 ± 0% 616 ± 0% +0.65% (p=0.000 n=8+8) Insert100_Cockroach-24 2.63k ± 0% 2.63k ± 0% +0.13% (p=0.000 n=9+6) Insert1000_Cockroach-24 22.6k ± 0% 22.6k ± 0% -0.17% (p=0.000 n=10+10) Update1_Cockroach-24 682 ± 0% 685 ± 0% +0.44% (p=0.000 n=10+10) Update10_Cockroach-24 1.01k ± 0% 1.02k ± 0% +0.39% (p=0.000 n=9+8) Update100_Cockroach-24 3.99k ± 0% 3.99k ± 0% +0.12% (p=0.000 n=9+10) Update1000_Cockroach-24 31.6k ± 0% 31.6k ± 0% +0.04% (p=0.000 n=8+7) Delete1_Cockroach-24 568 ± 0% 571 ± 0% +0.65% (p=0.000 n=10+10) Delete10_Cockroach-24 692 ± 0% 697 ± 0% +0.64% (p=0.000 n=10+10) Delete100_Cockroach-24 1.83k ± 0% 1.83k ± 0% +0.31% (p=0.000 n=10+9) Delete1000_Cockroach-24 12.7k ± 0% 12.7k ± 0% ~ (p=0.195 n=10+10) InterleavedSelect1000_Cockroach-24 3.11k ± 0% 3.11k ± 0% ~ (p=0.190 n=10+9) ``` Finally (just for completeness) this PR without proposer-eval'ed KV against the PR with proposer-eval'ed KV. ``` name old time/op new time/op delta Select1_Cockroach-24 80.7µs ± 3% 80.9µs ± 3% ~ (p=1.000 n=10+9) Insert1_Cockroach-24 525µs ± 5% 522µs ± 3% ~ (p=0.436 n=10+10) Insert10_Cockroach-24 674µs ± 4% 672µs ± 5% ~ (p=0.631 n=10+10) Insert100_Cockroach-24 1.80ms ± 4% 1.82ms ± 6% ~ (p=0.796 n=10+10) Insert1000_Cockroach-24 11.6ms ± 3% 11.9ms ± 1% +1.97% (p=0.008 n=10+9) Update1_Cockroach-24 693µs ± 1% 743µs ± 2% +7.29% (p=0.000 n=8+10) Update10_Cockroach-24 1.14ms ± 6% 1.18ms ± 5% ~ (p=0.075 n=10+10) Update100_Cockroach-24 4.52ms ± 7% 4.61ms ± 8% ~ (p=0.393 n=10+10) Update1000_Cockroach-24 34.3ms ± 5% 33.3ms ± 6% ~ (p=0.105 n=10+10) Delete1_Cockroach-24 672µs ± 2% 671µs ± 4% ~ (p=0.631 n=10+10) Delete10_Cockroach-24 827µs ± 3% 814µs ± 1% -1.59% (p=0.016 n=10+8) Delete100_Cockroach-24 2.52ms ± 4% 2.40ms ± 2% -4.94% (p=0.000 n=10+9) Delete1000_Cockroach-24 17.7ms ± 3% 17.2ms ± 3% -3.14% (p=0.009 n=10+10) InterleavedSelect1000_Cockroach-24 51.9ms ± 2% 51.8ms ± 3% ~ (p=0.633 n=8+10) name old alloc/op new alloc/op delta Select1_Cockroach-24 7.31kB ± 0% 7.31kB ± 0% ~ (p=0.246 n=9+10) Insert1_Cockroach-24 39.5kB ± 0% 40.0kB ± 0% +1.28% (p=0.000 n=10+10) Insert10_Cockroach-24 91.2kB ± 0% 90.4kB ± 0% -0.85% (p=0.000 n=10+10) Insert100_Cockroach-24 559kB ± 0% 543kB ± 0% -2.84% (p=0.000 n=10+9) Insert1000_Cockroach-24 4.80MB ± 0% 4.64MB ± 0% -3.36% (p=0.000 n=9+10) Update1_Cockroach-24 65.4kB ± 0% 65.5kB ± 0% +0.24% (p=0.000 n=10+10) Update10_Cockroach-24 143kB ± 0% 142kB ± 0% -0.93% (p=0.000 n=8+8) Update100_Cockroach-24 883kB ± 0% 865kB ± 0% -1.97% (p=0.000 n=10+8) Update1000_Cockroach-24 7.33MB ± 0% 7.15MB ± 0% -2.45% (p=0.000 n=10+10) Delete1_Cockroach-24 56.8kB ± 0% 57.4kB ± 0% +1.11% (p=0.000 n=10+10) Delete10_Cockroach-24 68.7kB ± 0% 67.9kB ± 0% -1.08% (p=0.000 n=10+10) Delete100_Cockroach-24 194kB ± 0% 178kB ± 0% -7.87% (p=0.000 n=10+10) Delete1000_Cockroach-24 1.26MB ± 0% 1.14MB ± 0% -9.15% (p=0.000 n=10+10) InterleavedSelect1000_Cockroach-24 796kB ± 0% 797kB ± 0% +0.05% (p=0.002 n=9+8) name old allocs/op new allocs/op delta Select1_Cockroach-24 111 ± 0% 111 ± 0% ~ (all equal) Insert1_Cockroach-24 389 ± 0% 395 ± 0% +1.54% (p=0.000 n=9+8) Insert10_Cockroach-24 616 ± 0% 622 ± 0% +0.97% (p=0.000 n=8+8) Insert100_Cockroach-24 2.63k ± 0% 2.64k ± 0% +0.24% (p=0.000 n=6+9) Insert1000_Cockroach-24 22.6k ± 0% 22.6k ± 0% +0.22% (p=0.000 n=10+10) Update1_Cockroach-24 685 ± 0% 691 ± 0% +0.88% (p=0.000 n=10+10) Update10_Cockroach-24 1.02k ± 0% 1.02k ± 0% +0.69% (p=0.000 n=8+7) Update100_Cockroach-24 3.99k ± 0% 4.00k ± 0% +0.13% (p=0.000 n=10+8) Update1000_Cockroach-24 31.6k ± 0% 31.6k ± 0% -0.02% (p=0.002 n=7+8) Delete1_Cockroach-24 571 ± 0% 578 ± 0% +1.10% (p=0.000 n=10+10) Delete10_Cockroach-24 697 ± 0% 704 ± 0% +0.98% (p=0.000 n=10+10) Delete100_Cockroach-24 1.83k ± 0% 1.84k ± 0% +0.28% (p=0.000 n=9+10) Delete1000_Cockroach-24 12.7k ± 0% 12.7k ± 0% ~ (p=1.000 n=10+10) InterleavedSelect1000_Cockroach-24 3.11k ± 0% 3.11k ± 0% ~ (p=0.079 n=9+10) ``` Block writer `master`, `PR, false` `PR, true`: ``` master BenchmarkBlockWriter 1435603 208981.4 ns/op PR wo BenchmarkBlockWriter 1508092 198933.8 ns/op with BenchmarkBlockWriter 1374589 218254.9 ns/op 1s: 4587.2/sec 4587.1/sec | 4349.7/sec 4349.4/sec | 4001.9/sec 4001.8/sec 2s: 4224.0/sec 4406.8/sec | 4330.0/sec 4339.8/sec | 3967.9/sec 3984.9/sec 3s: 4137.1/sec 4317.3/sec | 4185.0/sec 4288.4/sec | 3588.0/sec 3852.9/sec 4s: 4015.0/sec 4242.0/sec | 4204.7/sec 4267.5/sec | 3695.2/sec 3813.5/sec 5s: 4151.0/sec 4223.8/sec | 3983.2/sec 4210.7/sec | 3879.0/sec 3826.6/sec 6s: 4155.8/sec 4212.5/sec | 3851.0/sec 4150.9/sec | 3914.0/sec 3841.2/sec 7s: 4020.1/sec 4185.1/sec | 3895.2/sec 4114.4/sec | 3747.0/sec 3827.7/sec 8s: 3831.0/sec 4140.9/sec | 3807.9/sec 4076.1/sec | 3632.0/sec 3803.3/sec 9s: 3875.8/sec 4111.5/sec | 3914.8/sec 4058.2/sec | 3553.8/sec 3775.6/sec 10s: 3884.2/sec 4088.8/sec | 3987.2/sec 4051.1/sec | 3604.2/sec 3758.4/sec 11s: 4089.9/sec 4088.9/sec | 3911.0/sec 4038.4/sec | 3656.9/sec 3749.2/sec 12s: 3985.2/sec 4080.3/sec | 3901.9/sec 4027.0/sec | 3707.0/sec 3745.7/sec 13s: 4018.8/sec 4075.5/sec | 3804.0/sec 4009.9/sec | 3584.0/sec 3733.3/sec 14s: 3984.1/sec 4069.0/sec | 4010.9/sec 4010.0/sec | 3439.9/sec 3712.3/sec 15s: 3699.1/sec 4044.4/sec | 3665.1/sec 3987.0/sec | 3478.2/sec 3696.7/sec 16s: 3912.7/sec 4036.2/sec | 3764.0/sec 3973.1/sec | 3616.9/sec 3691.7/sec 17s: 3770.3/sec 4020.5/sec | 3747.0/sec 3959.8/sec | 3639.0/sec 3688.6/sec 18s: 3997.9/sec 4019.3/sec | 3903.9/sec 3956.7/sec | 3588.1/sec 3683.1/sec 19s: 3630.0/sec 3998.8/sec | 4115.8/sec 3965.0/sec | 3508.8/sec 3673.9/sec 20s: 4005.9/sec 3999.2/sec | 3870.2/sec 3960.3/sec | 3759.1/sec 3678.1/sec 21s: 3719.1/sec 3985.8/sec | 3618.1/sec 3944.0/sec | 3682.1/sec 3678.3/sec 22s: 3722.0/sec 3973.8/sec | 3970.1/sec 3945.2/sec | 3674.9/sec 3678.2/sec 23s: 3902.0/sec 3970.7/sec | 3441.9/sec 3923.3/sec | 3468.0/sec 3669.0/sec 24s: 3638.0/sec 3956.9/sec | 3684.9/sec 3913.4/sec | 3631.0/sec 3667.5/sec 25s: 6620.0/sec 4063.3/sec | 5308.0/sec 3969.2/sec | 3617.7/sec 3665.5/sec 26s: 6123.1/sec 4142.5/sec | 6115.1/sec 4051.7/sec | 3225.5/sec 3648.6/sec 27s: 6660.7/sec 4235.7/sec | 5974.1/sec 4122.8/sec | 5913.4/sec 3732.4/sec 28s: 6209.2/sec 4306.2/sec | 6431.0/sec 4205.2/sec | 5624.0/sec 3800.0/sec 29s: 6580.9/sec 4384.6/sec | 6189.8/sec 4273.7/sec | 6057.5/sec 3877.8/sec 30s: 6016.1/sec 4438.9/sec | 6489.3/sec 4347.5/sec | 5491.4/sec 3931.6/sec 31s: 6454.1/sec 4503.9/sec | 5094.9/sec 4371.6/sec | 6233.2/sec 4005.8/sec 32s: 5101.9/sec 4522.6/sec | 6427.0/sec 4435.8/sec | 5396.7/sec 4049.3/sec 33s: 6572.9/sec 4584.7/sec | 5962.2/sec 4482.0/sec | 5990.1/sec 4108.1/sec 34s: 4667.2/sec 4587.1/sec | 6078.0/sec 4529.0/sec | 5562.0/sec 4150.8/sec 35s: 6594.7/sec 4644.5/sec | 6301.0/sec 4579.6/sec | 5663.0/sec 4194.0/sec 36s: 6504.2/sec 4696.1/sec | 5936.9/sec 4617.3/sec | 5877.0/sec 4240.8/sec 37s: 6061.0/sec 4733.0/sec | 6466.9/sec 4667.3/sec | 5790.9/sec 4282.6/sec 38s: 6444.1/sec 4778.0/sec | 6123.9/sec 4705.6/sec | 5912.2/sec 4325.5/sec 39s: 6294.0/sec 4816.8/sec | 6248.1/sec 4745.1/sec | 5657.1/sec 4359.7/sec 40s: 6782.5/sec 4866.0/sec | 5872.2/sec 4773.3/sec | 5854.9/sec 4397.0/sec 41s: 9043.7/sec 4967.8/sec | 8606.9/sec 4866.8/sec | 5819.9/sec 4431.7/sec 42s: 8676.5/sec 5056.1/sec | 8308.8/sec 4948.7/sec | 5291.8/sec 4452.2/sec 43s: 8344.1/sec 5132.5/sec | 8517.3/sec 5031.7/sec | 6688.5/sec 4504.2/sec 44s: 8777.1/sec 5215.4/sec | 8169.6/sec 5103.0/sec | 8453.8/sec 4594.0/sec 45s: 8428.1/sec 5286.7/sec | 8810.7/sec 5185.4/sec | 7550.9/sec 4659.7/sec 46s: 8874.7/sec 5364.7/sec | 8286.6/sec 5252.8/sec | 8502.2/sec 4743.2/sec 47s: 8416.3/sec 5429.6/sec | 8596.9/sec 5323.9/sec | 6749.1/sec 4785.9/sec 48s: 8860.8/sec 5501.1/sec | 8173.3/sec 5383.2/sec | 8306.8/sec 4859.2/sec 49s: 8766.2/sec 5567.7/sec | 8589.9/sec 5448.7/sec | 8556.9/sec 4934.7/sec 50s: 8214.9/sec 5620.6/sec | 8577.9/sec 5511.3/sec | 6479.5/sec 4965.5/sec 51s: 8893.0/sec 5684.8/sec | 7346.0/sec 5547.2/sec | 8454.6/sec 5033.9/sec 52s: 8326.0/sec 5735.5/sec | 8664.9/sec 5607.2/sec | 6760.0/sec 5067.1/sec 53s: 8646.5/sec 5790.5/sec | 8677.1/sec 5665.1/sec | 8539.9/sec 5132.7/sec 54s: 8873.8/sec 5847.5/sec | 7655.2/sec 5701.9/sec | 6050.0/sec 5149.6/sec 55s: 8763.5/sec 5900.5/sec | 8749.0/sec 5757.3/sec | 8429.9/sec 5209.3/sec 56s: 6465.9/sec 5910.6/sec | 8723.9/sec 5810.3/sec | 8601.0/sec 5269.8/sec 57s: 3879.7/sec 5875.0/sec | 8659.0/sec 5860.3/sec | 3968.7/sec 5247.0/sec 58s: 2531.1/sec 5817.4/sec | 1449.9/sec 5784.2/sec | 6217.5/sec 5263.7/sec 59s: 4186.3/sec 5789.7/sec | 3661.2/sec 5748.2/sec | 8284.1/sec 5314.9/sec 1m0s: 7975.0/sec 5826.2/sec | 1211.1/sec 5672.6/sec | 2871.8/sec 5274.2/sec 1m1s: 6711.0/sec 5840.7/sec | 1431.9/sec 5603.1/sec | 4990.3/sec 5269.6/sec 1m2s: 4748.7/sec 5823.0/sec | 7576.7/sec 5635.0/sec | 6227.2/sec 5285.0/sec 1m3s: 5278.5/sec 5814.4/sec | 6698.6/sec 5651.8/sec | 8392.9/sec 5334.3/sec 1m4s: 2612.9/sec 5764.4/sec | 8542.8/sec 5697.0/sec | 4390.2/sec 5319.6/sec 1m5s: 4458.8/sec 5744.3/sec | 3849.8/sec 5668.6/sec | 15.0/sec 5238.0/sec 1m6s: 778.0/sec 5669.1/sec | 973.0/sec 5597.5/sec | 6057.5/sec 5250.4/sec 1m7s: 7397.8/sec 5694.9/sec | 6525.1/sec 5611.3/sec | 1583.9/sec 5195.7/sec 1m8s: 4206.3/sec 5673.0/sec | 5286.7/sec 5606.5/sec | 1670.1/sec 5143.8/sec 1m9s: 9329.0/sec 5726.0/sec | 145.0/sec 5527.4/sec | 7968.2/sec 5184.8/sec 1m10s: 4207.0/sec 5704.3/sec | 8178.6/sec 5565.2/sec | 6377.1/sec 5201.8/sec 1m11s: 467.1/sec 5630.5/sec | 5808.1/sec 5568.7/sec | 8555.2/sec 5249.0/sec 1m12s: 3249.5/sec 5597.5/sec | 3945.1/sec 5546.1/sec | 1160.3/sec 5192.3/sec 1m13s: 7547.9/sec 5624.2/sec |10219.6/sec 5610.1/sec | 753.0/sec 5131.5/sec 1m14s:10479.6/sec 5689.8/sec | 8955.5/sec 5655.3/sec | 4614.0/sec 5124.5/sec 1m15s: 2402.0/sec 5645.9/sec | 2536.8/sec 5613.8/sec | 818.0/sec 5067.1/sec 1m16s: 438.0/sec 5577.4/sec | 6229.0/sec 5621.8/sec | 1130.0/sec 5015.3/sec 1m17s: 3561.1/sec 5551.2/sec | 13.0/sec 5549.0/sec | 6255.3/sec 5031.4/sec 1m18s: 3588.8/sec 5526.1/sec | 9045.8/sec 5593.8/sec | 9972.2/sec 5094.7/sec 1m19s:10131.5/sec 5584.4/sec | 2488.7/sec 5554.5/sec | 7508.0/sec 5125.2/sec 1m20s: 1915.0/sec 5538.5/sec | 4666.1/sec 5543.4/sec | 9947.6/sec 5185.5/sec 1m21s: 1530.6/sec 5489.0/sec | 5259.1/sec 5539.9/sec | 8631.2/sec 5228.1/sec 1m22s: 5218.2/sec 5485.7/sec | 8594.8/sec 5577.2/sec | 2574.8/sec 5195.7/sec 1m23s: 1790.0/sec 5441.2/sec | 5812.0/sec 5580.0/sec | 6606.1/sec 5212.7/sec 1m24s: 1077.0/sec 5389.3/sec | 2105.3/sec 5538.6/sec | 2788.3/sec 5183.8/sec 1m25s: 9192.3/sec 5434.0/sec | 9297.8/sec 5582.9/sec | 7046.2/sec 5205.7/sec 1m26s: 2206.0/sec 5396.5/sec | 6761.8/sec 5596.6/sec | 2829.2/sec 5178.1/sec 1m27s: 8510.9/sec 5432.3/sec | 2130.1/sec 5556.7/sec | 7195.1/sec 5201.3/sec 1m28s: 7004.2/sec 5450.1/sec | 8941.0/sec 5595.2/sec | 4929.9/sec 5198.2/sec 1m29s:10563.6/sec 5507.6/sec | 6855.6/sec 5609.3/sec | 9913.1/sec 5251.2/sec 1m30s: 1207.0/sec 5459.8/sec | 2984.9/sec 5580.2/sec | 5823.7/sec 5257.5/sec 1m31s: 1888.0/sec 5420.5/sec | 9592.9/sec 5624.3/sec | 9037.5/sec 5299.1/sec 1m32s: 6090.1/sec 5427.8/sec | 8368.9/sec 5654.1/sec | 1075.9/sec 5253.2/sec 1m33s: 1273.0/sec 5383.1/sec | 9538.1/sec 5695.9/sec | 2864.3/sec 5227.5/sec 1m34s: 1743.9/sec 5344.4/sec | 5547.4/sec 5694.3/sec | 5787.0/sec 5233.4/sec 1m35s: 6407.9/sec 5355.6/sec | 1070.1/sec 5645.6/sec | 5361.6/sec 5234.8/sec 1m36s: 9923.0/sec 5403.2/sec | 9663.3/sec 5687.5/sec | 4910.0/sec 5231.4/sec 1m37s: 4074.8/sec 5389.5/sec | 6670.1/sec 5697.6/sec | 5113.5/sec 5230.2/sec 1m38s: 9495.4/sec 5431.4/sec | 5261.2/sec 5693.1/sec | 8203.4/sec 5260.5/sec 1m39s: 4153.8/sec 5418.5/sec | 22.0/sec 5635.9/sec | 4737.9/sec 5255.2/sec 1m40s: 103.0/sec 5365.3/sec | 6668.1/sec 5646.2/sec | 7005.8/sec 5272.8/sec 1m41s: 286.0/sec 5315.1/sec | 1530.9/sec 5605.4/sec | 5857.7/sec 5278.5/sec 1m42s: 7923.5/sec 5340.6/sec | 9456.7/sec 5643.2/sec | 8831.6/sec 5313.4/sec 1m43s:10307.9/sec 5388.8/sec | 8832.2/sec 5674.1/sec | 884.0/sec 5270.4/sec 1m44s: 4427.9/sec 5379.6/sec | 6880.4/sec 5685.7/sec | 935.0/sec 5228.7/sec 1m45s: 1116.0/sec 5339.0/sec | 264.0/sec 5634.1/sec | 4876.8/sec 5225.3/sec 1m46s: 9290.0/sec 5376.3/sec | 85.0/sec 5581.8/sec | 6945.1/sec 5241.6/sec 1m47s:10097.9/sec 5420.4/sec | 1739.3/sec 5545.9/sec | 4116.4/sec 5231.0/sec 1m48s: 2953.8/sec 5397.6/sec | 7145.1/sec 5560.7/sec | 4073.2/sec 5220.3/sec 1m49s: 2462.0/sec 5370.6/sec | 3741.0/sec 5544.0/sec | 6090.8/sec 5228.3/sec 1m50s: 6493.2/sec 5380.8/sec | 6756.6/sec 5555.0/sec | 1162.9/sec 5191.4/sec 1m51s: 5867.9/sec 5385.2/sec | 9214.1/sec 5588.0/sec | 9444.3/sec 5229.7/sec 1m52s: 80.0/sec 5337.9/sec | 7630.3/sec 5606.2/sec | 8124.0/sec 5255.5/sec 1m53s: 66.0/sec 5291.2/sec | 22.0/sec 5556.8/sec | 1259.9/sec 5220.1/sec 1m54s:5144.2/sec 5289.9/sec | 8677.7/sec 5584.1/sec | 7879.3/sec 5243.5/sec 1m55s: 10304.2/sec 5333.5/sec | 5515.0/sec 5583.5/sec | 15.0/sec 5198.0/sec 1m56s: 5323.0/sec 5333.4/sec | 4051.9/sec 5570.3/sec | 8838.8/sec 5229.4/sec 1m57s: 48.0/sec 5288.3/sec | 6286.7/sec 5576.5/sec | 35.0/sec 5185.0/sec 1m58s: 5770.6/sec 5292.3/sec | 9040.7/sec 5605.8/sec | 126.0/sec 5142.1/sec 1m59s: 3523.4/sec 5277.5/sec | 9035.0/sec 5634.6/sec | 2184.4/sec 5117.3/sec 2m0s: 8119.7/sec 5301.2/sec | 7646.5/sec 5651.4/sec | 7637.9/sec 5138.3/sec 2m1s: 4755.7/sec 5296.7/sec | 3334.8/sec 5632.3/sec | 1518.0/sec 5108.4/sec 2m2s:10283.7/sec 5337.5/sec | 755.0/sec 5592.3/sec | 4609.2/sec 5104.3/sec 2m3s: 5964.9/sec 5342.6/sec | 6245.0/sec 5597.6/sec | 5944.8/sec 5111.1/sec 2m4s: 2230.7/sec 5317.5/sec | 7813.5/sec 5615.5/sec | 5123.2/sec 5111.2/sec 2m5s: 5116.7/sec 5315.9/sec | 4813.5/sec 5609.0/sec | 6269.3/sec 5120.5/sec 2m6s: 3105.0/sec 5298.4/sec | 5171.3/sec 5605.6/sec | 4086.5/sec 5112.3/sec 2m7s: 6890.8/sec 5310.9/sec | 4732.9/sec 5598.7/sec | 3453.0/sec 5099.2/sec 2m8s: 1377.0/sec 5280.2/sec | 2662.3/sec 5575.8/sec | 42.0/sec 5059.7/sec 2m9s: 48.0/sec 5239.6/sec | 6369.8/sec 5581.9/sec | 5075.4/sec 5059.8/sec 2m10s: 5518.2/sec 5241.8/sec | 5810.2/sec 5583.7/sec | 5711.2/sec 5064.8/sec 2m11s: 9196.1/sec 5271.9/sec | 4373.2/sec 5574.4/sec | 1813.0/sec 5040.0/sec 2m12s: 1666.9/sec 5244.6/sec | 7006.9/sec 5585.3/sec | 3148.4/sec 5025.7/sec 2m13s: 6160.3/sec 5251.5/sec | 0.0/sec 5543.3/sec | 8122.3/sec 5048.9/sec 2m14s: 7106.3/sec 5265.4/sec | 0.0/sec 5501.9/sec | 4414.4/sec 5044.2/sec 2m15s: 4772.5/sec 5261.7/sec | 2.0/sec 5461.2/sec | 8962.5/sec 5073.2/sec 2m16s: 4069.8/sec 5253.0/sec | 1009.0/sec 5428.5/sec | 33.0/sec 5036.2/sec 2m17s: 5765.1/sec 5256.7/sec | 3482.2/sec 5414.3/sec | 41.0/sec 4999.7/sec 2m18s: 8875.2/sec 5282.9/sec | 9462.9/sec 5443.6/sec | 2424.8/sec 4981.1/sec 2m19s: 5182.5/sec 5282.2/sec | 9167.0/sec 5470.4/sec | 7051.9/sec 4996.0/sec 2m20s: 1634.1/sec 5256.1/sec |10603.1/sec 5507.0/sec | 5391.7/sec 4998.8/sec 2m21s: 323.0/sec 5221.2/sec | 1704.8/sec 5480.1/sec | 1377.0/sec 4973.1/sec 2m22s: 5411.2/sec 5222.5/sec | 8832.5/sec 5503.7/sec | 2668.0/sec 4956.9/sec 2m23s: 1981.0/sec 5199.8/sec | 712.0/sec 5470.2/sec | 6169.5/sec 4965.3/sec 2m24s: 3647.6/sec 5189.0/sec |11296.1/sec 5510.6/sec | 3489.7/sec 4955.1/sec 2m25s: 166.0/sec 5154.4/sec | 2499.8/sec 5489.9/sec | 5114.0/sec 4956.2/sec 2m26s: 5974.4/sec 5160.0/sec | 6934.7/sec 5499.8/sec | 6929.5/sec 4969.7/sec 2m27s: 1527.1/sec 5135.3/sec | 2523.4/sec 5479.5/sec | 731.9/sec 4940.9/sec 2m28s: 359.0/sec 5103.0/sec | 54.0/sec 5442.8/sec | 0.0/sec 4907.5/sec 2m29s: 8377.1/sec 5125.0/sec | 6178.9/sec 5447.8/sec | 680.0/sec 4879.1/sec 2m30s: 4546.7/sec 5121.2/sec | 6312.5/sec 5453.5/sec | 8254.5/sec 4901.6/sec 2m31s: 3130.9/sec 5108.0/sec | 9287.2/sec 5478.9/sec | 1600.0/sec 4879.8/sec 2m32s: 1238.9/sec 5082.5/sec | 10.0/sec 5443.0/sec | 2549.9/sec 4864.4/sec 2m33s: 8487.0/sec 5104.8/sec | 2116.2/sec 5421.2/sec | 9079.3/sec 4892.0/sec 2m34s: 42.0/sec 5071.9/sec | 5653.3/sec 5422.7/sec | 1373.9/sec 4869.1/sec 2m35s: 73.0/sec 5039.6/sec |10985.9/sec 5458.6/sec | 1715.9/sec 4848.8/sec 2m36s: 8468.5/sec 5061.6/sec | 5165.2/sec 5456.7/sec | 3184.1/sec 4838.1/sec 2m37s: 7946.5/sec 5080.0/sec | 1527.3/sec 5431.7/sec | 8996.7/sec 4864.6/sec 2m38s: 2291.8/sec 5062.3/sec | 5349.0/sec 5431.2/sec | 5937.5/sec 4871.4/sec 2m39s:10515.7/sec 5096.6/sec | 4281.7/sec 5424.0/sec | 7707.2/sec 4889.2/sec 2m40s: 7624.0/sec 5112.4/sec | 3941.0/sec 5414.7/sec | 41.0/sec 4858.9/sec 2m41s: 6627.3/sec 5121.8/sec | 7552.6/sec 5428.0/sec | 5506.6/sec 4862.9/sec 2m42s: 779.0/sec 5095.0/sec | 6501.1/sec 5434.6/sec | 2935.9/sec 4851.1/sec 2m43s: 6369.6/sec 5102.9/sec | 7616.9/sec 5448.0/sec |10298.3/sec 4884.5/sec 2m44s: 1653.0/sec 5081.8/sec | 2503.4/sec 5430.0/sec | 5839.0/sec 4890.3/sec 2m45s: 2058.1/sec 5063.5/sec | 33.0/sec 5397.3/sec |10947.2/sec 4927.0/sec 2m46s: 9075.9/sec 5087.7/sec | 9044.7/sec 5419.3/sec | 4132.5/sec 4922.2/sec 2m47s:10425.3/sec 5119.6/sec | 3219.8/sec 5406.1/sec | 8988.0/sec 4946.6/sec 2m48s: 3264.3/sec 5108.6/sec | 1196.1/sec 5381.1/sec |10646.2/sec 4980.5/sec 2m49s: 2728.0/sec 5094.5/sec | 4809.8/sec 5377.7/sec | 2066.0/sec 4963.2/sec 2m50s:10931.3/sec 5128.8/sec | 6443.3/sec 5383.9/sec | 1873.0/sec 4945.1/sec 2m51s: 6323.5/sec 5135.8/sec | 164.0/sec 5353.4/sec | 4180.2/sec 4940.6/sec 2m52s: 1607.0/sec 5115.3/sec | 6430.5/sec 5359.7/sec | 2776.9/sec 4928.0/sec 2m53s: 982.0/sec 5091.4/sec | 9842.2/sec 5385.6/sec |11246.5/sec 4964.5/sec 2m54s: 7885.9/sec 5107.5/sec | 1940.9/sec 5365.8/sec | 3676.7/sec 4957.1/sec 2m55s: 1905.9/sec 5089.2/sec |10557.4/sec 5395.5/sec | 9840.8/sec 4985.0/sec 2m56s: 1937.1/sec 5071.3/sec | 7563.9/sec 5407.8/sec | 5596.0/sec 4988.5/sec 2m57s: 1679.0/sec 5052.1/sec | 1272.1/sec 5384.4/sec | 4720.2/sec 4987.0/sec 2m58s: 4255.4/sec 5047.6/sec | 9745.7/sec 5408.9/sec | 1282.2/sec 4966.2/sec 2m59s: 8426.2/sec 5066.5/sec | 7192.5/sec 5418.9/sec | 0.0/sec 4938.4/sec 3m0s: 1167.0/sec 5044.8/sec | 1870.0/sec 5399.2/sec | 2833.1/sec 4926.7/sec 3m1s: 3522.1/sec 5036.4/sec | 48.0/sec 5369.6/sec | 56.0/sec 4899.8/sec 3m2s: 3810.0/sec 5029.7/sec | 3947.4/sec 5361.8/sec | 6804.3/sec 4910.3/sec 3m3s: 3109.1/sec 5019.2/sec | 4846.9/sec 5359.0/sec | 5008.5/sec 4910.8/sec 3m4s: 57.0/sec 4992.2/sec | 44.0/sec 5330.1/sec | 4704.1/sec 4909.7/sec 3m5s: 4918.1/sec 4991.8/sec | 8990.2/sec 5349.9/sec | 5397.7/sec 4912.3/sec 3m6s: 9671.3/sec 5017.0/sec | 3887.1/sec 5342.0/sec | 55.0/sec 4886.2/sec 3m7s: 130.0/sec 4990.9/sec | 8050.1/sec 5356.5/sec | 5212.2/sec 4888.0/sec 3m8s: 199.0/sec 4965.4/sec | 8238.1/sec 5371.8/sec | 3365.8/sec 4879.9/sec 3m9s: 836.0/sec 4943.5/sec | 9291.2/sec 5392.6/sec | 7874.5/sec 4895.7/sec 3m10s: 1827.0/sec 4927.1/sec | 19.0/sec 5364.3/sec | 2081.9/sec 4880.9/sec 3m11s: 4937.3/sec 4927.2/sec | 4660.5/sec 5360.6/sec | 41.0/sec 4855.6/sec 3m12s: 336.0/sec 4903.3/sec | 5277.5/sec 5360.2/sec | 6003.1/sec 4861.5/sec 3m13s: 2059.3/sec 4888.5/sec | 1565.1/sec 5340.5/sec | 1630.1/sec 4844.8/sec 3m14s:11245.9/sec 4921.3/sec | 6013.7/sec 5344.0/sec | 1080.9/sec 4825.4/sec 3m15s: 2425.9/sec 4908.5/sec | 2092.2/sec 5327.3/sec | 1754.1/sec 4809.6/sec 3m16s:10817.1/sec 4938.6/sec | 5876.5/sec 5330.1/sec | 7598.4/sec 4823.9/sec 3m17s: 1429.7/sec 4920.8/sec | 6989.3/sec 5338.5/sec | 8318.6/sec 4841.6/sec 3m18s: 4555.1/sec 4919.0/sec |10488.4/sec 5364.5/sec | 4193.9/sec 4838.3/sec 3m19s: 2661.3/sec 4907.6/sec | 3623.0/sec 5355.8/sec | 3804.2/sec 4833.1/sec 3m20s: 17.0/sec 4883.2/sec | 5813.5/sec 5358.1/sec | 4302.8/sec 4830.5/sec 3m21s: 2624.1/sec 4872.0/sec | 8798.3/sec 5375.2/sec | 5396.1/sec 4833.3/sec 3m22s: 2060.1/sec 4858.0/sec | 4410.4/sec 5370.4/sec | 3108.5/sec 4824.8/sec 3m23s: 8427.1/sec 4875.6/sec | 9463.4/sec 5390.6/sec | 4465.0/sec 4823.0/sec 3m24s: 5281.8/sec 4877.6/sec | 3938.9/sec 5383.4/sec | 122.0/sec 4799.9/sec 3m25s: 1801.8/sec 4862.6/sec | 10357.4/sec 5407.7/sec| 8710.2/sec 4819.0/sec 3m26s: 5574.2/sec 4866.1/sec | 9170.2/sec 5426.0/sec | 2832.1/sec 4809.4/sec 3m27s: 6970.0/sec 4876.2/sec | 62.0/sec 5400.1/sec | 5623.2/sec 4813.3/sec 3m28s: 4561.1/sec 4874.7/sec | 4262.5/sec 5394.6/sec | 3071.6/sec 4804.9/sec 3m29s: 5857.8/sec 4879.4/sec | 828.0/sec 5372.7/sec | 0.0/sec 4781.9/sec 3m30s: 1756.0/sec 4864.5/sec | 3004.9/sec 5361.5/sec | 0.0/sec 4759.2/sec 3m31s: 3234.0/sec 4856.8/sec | 3952.0/sec 5354.8/sec | 0.0/sec 4736.6/sec 3m32s: 3143.0/sec 4848.7/sec | 3447.0/sec 5345.8/sec | 0.0/sec 4714.3/sec 3m33s: 2955.2/sec 4839.8/sec | 2388.0/sec 5331.9/sec | 0.0/sec 4692.1/sec 3m34s: 2099.6/sec 4827.0/sec | 5861.5/sec 5334.4/sec | 4770.4/sec 4692.5/sec 3m35s: 4698.0/sec 4826.4/sec | 7248.3/sec 5343.3/sec | 8676.7/sec 4711.0/sec 3m36s: 7993.7/sec 4841.1/sec | 3046.1/sec 5332.6/sec | 88.0/sec 4689.6/sec 3m37s: 7314.4/sec 4852.5/sec | 8254.4/sec 5346.1/sec | 5004.5/sec 4691.1/sec 3m38s: 9046.5/sec 4871.7/sec | 4847.6/sec 5343.8/sec | 2257.1/sec 4679.9/sec 3m39s: 9369.4/sec 4892.3/sec | 8348.0/sec 5357.5/sec | 2752.0/sec 4671.1/sec 3m40s: 4950.4/sec 4892.5/sec | 62.0/sec 5333.5/sec | 6111.4/sec 4677.7/sec 3m41s: 61.0/sec 4870.7/sec | 2671.0/sec 5321.4/sec | 407.1/sec 4658.3/sec 3m42s: 7474.4/sec 4882.4/sec | 2607.3/sec 5309.2/sec | 57.0/sec 4637.6/sec 3m43s: 7187.7/sec 4892.7/sec | 5913.5/sec 5311.9/sec | 4693.3/sec 4637.9/sec 3m44s: 7027.2/sec 4902.2/sec | 4433.3/sec 5308.0/sec | 2793.8/sec 4629.6/sec 3m45s: 3752.3/sec 4897.1/sec |10589.6/sec 5331.5/sec | 8715.7/sec 4647.8/sec 3m46s: 9824.8/sec 4918.9/sec | 4915.1/sec 5329.6/sec | 4384.0/sec 4646.6/sec 3m47s:10184.8/sec 4942.1/sec | 8602.4/sec 5344.0/sec | 106.0/sec 4626.6/sec 3m48s: 2577.9/sec 4931.8/sec | 1374.0/sec 5326.6/sec | 9405.0/sec 4647.6/sec 3m49s: 8359.3/sec 4946.7/sec | 2558.1/sec 5314.5/sec | 3944.3/sec 4644.5/sec 3m50s: 8608.3/sec 4962.7/sec | 966.0/sec 5295.6/sec | 5170.7/sec 4646.8/sec 3m51s: 3402.7/sec 4955.9/sec | 5807.4/sec 5297.8/sec | 3573.7/sec 4642.1/sec 3m52s: 5969.5/sec 4960.3/sec | 6293.8/sec 5302.1/sec | 9580.1/sec 4663.4/sec 3m53s: 4655.6/sec 4959.0/sec | 330.1/sec 5280.8/sec |10790.9/sec 4689.7/sec 3m54s: 4637.9/sec 4957.6/sec | 1899.0/sec 5266.4/sec | 5911.4/sec 4694.9/sec 3m55s: 7502.2/sec 4968.4/sec | 5942.4/sec 5269.2/sec | 3858.4/sec 4691.4/sec 3m56s: 3467.5/sec 4962.1/sec | 8172.9/sec 5281.5/sec | 8306.7/sec 4706.7/sec 3m57s: 1986.1/sec 4949.5/sec | 500.0/sec 5261.4/sec | 2844.8/sec 4698.8/sec 3m58s: 6150.3/sec 4954.5/sec | 6793.5/sec 5267.8/sec | 0.0/sec 4679.1/sec 3m59s: 1767.9/sec 4941.2/sec | 9447.6/sec 5285.3/sec | 0.0/sec 4659.5/sec 4m0s: 0.0/sec 4920.6/sec | 4337.1/sec 5281.3/sec | 5272.7/sec 4662.1/sec 4m1s: 0.0/sec 4900.2/sec | 1792.2/sec 5266.9/sec | 2052.0/sec 4651.3/sec 4m2s: 0.0/sec 4880.0/sec | 4982.0/sec 5265.7/sec | 40.0/sec 4632.2/sec 4m3s: 0.0/sec 4859.9/sec | 8689.0/sec 5279.8/sec | 7462.2/sec 4643.8/sec 4m4s: 5126.8/sec 4861.0/sec | 3963.1/sec 5274.4/sec | 6461.0/sec 4651.3/sec 4m5s: 9416.8/sec 4879.6/sec | 4915.1/sec 5272.9/sec | 3797.6/sec 4647.8/sec 4m6s: 5596.7/sec 4882.5/sec | 5439.8/sec 5273.6/sec | 6717.2/sec 4656.2/sec 4m7s: 7137.0/sec 4891.6/sec | 2062.2/sec 5260.6/sec | 5986.5/sec 4661.6/sec 4m8s: 2900.1/sec 4883.6/sec | 4199.6/sec 5256.3/sec | 5423.3/sec 4664.7/sec 4m9s: 2352.7/sec 4873.4/sec | 5207.5/sec 5256.1/sec | 5621.3/sec 4668.5/sec 4m10s: 7146.7/sec 4882.5/sec | 49.0/sec 5235.3/sec | 59.0/sec 4650.1/sec 4m11s: 7591.3/sec 4893.3/sec | 3108.5/sec 5226.8/sec | 4744.5/sec 4650.5/sec 4m12s: 3887.8/sec 4889.3/sec | 2972.8/sec 5217.9/sec | 5453.8/sec 4653.6/sec 4m13s: 0.0/sec 4870.0/sec | 5755.6/sec 5220.0/sec | 5422.1/sec 4656.7/sec 4m14s: 1462.2/sec 4856.6/sec | 1625.7/sec 5205.9/sec | 3522.7/sec 4652.2/sec 4m15s: 4982.9/sec 4857.1/sec | 368.0/sec 5186.9/sec |10206.9/sec 4674.1/sec 4m16s:10435.8/sec 4878.9/sec |11150.1/sec 5210.2/sec | 617.9/sec 4658.3/sec 4m17s: 5842.5/sec 4882.6/sec | 2585.7/sec 5200.0/sec | 3725.3/sec 4654.7/sec 4m18s: 3380.2/sec 4876.8/sec |10330.0/sec 5219.9/sec | 5500.1/sec 4658.0/sec 4m19s: 6659.0/sec 4883.7/sec | 7819.7/sec 5229.9/sec | 6104.0/sec 4663.6/sec 4m20s: 8776.2/sec 4898.6/sec | 0.0/sec 5209.8/sec | 4216.6/sec 4661.8/sec 4m21s: 6155.8/sec 4903.5/sec | 6136.3/sec 5213.3/sec | 7253.9/sec 4671.8/sec 4m22s: 3866.0/sec 4899.5/sec | 3681.5/sec 5207.5/sec | 29.0/sec 4654.1/sec 4m23s: 8734.5/sec 4914.1/sec | 8508.6/sec 5220.0/sec | 2136.9/sec 4644.5/sec 4m24s: 3044.3/sec 4907.0/sec |10632.0/sec 5240.5/sec |10642.7/sec 4667.2/sec 4m25s: 5469.2/sec 4909.1/sec | 3237.3/sec 5233.0/sec | 3578.8/sec 4663.1/sec 4m26s: 141.0/sec 4891.2/sec | 2676.2/sec 5223.3/sec | 84.0/sec 4645.9/sec 4m27s: 9001.3/sec 4906.6/sec | 5368.3/sec 5223.9/sec | 9260.7/sec 4663.2/sec 4m28s: 2234.9/sec 4896.6/sec | 2853.3/sec 5215.0/sec | 2918.7/sec 4656.6/sec 4m29s: 9095.2/sec 4912.2/sec | 5955.1/sec 5217.8/sec | 2949.2/sec 4650.3/sec 4m30s: 1032.0/sec 4897.9/sec | 2187.2/sec 5206.6/sec |10807.0/sec 4673.1/sec 4m31s: 4271.4/sec 4895.5/sec | 4461.0/sec 5203.8/sec | 5457.1/sec 4676.0/sec 4m32s: 1673.0/sec 4883.7/sec | 4818.8/sec 5202.4/sec | 6558.1/sec 4682.9/sec 4m33s: 3209.4/sec 4877.6/sec | 5941.2/sec 5205.1/sec | 5019.6/sec 4684.1/sec 4m34s: 288.0/sec 4860.8/sec | 2781.2/sec 5196.3/sec | 2193.2/sec 4675.1/sec 4m35s: 3475.2/sec 4855.8/sec | 1115.3/sec 5181.4/sec | 7251.0/sec 4684.4/sec 4m36s: 8060.5/sec 4867.4/sec | 1429.7/sec 5167.8/sec | 296.0/sec 4668.5/sec 4m37s: 85.0/sec 4850.1/sec | 4989.5/sec 5167.2/sec | 5162.4/sec 4670.3/sec 4m38s: 6838.7/sec 4857.3/sec | 1986.1/sec 5155.7/sec | 3189.4/sec 4665.0/sec 4m39s: 3861.7/sec 4853.7/sec | 98.0/sec 5137.6/sec | 8847.6/sec 4680.0/sec 4m40s: 8261.0/sec 4865.9/sec | 3867.6/sec 5133.1/sec | 2022.0/sec 4670.5/sec 4m41s: 3799.3/sec 4862.1/sec | 4147.5/sec 5129.6/sec | 1072.0/sec 4657.7/sec 4m42s: 1449.2/sec 4850.0/sec | 8232.9/sec 5140.6/sec | 8564.0/sec 4671.5/sec 4m43s: 3237.0/sec 4844.3/sec | 2193.8/sec 5130.2/sec | 2522.2/sec 4663.9/sec 4m44s: 2437.2/sec 4835.8/sec | 0.0/sec 5112.1/sec | 2106.9/sec 4654.9/sec 4m45s: 5582.3/sec 4838.4/sec | 0.0/sec 5094.2/sec | 4277.9/sec 4653.6/sec 4m46s: 4077.4/sec 4835.8/sec | 1699.9/sec 5082.3/sec | 6279.3/sec 4659.3/sec 4m47s: 1518.0/sec 4824.2/sec | 9092.4/sec 5096.2/sec | 5723.4/sec 4663.0/sec 4m48s: 5973.9/sec 4828.2/sec | 1382.1/sec 5083.4/sec | 2328.0/sec 4654.9/sec 4m49s: 1477.9/sec 4816.6/sec | 3623.8/sec 5078.3/sec |10018.7/sec 4673.5/sec 4m50s: 19.0/sec 4800.1/sec | 7984.6/sec 5088.3/sec | 6391.7/sec 4679.4/sec 4m51s: 1320.0/sec 4788.1/sec | 1828.9/sec 5077.1/sec | 7006.3/sec 4687.4/sec 4m52s: 9922.4/sec 4805.7/sec | 1778.0/sec 5065.8/sec | 507.0/sec 4673.1/sec 4m53s: 5176.3/sec 4806.9/sec | 39.0/sec 5048.7/sec | 646.0/sec 4659.3/sec 4m54s: 197.0/sec 4791.3/sec | 1036.1/sec 5035.0/sec | 1559.8/sec 4648.8/sec 4m55s: 22.0/sec 4775.1/sec | 8240.7/sec 5045.9/sec | 196.0/sec 4633.7/sec 4m56s:10154.1/sec 4793.3/sec | 9409.3/sec 5060.6/sec | 5429.9/sec 4636.4/sec 4m57s: 2641.1/sec 4786.0/sec | 8583.7/sec 5072.5/sec | 880.0/sec 4623.7/sec 4m58s:10420.8/sec 4804.9/sec | 692.0/sec 5057.8/sec | 935.0/sec 4611.3/sec 4m59s: 599.9/sec 4790.9/sec | 789.0/sec 5043.5/sec | 0.0/sec 4595.9/sec 5m0s: 3067.3/sec 4785.1/sec | 34.0/sec 5026.8/sec | 329.1/sec 4581.7/sec

The roachpb.Method type is sometimes a convenient shorthand for referring to different kinds of roachpb.Requests, but where it matters we always use type switches. The Method of a request has occasionally diverged from its type, leading to problems as in cockroachdb#11919. Additionally, as a part of cleaning up cockroachdb#10084, I'm about to introduce a new type storage.Command which makes roachpb.Method feel redundant.

All the "command" methods of Replica are now resolved through a Command object. This gives us a place to put new methods associated with commands, for a cleaned-up reintroduction of cockroachdb#10084.

Adds DeclareKeys implementations for all commands. Does not yet cover keys whose use is implied by use of in-memory fields of the replica state. This is an update and cleaned-up version of cockroachdb#10084.

Access the abort cache through the same batch used for the rest of the evaluation, instead of going directly to store.Engine (this adds SpanSet enforcement to it, as discussed in cockroachdb#10084). This also consolidates two bits of slightly different code on the read and write paths, and removes a special case that exempted HeartbeatTxn from abort cache checks.

The command queue issue cockroachdb#10084 that caused parts of this test to be disabled has been resolved.

tbg · 2017-04-27T17:40:06Z

This PR is obsolete given @bdarnell's work on proposer-evaluated KV.

Now that the command-queue issues in cockroachdb#10084 have been fixed, the only reason to hold raftMu during evaluation was because it controlled access to the replicaStateLoader (and lock-ordering requirements forced us to give the lock a broad scope). By forking the replicaStateLoader into several copies (one protected by Replica.mu and one by Replica.raftMu, plus a few temporary loaders created off the critical path), we can reduce the scope of this coarse-grained lock (which in turn facilitates cockroachdb#15802).

tbg mentioned this pull request Oct 19, 2016

storage: implement proposer-evaluated kv #6290

Closed

petermattis reviewed Oct 19, 2016

View reviewed changes

bdarnell reviewed Oct 20, 2016

View reviewed changes

tbg force-pushed the assert-cmdq branch 2 times, most recently from 6ef9db3 to 725e79f Compare October 20, 2016 14:14

petermattis mentioned this pull request Oct 20, 2016

storage: evaluate proposals without lock #10012

Merged

tbg force-pushed the assert-cmdq branch from 725e79f to a9599a4 Compare October 24, 2016 11:37

bdarnell mentioned this pull request Dec 23, 2016

storage: Introduce a Command type to replica_command.go #12561

Merged

bdarnell mentioned this pull request Feb 5, 2017

storage: Refactor span collection for command queue #13418

Merged

bdarnell mentioned this pull request Mar 1, 2017

storage: Enforce that all keys accessed are declared in SpanSet #13864

Merged

bdarnell mentioned this pull request Mar 31, 2017

storage: Check SpanSet access in ReplicaEvalContext #14524

Merged

bdarnell mentioned this pull request Apr 3, 2017

storage: Move AbortCache check to evaluateBatch #14567

Merged

bdarnell added a commit to bdarnell/cockroach that referenced this pull request Apr 13, 2017

storage: Remove a test that was partially skipped in propEvalKV

bb9d65b

The command queue issue cockroachdb#10084 that caused parts of this test to be disabled has been resolved.

tbg closed this Apr 27, 2017

bdarnell mentioned this pull request May 15, 2017

storage: Avoid holding raftMu during evaluation #15935

Merged

tbg deleted the assert-cmdq branch May 8, 2018 15:03

Conversation

tbg commented Oct 19, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

petermattis commented Oct 19, 2016

Uh oh!

petermattis commented Oct 19, 2016

Uh oh!

tbg commented Oct 19, 2016

Uh oh!

petermattis commented Oct 19, 2016

Uh oh!

tbg commented Oct 19, 2016

Uh oh!

tbg commented Oct 19, 2016

Uh oh!

petermattis commented Oct 19, 2016

Uh oh!

tbg commented Oct 19, 2016

Uh oh!

tbg commented Oct 19, 2016

Uh oh!

petermattis Oct 19, 2016

Choose a reason for hiding this comment

Uh oh!

tbg Oct 20, 2016

Choose a reason for hiding this comment

Uh oh!

petermattis Oct 19, 2016

Choose a reason for hiding this comment

Uh oh!

tbg Oct 20, 2016

Choose a reason for hiding this comment

Uh oh!

petermattis commented Oct 19, 2016

Uh oh!

bdarnell left a comment

Choose a reason for hiding this comment

Uh oh!

bdarnell Oct 20, 2016

Choose a reason for hiding this comment

Uh oh!

tbg Oct 20, 2016

Choose a reason for hiding this comment

Uh oh!

tbg commented Oct 20, 2016

Uh oh!

petermattis commented Oct 20, 2016

Uh oh!

tbg commented Oct 20, 2016

Uh oh!

tbg commented Oct 20, 2016

Uh oh!

petermattis commented Oct 20, 2016

Uh oh!

nvb commented Oct 21, 2016

Uh oh!

bdarnell commented Oct 23, 2016

Uh oh!

tbg commented Oct 23, 2016

Uh oh!

tbg commented Oct 24, 2016

Uh oh!

tbg commented Apr 27, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

tbg commented Oct 19, 2016 •

edited

Loading