kv: major performance regression in kv workload on single node GCE after txn.go changes #30208
Description
It appears that #28185 caused a big performance regression. Does not show up on macs, but happens on GCE - which might be because the workload's default concurrency is much higher there.
bin/workload run kv --read-percent 95 --duration 30s
Before:
_elapsed___errors_____ops(total)___ops/sec(cum)__avg(ms)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)__total
16.0s 0 311615 19499.6 1.8 1.4 4.5 5.8 35.7 read
16.0s 0 16540 1035.0 12.9 12.6 22.0 27.3 39.8 write
After:
_elapsed___errors_____ops(total)___ops/sec(cum)__avg(ms)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)__total
30.0s 0 367987 12265.9 2.8 2.9 5.8 7.1 37.7 read
30.0s 0 19597 653.2 20.1 19.9 33.6 37.7 50.3 write
The perf diff doesn't appear to happen with --read-percent 100, which would indicate that we need to focus on --read-percent 0, but I haven't done more tests yet.
One first suspicion is that perhaps this workload only does 1PC txns, in which case we're not spinning up heartbeat loops that we weren't before?
cc @jordanlewis