kvclient: regression in BenchmarkColBatchScan due to dist sender metrics #111142
Description
I bisected most of the regression on BenchmarkColBatchScan from pkg/sql/colflow between 23.1 and master to 3502ca2
name old time/op new time/op delta
ColBatchScan/rows=16-24 72.4µs ± 7% 96.4µs ± 5% +33.18% (p=0.000 n=10+10)
ColBatchScan/rows=256-24 168µs ± 4% 195µs ± 6% +15.65% (p=0.000 n=9+9)
ColBatchScan/rows=4096-24 1.40ms ± 1% 1.41ms ± 1% +1.23% (p=0.001 n=8+8)
ColBatchScan/rows=65536-24 21.8ms ± 1% 21.7ms ± 1% ~ (p=0.481 n=10+10)
name old speed new speed delta
ColBatchScan/rows=16-24 3.54MB/s ± 7% 2.66MB/s ± 5% -24.96% (p=0.000 n=10+10)
ColBatchScan/rows=256-24 24.3MB/s ± 4% 21.0MB/s ± 5% -13.52% (p=0.000 n=9+9)
ColBatchScan/rows=4096-24 46.9MB/s ± 1% 46.3MB/s ± 1% -1.21% (p=0.001 n=8+8)
ColBatchScan/rows=65536-24 48.1MB/s ± 1% 48.2MB/s ± 1% ~ (p=0.469 n=10+10)
name old alloc/op new alloc/op delta
ColBatchScan/rows=16-24 8.21kB ±11% 8.78kB ± 9% ~ (p=0.053 n=10+9)
ColBatchScan/rows=256-24 28.0kB ± 1% 28.8kB ± 1% +3.15% (p=0.000 n=9+9)
ColBatchScan/rows=4096-24 195kB ±14% 196kB ±14% ~ (p=0.195 n=8+8)
ColBatchScan/rows=65536-24 2.19MB ± 1% 2.19MB ± 1% ~ (p=0.815 n=9+8)
name old allocs/op new allocs/op delta
ColBatchScan/rows=16-24 114 ±10% 119 ± 6% ~ (p=0.076 n=10+9)
ColBatchScan/rows=256-24 158 ± 1% 168 ± 1% +6.55% (p=0.000 n=9+9)
ColBatchScan/rows=4096-24 246 ±130% 254 ±127% +3.61% (p=0.046 n=8+8)
ColBatchScan/rows=65536-24 292 ±24% 326 ±39% +11.82% (p=0.035 n=8+8)
My guess (based on a quick glance at the code) is that this is due to having to compute ba.Size() and br.Size(), so we might not be able to do much about it, but will defer to KV.
Jira issue: CRDB-31787