kv/kvserver: BenchmarkMVCCGCWithForegroundTraffic fails with expected nil #96266
Description
To reproduce:
./dev bench pkg/kv/kvserver -f=BenchmarkMVCCGCWithForegroundTraffic --ignore-cache --bench-time=2s --verbose --stream-output --timeout 10m --test-args="-test.failfast"
Note: Seems to require quite a bit of iterations for the failure to happen, 2s or > 200k iterations seems to reproduce it reliably on my machine.
Failure:
BenchmarkMVCCGCWithForegroundTraffic
test_log_scope.go:161: test logs captured to: /Users/herko/go/src/github.com/cockroachdb/cockroach/tmp/_tmp/577bfc8ed46850d1aaafa923fa37ce78/logBenchmarkMVCCGCWithForegroundTraffic2899829498
test_log_scope.go:79: use -show-logs to present logs inline
BenchmarkMVCCGCWithForegroundTraffic/noop_gc_with_reads
BenchmarkMVCCGCWithForegroundTraffic/noop_gc_with_reads-10 154018 7660 ns/op
BenchmarkMVCCGCWithForegroundTraffic/gc_with_reads_and_writes
replica_test.go:8776:
Error Trace: /private/var/tmp/_bazel_herko/7349b8c7c2d76abd184572c7559a04c3/sandbox/darwin-sandbox/3595/execroot/com_github_cockroachdb_cockroach/bazel-out/darwin_arm64-fastbuild/bin/pkg/kv/kvserver/kvserver_test_/kvserver_test.runfiles/com_github_cockroachdb_cockroach/pkg/kv/kvserver/replica_test.go:8776
/private/var/tmp/_bazel_herko/7349b8c7c2d76abd184572c7559a04c3/sandbox/darwin-sandbox/3595/execroot/com_github_cockroachdb_cockroach/bazel-out/darwin_arm64-fastbuild/bin/pkg/kv/kvserver/kvserver_test_/kvserver_test.runfiles/com_github_cockroachdb_cockroach/pkg/kv/kvserver/replica_test.go:8785
/private/var/tmp/_bazel_herko/7349b8c7c2d76abd184572c7559a04c3/sandbox/darwin-sandbox/3595/execroot/com_github_cockroachdb_cockroach/bazel-out/darwin_arm64-fastbuild/bin/pkg/kv/kvserver/kvserver_test_/kvserver_test.runfiles/com_github_cockroachdb_cockroach/pkg/kv/kvserver/replica_test.go:8845
/private/var/tmp/_bazel_herko/7349b8c7c2d76abd184572c7559a04c3/sandbox/darwin-sandbox/3595/execroot/com_github_cockroachdb_cockroach/bazel-out/darwin_arm64-fastbuild/bin/pkg/kv/kvserver/kvserver_test_/kvserver_test.runfiles/com_github_cockroachdb_cockroach/pkg/kv/kvserver/asm_arm64.s:1172
Error: Expected nil, but got: &roachpb.Error{UnexposedTxn:(*roachpb.Transaction)(nil), OriginNode:0, EncodedError:errorspb.EncodedError{Error:(*errorspb.EncodedError_Wrapper)(0x14001cf7c40)}, Index:(*roachpb.ErrPosition)(0x140025e8cbc), Now:hlc.ClockTimestamp{WallTime:0, Logical:0, Synthetic:false}}
Test: BenchmarkMVCCGCWithForegroundTraffic
replica_test.go:8776:
Error Trace: /private/var/tmp/_bazel_herko/7349b8c7c2d76abd184572c7559a04c3/sandbox/darwin-sandbox/3595/execroot/com_github_cockroachdb_cockroach/bazel-out/darwin_arm64-fastbuild/bin/pkg/kv/kvserver/kvserver_test_/kvserver_test.runfiles/com_github_cockroachdb_cockroach/pkg/kv/kvserver/replica_test.go:8776
/private/var/tmp/_bazel_herko/7349b8c7c2d76abd184572c7559a04c3/sandbox/darwin-sandbox/3595/execroot/com_github_cockroachdb_cockroach/bazel-out/darwin_arm64-fastbuild/bin/pkg/kv/kvserver/kvserver_test_/kvserver_test.runfiles/com_github_cockroachdb_cockroach/pkg/kv/kvserver/replica_test.go:8785
/private/var/tmp/_bazel_herko/7349b8c7c2d76abd184572c7559a04c3/sandbox/darwin-sandbox/3595/execroot/com_github_cockroachdb_cockroach/bazel-out/darwin_arm64-fastbuild/bin/pkg/kv/kvserver/kvserver_test_/kvserver_test.runfiles/com_github_cockroachdb_cockroach/pkg/kv/kvserver/replica_test.go:8845
/private/var/tmp/_bazel_herko/7349b8c7c2d76abd184572c7559a04c3/sandbox/darwin-sandbox/3595/execroot/com_github_cockroachdb_cockroach/bazel-out/darwin_arm64-fastbuild/bin/pkg/kv/kvserver/kvserver_test_/kvserver_test.runfiles/com_github_cockroachdb_cockroach/pkg/kv/kvserver/asm_arm64.s:1172
Error: Expected nil, but got: &roachpb.Error{UnexposedTxn:(*roachpb.Transaction)(nil), OriginNode:0, EncodedError:errorspb.EncodedError{Error:(*errorspb.EncodedError_Wrapper)(0x1400190a728)}, Index:(*roachpb.ErrPosition)(0x1400335807c), Now:hlc.ClockTimestamp{WallTime:0, Logical:0, Synthetic:false}}
Test: BenchmarkMVCCGCWithForegroundTraffic
BenchmarkMVCCGCWithForegroundTraffic/gc_with_reads_and_writes-10 102026 10575 ns/op
BenchmarkMVCCGCWithForegroundTraffic/gc_with_reads
BenchmarkMVCCGCWithForegroundTraffic/gc_with_reads-10 99932 16486 ns/op
replica_test.go:8901: -- test log scope end --
test logs left over in: /Users/herko/go/src/github.com/cockroachdb/cockroach/tmp/_tmp/577bfc8ed46850d1aaafa923fa37ce78/logBenchmarkMVCCGCWithForegroundTraffic2899829498
--- FAIL: BenchmarkMVCCGCWithForegroundTraffic
FAIL
I230131 11:48:23.158268 1 (gostd) testmain.go:1492 [T1] 1 Test //pkg/kv/kvserver:kvserver_test exited with error code 1
Error:
Error: Expected nil, but got: &roachpb.Error{UnexposedTxn:(*roachpb.Transaction)(nil), OriginNode:0, EncodedError:errorspb.EncodedError{Error:(*errorspb.EncodedError_Wrapper)(0x1400190a728)}, Index:(*roachpb.ErrPosition)(0x1400335807c), Now:hlc.ClockTimestamp{WallTime:0, Logical:0, Synthetic:false}}
Jira issue: CRDB-24035