kvserver: panic: use of Span after Finish from RangeFeed code #82603
Description
Describe the problem
This was seen use of Span after Finish panics in a few different nightly failures.
- kv/kvserver/protectedts/ptstorage: TestStorage failed [use of Span after Finish] #82579
- jobs: TestRegistryUsePartialIndex failed #82581
- sql/importer: TestReadsAvroRecords failed #82585
They all seem to have the same source:
08:28:49 * ERROR: a panic has occurred!
08:28:49 * panic: use of Span after Finish. Span: dist sender. Finish previously called at: <stack not captured. Set debugUseAfterFinish>
08:28:49 * (1) attached stack trace
08:28:49 * -- stack trace:
08:28:49 * | runtime.gopanic
08:28:49 * | GOROOT/src/runtime/panic.go:1038
08:28:49 * | github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).detectUseAfterFinish
08:28:49 * | github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:186
08:28:49 * | github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).Tracer
08:28:49 * | github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:229
08:28:49 * | github.com/cockroachdb/cockroach/pkg/util/tracing.EnsureForkSpan
08:28:49 * | github.com/cockroachdb/cockroach/pkg/util/tracing/tracer.go:1554
08:28:49 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleClosedTimestampUpdateRaftMuLocked
08:28:49 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_rangefeed.go:698
08:28:49 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).registerWithRangefeedRaftMuLocked
08:28:49 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_rangefeed.go:428
08:28:49 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).rangeFeedWithRangeID
08:28:49 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_rangefeed.go:228
08:28:49 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).RangeFeed
08:28:49 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_rangefeed.go:139
08:28:49 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).RangeFeed
08:28:49 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store.go:3088
08:28:49 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).RangeFeed
08:28:49 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:216
08:28:49 * | github.com/cockroachdb/cockroach/pkg/server.(*Node).RangeFeed
08:28:49 * | github.com/cockroachdb/cockroach/pkg/server/node.go:1225
08:28:49 * | github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed.func2.1
08:28:49 * | github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:934
08:28:49 * | github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor.StreamServerInterceptor.func1
08:28:49 * | github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:163
08:28:49 * | github.com/cockroachdb/cockroach/pkg/rpc.serverStreamInterceptorsChain.run.func1
08:28:49 * | github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:825
08:28:49 * | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4
08:28:49 * | github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:273
08:28:49 * | github.com/cockroachdb/cockroach/pkg/rpc.serverStreamInterceptorsChain.run.func1
08:28:49 * | github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:828
08:28:49 * | github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor
08:28:49 * | github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:98
08:28:49 * | github.com/cockroachdb/cockroach/pkg/rpc.serverStreamInterceptorsChain.run.func1
08:28:49 * | github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:828
08:28:49 * | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1
08:28:49 * | github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:242
08:28:49 * | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
08:28:49 * | github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:344
08:28:49 * | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2
08:28:49 * | github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:241
08:28:49 * | github.com/cockroachdb/cockroach/pkg/rpc.serverStreamInterceptorsChain.run.func1
08:28:49 * | github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:828
08:28:49 * | github.com/cockroachdb/cockroach/pkg/rpc.serverStreamInterceptorsChain.run
08:28:49 * | github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:830
08:28:49 * | github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed.func2
08:28:49 * | github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:938
08:28:49 * | runtime.goexit
08:28:49 * | GOROOT/src/runtime/asm_amd64.s:1581
08:28:49 * Wraps: (2) panic: use of Span after Finish. Span: dist sender. Finish previously called at: <stack not captured. Set debugUseAfterFinish>
08:28:49 * Error types: (1) *withstack.withStack (2) *errutil.leafError
08:28:49 *
08:28:49 panic: use of Span after Finish. Span: dist sender. Finish previously called at: <stack not captured. Set debugUseAfterFinish> [recovered]
08:28:49 panic: use of Span after Finish. Span: dist sender. Finish previously called at: <stack not captured. Set debugUseAfterFinish>
08:28:49
08:28:49 goroutine 52747530 [running]:
08:28:49 github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).detectUseAfterFinish(0x99e7b28)
08:28:49 github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:186 +0x134
08:28:49 github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).RecordingType(0xc00260f180)
08:28:49 github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:390 +0x1e
08:28:49 github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).IsVerbose(...)
08:28:49 github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:398
08:28:49 github.com/cockroachdb/cockroach/pkg/util/log.getSpanOrEventLog({0x6818c48, 0xc015ee4a20})
08:28:49 github.com/cockroachdb/cockroach/pkg/util/log/trace.go:92 +0x3a
08:28:49 github.com/cockroachdb/cockroach/pkg/util/log.logfDepth({0x6818c48, 0xc015ee4a20}, 0x2, 0x3, 0x1, {0x5115998, 0x19}, {0xc004dc0ec8, 0x1, 0x1})
08:28:49 github.com/cockroachdb/cockroach/pkg/util/log/channels.go:55 +0x2a8
08:28:49 github.com/cockroachdb/cockroach/pkg/util/log.shoutfDepth({0x6818c48, 0xc015ee4a20}, 0x1, 0x3, 0x0, {0x5115998, 0x19}, {0xc004dc0ec8, 0x1, 0x1})
08:28:49 github.com/cockroachdb/cockroach/pkg/util/log/channels.go:84 +0x28d
08:28:49 github.com/cockroachdb/cockroach/pkg/util/log.loggerOps.Shoutf(...)
08:28:49 github.com/cockroachdb/cockroach/bazel-out/k8-dbg/bin/pkg/util/log/log_channels_generated.go:1422
08:28:49 github.com/cockroachdb/cockroach/pkg/util/log/logcrash.ReportPanic({0x6818c48, 0xc015ee4a20}, 0xc004dc0f40, {0x47f3ae0, 0xc019117f50}, 0xc004dc0f40)
08:28:49 github.com/cockroachdb/cockroach/pkg/util/log/logcrash/crash_reporting.go:174 +0xc5
08:28:49 github.com/cockroachdb/cockroach/pkg/util/log/logcrash.ReportPanicWithGlobalSettings({0x6818c48, 0xc015ee4a20}, {0x47f3ae0, 0xc019117f50}, 0xc0078eec40)
08:28:49 github.com/cockroachdb/cockroach/pkg/util/log/logcrash/crash_reporting.go:114 +0x97
08:28:49 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Recover(0xc015283cb0, {0x6818c48, 0xc015ee4a20})
08:28:49 github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:250 +0x85
08:28:49 panic({0x47f3ae0, 0xc019117f50})
08:28:49 GOROOT/src/runtime/panic.go:1038 +0x215
08:28:49 github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).detectUseAfterFinish(0x6818c48)
08:28:49 github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:186 +0x134
08:28:49 github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).Tracer(...)
08:28:49 github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:229
08:28:49 github.com/cockroachdb/cockroach/pkg/util/tracing.EnsureForkSpan({0x6818c48, 0xc019ab41b0}, 0xc00fb4d770, {0xc00999b9b0, 0x29})
08:28:49 github.com/cockroachdb/cockroach/pkg/util/tracing/tracer.go:1554 +0x9a
08:28:49 github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleClosedTimestampUpdateRaftMuLocked(0xc00b256400, {0x6818c48, 0xc019ab41b0}, {0xc00b6ab740, 0x15615020, 0xc0})
08:28:49 github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_rangefeed.go:698 +0x314
08:28:49 github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).registerWithRangefeedRaftMuLocked(0xc00b256400, {0x6818c48, 0xc019ab41b0}, {{0xc01dc8c0f8, 0x1, 0x8}, {0xc01dc8c100, 0x1, 0x8}}, {0x16f697e66fdd1849, ...}, ...)
08:28:49 github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_rangefeed.go:428 +0x755
08:28:49 github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).rangeFeedWithRangeID(0xc00b256400, 0x688ec70, 0xc00bf9b8c0, {0x7f981d0db1b0, 0xc01bf4e300})
08:28:49 github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_rangefeed.go:228 +0x5d9
08:28:49 github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).RangeFeed(...)
08:28:49 github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_rangefeed.go:139
08:28:49 github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).RangeFeed(0xc00166b500, 0xc00bf9b8c0, {0x688ec70, 0xc01bf4e300})
08:28:49 github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store.go:3088 +0xf7
08:28:49 github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).RangeFeed(0xc018be9868, 0xc00bf9b8c0, {0x688ec70, 0xc01bf4e300})
08:28:49 github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:216 +0xe8
08:28:49 github.com/cockroachdb/cockroach/pkg/server.(*Node).RangeFeed(0x688ec70, 0xc000100400, {0x688ec70, 0xc01bf4e300})
08:28:49 github.com/cockroachdb/cockroach/pkg/server/node.go:1225 +0x38
08:28:49 github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed.func2.1({0xc01ca348e0, 0x6818c48}, {0xc015ee4a20, 0x51655f5})
08:28:49 github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:934 +0x7b
08:28:49 github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor.StreamServerInterceptor.func1({0x502f3e0, 0xc008b76d00}, {0x6853ab8, 0xc01ca348e0}, 0x94a9b70, 0xc00ff38c00)
08:28:49 github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:163 +0x6c2
08:28:49 github.com/cockroachdb/cockroach/pkg/rpc.serverStreamInterceptorsChain.run.func1({0x502f3e0, 0xc008b76d00}, {0x6853ab8, 0xc01ca348e0})
08:28:49 github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:825 +0x5b
08:28:49 github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4({0x502f3e0, 0xc008b76d00}, {0x6853ab8, 0xc01ca348e0}, 0xc008402780, 0xc009bfb400)
08:28:49 github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:273 +0x74
08:28:49 github.com/cockroachdb/cockroach/pkg/rpc.serverStreamInterceptorsChain.run.func1({0x502f3e0, 0xc008b76d00}, {0x6853ab8, 0xc01ca348e0})
08:28:49 github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:828 +0x83
08:28:49 github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor({{{0xc015283cd0}}}, {0x502f3e0, 0xc008b76d00}, {0x6853ab8, 0xc01ca348e0}, 0x94a9b70, 0xc009bfb400)
08:28:49 github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:98 +0x1c3
08:28:49 github.com/cockroachdb/cockroach/pkg/rpc.serverStreamInterceptorsChain.run.func1({0x502f3e0, 0xc008b76d00}, {0x6853ab8, 0xc01ca348e0})
08:28:49 github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:828 +0x83
08:28:49 github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1({0xc015283cb0, 0x0})
08:28:49 github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:242 +0x2d
08:28:49 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc015283cb0, {0x6818c48, 0xc015ee4a20}, {0x452594, 0x929746}, 0xc0115b7e58)
08:28:49 github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:344 +0xdd
08:28:49 github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2({0x502f3e0, 0xc008b76d00}, {0x6853ab8, 0xc01ca348e0}, 0x94a9b70, 0xc009bfb400)
08:28:49 github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:241 +0xf3
08:28:49 github.com/cockroachdb/cockroach/pkg/rpc.serverStreamInterceptorsChain.run.func1({0x502f3e0, 0xc008b76d00}, {0x6853ab8, 0xc01ca348e0})
08:28:49 github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:828 +0x83
08:28:49 github.com/cockroachdb/cockroach/pkg/rpc.serverStreamInterceptorsChain.run({0xc010b33700, 0x4, 0x4}, {0x502f3e0, 0xc008b76d00}, {0x6853ab8, 0xc01ca348e0}, 0x94a9b70, 0xc00ff38c00)
08:28:49 github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:830 +0x142
08:28:49 github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed.func2()
08:28:49 github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:938 +0x1f5
08:28:49 created by github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed
08:28:49 github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:931 +0x35c
08:28:49 I220608 08:28:49.567884 1 (gostd) testmain.go:246 [-] 1 Test //pkg/sql/importer:importer_test exited with error code 2
and
07:27:26 panic: use of Span after Finish. Span: dist sender. Finish previously called at: <stack not captured. Set debugUseAfterFinish> [recovered]
07:27:26 panic: use of Span after Finish. Span: dist sender. Finish previously called at: <stack not captured. Set debugUseAfterFinish>
07:27:26
07:27:26 goroutine 13743713 [running]:
07:27:26 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Recover(0xc000b63320, {0x579e418, 0xc010563170})
07:27:26 github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:251 +0x94
07:27:26 panic({0x3bf6240, 0xc0090f38a0})
07:27:26 GOROOT/src/runtime/panic.go:1038 +0x215
07:27:26 github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).detectUseAfterFinish(0x579e418)
07:27:26 github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:186 +0x134
07:27:26 github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).Tracer(...)
07:27:26 github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:229
07:27:26 github.com/cockroachdb/cockroach/pkg/util/tracing.EnsureForkSpan({0x579e418, 0xc0105632f0}, 0xc00ced04b0, {0xc0058b8ea0, 0x29})
07:27:26 github.com/cockroachdb/cockroach/pkg/util/tracing/tracer.go:1554 +0x9a
07:27:26 github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleClosedTimestampUpdateRaftMuLocked(0xc005ede000, {0x579e418, 0xc0105632f0}, {0xc01c7b24e0, 0x9b7cd80, 0xc0})
07:27:26 github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_rangefeed.go:698 +0x314
07:27:26 github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).registerWithRangefeedRaftMuLocked(0xc005ede000, {0x579e418, 0xc0105632f0}, {{0xc00b1cda18, 0x1, 0x8}, {0xc00b1cda20, 0x1, 0x8}}, {0x16f6948d0ca38f87, ...}, ...)
07:27:26 github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_rangefeed.go:428 +0x755
07:27:26 github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).rangeFeedWithRangeID(0xc005ede000, 0x58016b0, 0xc0148525a0, {0x7f2668941348, 0xc015eabde0})
07:27:26 github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_rangefeed.go:228 +0x5d9
07:27:26 github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).RangeFeed(...)
07:27:26 github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_rangefeed.go:139
07:27:26 github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).RangeFeed(0xc004b21500, 0xc0148525a0, {0x58016b0, 0xc015eabde0})
07:27:26 github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store.go:3088 +0xf7
07:27:26 github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).RangeFeed(0xc0013d5868, 0xc0148525a0, {0x58016b0, 0xc015eabde0})
07:27:26 github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:216 +0xe8
07:27:26 github.com/cockroachdb/cockroach/pkg/server.(*Node).RangeFeed(0x58016b0, 0xc000212400, {0x58016b0, 0xc015eabde0})
07:27:26 github.com/cockroachdb/cockroach/pkg/server/node.go:1225 +0x38
07:27:26 github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed.func2.1({0xc015eabda0, 0x579e418}, {0xc010563170, 0x4354c70})
07:27:26 github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:934 +0x7b
07:27:26 github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor.StreamServerInterceptor.func1({0x426a220, 0xc014839600}, {0x57d43f0, 0xc015eabda0}, 0x7c4a130, 0xc018cf9680)
07:27:26 github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:163 +0x6c2
07:27:26 github.com/cockroachdb/cockroach/pkg/rpc.serverStreamInterceptorsChain.run.func1({0x426a220, 0xc014839600}, {0x57d43f0, 0xc015eabda0})
07:27:26 github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:825 +0x5b
07:27:26 github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4({0x426a220, 0xc014839600}, {0x57d43f0, 0xc015eabda0}, 0xc00786a720, 0xc01704e840)
07:27:26 github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:273 +0x74
07:27:26 github.com/cockroachdb/cockroach/pkg/rpc.serverStreamInterceptorsChain.run.func1({0x426a220, 0xc014839600}, {0x57d43f0, 0xc015eabda0})
07:27:26 github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:828 +0x83
07:27:26 github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor({{{0xc000b63340}}}, {0x426a220, 0xc014839600}, {0x57d43f0, 0xc015eabda0}, 0x7c4a130, 0xc01704e840)
07:27:26 github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:98 +0x1c3
07:27:26 github.com/cockroachdb/cockroach/pkg/rpc.serverStreamInterceptorsChain.run.func1({0x426a220, 0xc014839600}, {0x57d43f0, 0xc015eabda0})
07:27:26 github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:828 +0x83
07:27:26 github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1({0xc000b63320, 0x0})
07:27:26 github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:242 +0x2d
07:27:26 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc000b63320, {0x579e418, 0xc010563170}, {0x4286d4, 0x8f5c06}, 0xc00338be58)
07:27:26 github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:344 +0xdd
07:27:26 github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2({0x426a220, 0xc014839600}, {0x57d43f0, 0xc015eabda0}, 0x7c4a130, 0xc01704e840)
07:27:26 github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:241 +0xf3
07:27:26 github.com/cockroachdb/cockroach/pkg/rpc.serverStreamInterceptorsChain.run.func1({0x426a220, 0xc014839600}, {0x57d43f0, 0xc015eabda0})
07:27:26 github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:828 +0x83
07:27:26 github.com/cockroachdb/cockroach/pkg/rpc.serverStreamInterceptorsChain.run({0xc00b3464a0, 0x4, 0x4}, {0x426a220, 0xc014839600}, {0x57d43f0, 0xc015eabda0}, 0x7c4a130, 0xc018cf9680)
07:27:26 github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:830 +0x142
07:27:26 github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed.func2()
07:27:26 github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:938 +0x1f5
07:27:26 created by github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed
07:27:26 github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:931 +0x35c
07:27:26 I220608 07:27:26.747876 1 (gostd) testmain.go:198 [-] 1 Test //pkg/jobs:jobs_test exited with error code 2
Jira issue: CRDB-16530