ccl/sqlproxyccl: TestCancelQuery failed #136573
Description
ccl/sqlproxyccl.TestCancelQuery failed on release-24.3.0-rc @ 330209cefef4383a9d67ce7d86e2d3a9794ccd4f:
Fatal error:
panic: use of Span after Finish. Span: proxy-con-serve. Finish previously called at: <stack not captured. Set debugUseAfterFinish> [recovered]
panic: use of Span after Finish. Span: proxy-con-serve. Finish previously called at: <stack not captured. Set debugUseAfterFinish>
Stack:
goroutine 4736 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xc00aaf8c70?, {0xe2ff2d8, 0xc00bf40210})
pkg/util/stop/stopper.go:226 +0x74
panic({0xa6d0c20?, 0xc0088a63b0?})
GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).detectUseAfterFinish(0xc005adc408)
pkg/util/tracing/span.go:177 +0x274
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).RecordingType(0xc005adc408)
pkg/util/tracing/span.go:437 +0x27
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).IsVerbose(...)
pkg/util/tracing/span.go:445
github.com/cockroachdb/cockroach/pkg/util/log.getSpan({0xe2ff170, 0xc0087b4d60})
pkg/util/log/trace.go:22 +0x45
github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0xe2ff170, 0xc0087b4d60}, 0x2, 0x3, 0x1, 0x0, {0xaebfbe7, 0x27}, {0xc00aaf9698, 0x2, ...})
pkg/util/log/channels.go:99 +0x66b
github.com/cockroachdb/cockroach/pkg/util/log.logfDepth(...)
pkg/util/log/channels.go:34
github.com/cockroachdb/cockroach/pkg/util/log.loggerOps.Errorf(...)
bazel-out/k8-fastbuild/bin/pkg/util/log/log_channels_generated.go:1281
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl.(*connector).dialTenantCluster(0xc001b5b200, {0xe2ff170, 0xc0087b4d60}, {0xe2ff020, 0xc003fda000})
pkg/ccl/sqlproxyccl/connector.go:243 +0xcd8
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl.(*connector).OpenTenantConnWithToken(0xc001b5b200, {0xe2ff170, 0xc0087b4d60}, {0xe2ff020, 0xc003fda000}, {0xc003ede0a0, 0x9c})
pkg/ccl/sqlproxyccl/connector.go:106 +0x1ba
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl.transferConnection(0xc0087b4d60, {0xe2ff020, 0xc003fda000}, 0xc001b5b200, 0xc0084265a0, 0xc006491920, 0xc0068c7848)
pkg/ccl/sqlproxyccl/conn_migration.go:274 +0x40d
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl.(*forwarder).TransferConnection(0xc003fda000)
pkg/ccl/sqlproxyccl/conn_migration.go:213 +0x509
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/balancer.(*Balancer).processQueue.func1.1({0xe2ff2d8, 0xc00bf40210})
pkg/ccl/sqlproxyccl/balancer/balancer.go:371 +0x268
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2({0xae02452, 0x11})
pkg/util/stop/stopper.go:498 +0x339
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 4543
pkg/util/stop/stopper.go:488 +0x786
Log preceding fatal error
=== RUN TestCancelQuery
test_log_scope.go:165: test logs captured to: outputs.zip/logTestCancelQuery3134122166
test_log_scope.go:76: use -show-logs to present logs inline
=== RUN TestCancelQuery/cancel_over_sql
=== RUN TestCancelQuery/forward_over_http
=== RUN TestCancelQuery/ignore_unknown_secret_key
*
* ERROR: a panic has occurred!
* use of Span after Finish. Span: proxy-con-serve. Finish previously called at: <stack not captured. Set debugUseAfterFinish>
* (1) attached stack trace
* -- stack trace:
* | runtime.gopanic
* | GOROOT/src/runtime/panic.go:770
* | [...repeated from below...]
* Wraps: (2) assertion failure
* Wraps: (3) attached stack trace
* -- stack trace:
* | github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).detectUseAfterFinish
* | pkg/util/tracing/span.go:177
* | github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).RecordingType
* | pkg/util/tracing/span.go:437
* | github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).IsVerbose
* | pkg/util/tracing/span.go:445
* | github.com/cockroachdb/cockroach/pkg/util/log.getSpan
* | pkg/util/log/trace.go:22
* | github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal
* | pkg/util/log/channels.go:99
* | github.com/cockroachdb/cockroach/pkg/util/log.logfDepth
* | pkg/util/log/channels.go:34
* | github.com/cockroachdb/cockroach/pkg/util/log.loggerOps.Errorf
* | bazel-out/k8-fastbuild/bin/pkg/util/log/log_channels_generated.go:1281
* | github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl.(*connector).dialTenantCluster
* | pkg/ccl/sqlproxyccl/connector.go:243
* | github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl.(*connector).OpenTenantConnWithToken
* | pkg/ccl/sqlproxyccl/connector.go:106
* | github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl.transferConnection
* | pkg/ccl/sqlproxyccl/conn_migration.go:274
* | github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl.(*forwarder).TransferConnection
* | pkg/ccl/sqlproxyccl/conn_migration.go:213
* | github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/balancer.(*Balancer).processQueue.func1.1
* | pkg/ccl/sqlproxyccl/balancer/balancer.go:371
* | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
* | pkg/util/stop/stopper.go:498
* | runtime.goexit
* | src/runtime/asm_amd64.s:1695
* Wraps: (4) use of Span after Finish. Span: proxy-con-serve. Finish previously called at: <stack not captured. Set debugUseAfterFinish>
* Error types: (1) *withstack.withStack (2) *assert.withAssertionFailure (3) *withstack.withStack (4) *errutil.leafError
*
Parameters:
attempt=1race=truerun=3shard=7
Same failure on other branches
- ccl/sqlproxyccl: TestCancelQuery failed #135551 ccl/sqlproxyccl: TestCancelQuery failed [C-test-failure O-robot T-serverless branch-release-24.2 release-blocker]
- ccl/sqlproxyccl: TestCancelQuery failed #131008 ccl/sqlproxyccl: TestCancelQuery failed [C-test-failure O-robot T-cloud-platform branch-release-24.1.5-rc release-blocker]
This test on roachdash | Improve this report!
Jira issue: CRDB-45139