ccl/sqlproxyccl: TestCancelQuery failed

ccl/sqlproxyccl.TestCancelQuery [failed](https://tanzanite.cluster.engflow.com/invocations/default/1df245cc-80b2-4291-8d28-853616b2291d?testReportRun=2&testReportShard=7&testReportAttempt=1#targets-Ly9wa2cvY2NsL3NxbHByb3h5Y2NsOnNxbHByb3h5Y2NsX3Rlc3Q=) on release-24.2 @ [b488fe3bd9d58ffa566bd6dec8d68f56672fe95a](https://github.com/cockroachdb/cockroach/commits/b488fe3bd9d58ffa566bd6dec8d68f56672fe95a):

Fatal error:

```
panic: use of Span after Finish. Span: proxy-con-serve. Finish previously called at: <stack not captured. Set debugUseAfterFinish> [recovered]
	panic: use of Span after Finish. Span: proxy-con-serve. Finish previously called at: <stack not captured. Set debugUseAfterFinish>

```
Stack: 

```
goroutine 672171 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x19c260a?, {0x8545cf8, 0xc00e7476b0})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:226 +0x65
panic({0x64f6720?, 0xc01234ca40?})
	GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).detectUseAfterFinish(0xcc91c40?)
	github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:177 +0x114
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).RecordingType(0xc010456808)
	github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:437 +0x18
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).IsVerbose(...)
	github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:445
github.com/cockroachdb/cockroach/pkg/util/log.getSpan({0x8545ba0?, 0xc0127d2060?})
	github.com/cockroachdb/cockroach/pkg/util/log/trace.go:22 +0x34
github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x8545ba0, 0xc0127d2060}, 0x2, 0x1, 0x0, 0x20?, {0x6c3156c, 0x19}, {0xc0109b5640, 0x1, ...})
	github.com/cockroachdb/cockroach/pkg/util/log/channels.go:99 +0x4b4
github.com/cockroachdb/cockroach/pkg/util/log.logfDepth(...)
	github.com/cockroachdb/cockroach/pkg/util/log/channels.go:34
github.com/cockroachdb/cockroach/pkg/util/log.Infof(...)
	github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/util/log/log_channels_generated.go:277
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenant.(*tenantEntry).RefreshPods(0xc00d7a17a0, {0x8545ba0, 0xc0127d2060}, {0x856ba80, 0xc0109dad20})
	github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenant/pkg/ccl/sqlproxyccl/tenant/entry.go:127 +0x165
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenant.(*directoryCache).ReportFailure(0xc0091483c0, {0x8545ba0, 0xc0127d2060}, {0xc01248d540?}, {0x2?, 0x0?})
	github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenant/pkg/ccl/sqlproxyccl/tenant/directory_cache.go:289 +0x4a
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl.init.func7({0x8545ba0?, 0xc0127d2060?}, {0x2?}, {0xc00f71a8d0?, 0xc0127d2060?}, {0x85460e8?, 0xc0091483c0?})
	github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/connector.go:440 +0x3f
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl.(*connector).dialTenantCluster(0xc008c98c00, {0x8545ba0, 0xc0127d2060}, {0x8545a50, 0xc00bfff400})
	github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/connector.go:251 +0x965
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl.(*connector).OpenTenantConnWithToken(0xc008c98c00, {0x8545ba0, 0xc0127d2060}, {0x8545a50, 0xc00bfff400}, {0xc00dfa6140, 0x98})
	github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/connector.go:106 +0x12a
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl.transferConnection(0xc0127d2060, {0x8545a50, 0xc00bfff400}, 0xc008c98c00, 0xc00ba15560, 0xc0065da438, 0xc01011ee10)
	github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/conn_migration.go:274 +0x2d3
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl.(*forwarder).TransferConnection(0xc00bfff400)
	github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/conn_migration.go:213 +0x2f6
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/balancer.(*Balancer).processQueue.func1.1({0x8545cf8, 0xc00e7476b0})
	github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/balancer/balancer.go:371 +0x13f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:480 +0x13a
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 641556
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:471 +0x3fe
```
<details><summary>Log preceding fatal error</summary>


```
=== RUN TestCancelQuery/forward_over_http
=== RUN TestCancelQuery/ignore_unknown_secret_key
*
* ERROR: a panic has occurred!
* use of Span after Finish. Span: proxy-con-serve. Finish previously called at: <stack not captured. Set debugUseAfterFinish>
* (1) attached stack trace
* -- stack trace:
* | runtime.gopanic
* | 	GOROOT/src/runtime/panic.go:770
* | [...repeated from below...]
* Wraps: (2) assertion failure
* Wraps: (3) attached stack trace
* -- stack trace:
* | github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).detectUseAfterFinish
* | 	github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:177
* | github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).RecordingType
* | 	github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:437
* | github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).IsVerbose
* | 	github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:445
* | github.com/cockroachdb/cockroach/pkg/util/log.getSpan
* | 	github.com/cockroachdb/cockroach/pkg/util/log/trace.go:22
* | github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal
* | 	github.com/cockroachdb/cockroach/pkg/util/log/channels.go:99
* | github.com/cockroachdb/cockroach/pkg/util/log.logfDepth
* | 	github.com/cockroachdb/cockroach/pkg/util/log/channels.go:34
* | github.com/cockroachdb/cockroach/pkg/util/log.Infof
* | 	github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/util/log/log_channels_generated.go:277
* | github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenant.(*tenantEntry).RefreshPods
* | 	github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenant/pkg/ccl/sqlproxyccl/tenant/entry.go:127
* | github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenant.(*directoryCache).ReportFailure
* | 	github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenant/pkg/ccl/sqlproxyccl/tenant/directory_cache.go:289
* | github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl.init.func7
* | 	github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/connector.go:440
* | github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl.(*connector).dialTenantCluster
* | 	github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/connector.go:251
* | github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl.(*connector).OpenTenantConnWithToken
* | 	github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/connector.go:106
* | github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl.transferConnection
* | 	github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/conn_migration.go:274
* | github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl.(*forwarder).TransferConnection
* | 	github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/conn_migration.go:213
* | github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/balancer.(*Balancer).processQueue.func1.1
* | 	github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/balancer/balancer.go:371
* | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
* | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:480
* | runtime.goexit
* | 	src/runtime/asm_amd64.s:1695
* Wraps: (4) use of Span after Finish. Span: proxy-con-serve. Finish previously called at: <stack not captured. Set debugUseAfterFinish>
* Error types: (1) *withstack.withStack (2) *assert.withAssertionFailure (3) *withstack.withStack (4) *errutil.leafError
*
```

</details>

Parameters:
 - <code>attempt=1</code>
 - <code>deadlock=true</code>
 - <code>run=2</code>
 - <code>shard=7</code>
<details><summary>Help</summary>


See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

</details>
<details><summary>Same failure on other branches</summary>


- #131008 ccl/sqlproxyccl: TestCancelQuery failed [C-test-failure O-robot T-cloud-platform branch-release-24.1.5-rc release-blocker]

</details>
/cc @cockroachdb/sql-foundations @cockroachdb/server


[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*TestCancelQuery.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)




Jira issue: CRDB-44642

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ccl/sqlproxyccl: TestCancelQuery failed #135551

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

ccl/sqlproxyccl: TestCancelQuery failed #135551

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions