Skip to content

ccl/sqlproxyccl: TestCancelQuery failed #148599

Closed
Closed
ccl/sqlproxyccl: TestCancelQuery failed#148599
Labels
C-test-failureBroken test (automatically or manually discovered).O-robotOriginated from a bot.T-cloud-platformbranch-release-24.1Used to mark GA and release blockers, technical advisories, and bugs for 24.1
Milestone
24.1
@cockroach-teamcity

Description

@cockroach-teamcity
cockroach-teamcity
opened on Jun 20, 2025

ccl/sqlproxyccl.TestCancelQuery failed on release-24.1 @ 7b38b443e94782ed6e1e5e4556d11b48de35f1e9:

Fatal error:

panic: use of Span after Finish. Span: proxy-con-serve. Finish previously called at: <stack not captured. Set debugUseAfterFinish> [recovered]
	panic: use of Span after Finish. Span: proxy-con-serve. Finish previously called at: <stack not captured. Set debugUseAfterFinish>

Stack:

goroutine 4632 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xc00956ad30?, {0xe0c5818, 0xc007772c60})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:226 +0x74
panic({0x9cef240?, 0xc00c908670?})
	GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).detectUseAfterFinish(0xc0061c5808)
	github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:177 +0x274
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).RecordingType(0xc0061c5808)
	github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:437 +0x27
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).IsVerbose(...)
	github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:445
github.com/cockroachdb/cockroach/pkg/util/log.getSpan({0xe0c56c0, 0xc00a2adc40})
	github.com/cockroachdb/cockroach/pkg/util/log/trace.go:22 +0x45
github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0xe0c56c0, 0xc00a2adc40}, 0x2, 0x3, 0x1, 0x0, {0xa47e4c3, 0x27}, {0xc00956b760, 0x2, ...})
	github.com/cockroachdb/cockroach/pkg/util/log/channels.go:99 +0x66b
github.com/cockroachdb/cockroach/pkg/util/log.logfDepth(...)
	github.com/cockroachdb/cockroach/pkg/util/log/channels.go:34
github.com/cockroachdb/cockroach/pkg/util/log.loggerOps.Errorf(...)
	github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/util/log/log_channels_generated.go:1281
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl.(*connector).dialTenantCluster(0xc002b82300, {0xe0c56c0, 0xc00a2adc40}, {0xe0c5570, 0xc00349dea0})
	github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/connector.go:243 +0xcd8
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl.(*connector).OpenTenantConnWithToken(0xc002b82300, {0xe0c56c0, 0xc00a2adc40}, {0xe0c5570, 0xc00349dea0}, {0xc003c406e0, 0x98})
	github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/connector.go:106 +0x1ba
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl.transferConnection(0xc00a2adc40, {0xe0c5570, 0xc00349dea0}, 0xc002b82300, 0xc0043a2000, 0xc00bb09788, 0xc009b65578)
	github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/conn_migration.go:274 +0x3ed
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl.(*forwarder).TransferConnection(0xc00349dea0)
	github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/conn_migration.go:213 +0x509
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/balancer.(*Balancer).processQueue.func1.1({0xe0c5818, 0xc007772c60})
	github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/balancer/balancer.go:371 +0x268
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:480 +0x263
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 4499
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:471 +0x69d
Log preceding fatal error 

=== RUN   TestCancelQuery
    test_log_scope.go:165: test logs captured to: outputs.zip/logTestCancelQuery3139039373
    test_log_scope.go:76: use -show-logs to present logs inline
=== RUN   TestCancelQuery/cancel_after_migrating_a_session
=== RUN   TestCancelQuery/cancel_over_http
=== RUN   TestCancelQuery/cancel_over_sql
=== RUN   TestCancelQuery/forward_over_http
=== RUN   TestCancelQuery/ignore_unknown_secret_key
*
* ERROR: a panic has occurred!
* use of Span after Finish. Span: proxy-con-serve. Finish previously called at: <stack not captured. Set debugUseAfterFinish>
* (1) attached stack trace
*   -- stack trace:
*   | runtime.gopanic
*   | 	GOROOT/src/runtime/panic.go:770
*   | [...repeated from below...]
* Wraps: (2) assertion failure
* Wraps: (3) attached stack trace
*   -- stack trace:
*   | github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).detectUseAfterFinish
*   | 	github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:177
*   | github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).RecordingType
*   | 	github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:437
*   | github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).IsVerbose
*   | 	github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:445
*   | github.com/cockroachdb/cockroach/pkg/util/log.getSpan
*   | 	github.com/cockroachdb/cockroach/pkg/util/log/trace.go:22
*   | github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal
*   | 	github.com/cockroachdb/cockroach/pkg/util/log/channels.go:99
*   | github.com/cockroachdb/cockroach/pkg/util/log.logfDepth
*   | 	github.com/cockroachdb/cockroach/pkg/util/log/channels.go:34
*   | github.com/cockroachdb/cockroach/pkg/util/log.loggerOps.Errorf
*   | 	github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/util/log/log_channels_generated.go:1281
*   | github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl.(*connector).dialTenantCluster
*   | 	github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/connector.go:243
*   | github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl.(*connector).OpenTenantConnWithToken
*   | 	github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/connector.go:106
*   | github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl.transferConnection
*   | 	github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/conn_migration.go:274
*   | github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl.(*forwarder).TransferConnection
*   | 	github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/conn_migration.go:213
*   | github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/balancer.(*Balancer).processQueue.func1.1
*   | 	github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/balancer/balancer.go:371
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
*   | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:480
*   | runtime.goexit
*   | 	src/runtime/asm_amd64.s:1695
* Wraps: (4) use of Span after Finish. Span: proxy-con-serve. Finish previously called at: <stack not captured. Set debugUseAfterFinish>
* Error types: (1) *withstack.withStack (2) *assert.withAssertionFailure (3) *withstack.withStack (4) *errutil.leafError
*

Parameters:

  • attempt=1
  • race=true
  • run=2
  • shard=7
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

/cc @cockroachdb/sql-foundations @cockroachdb/server

This test on roachdash | Improve this report!

Jira issue: CRDB-51695

Metadata

Metadata

Assignees

No one assigned

    Labels

    C-test-failureBroken test (automatically or manually discovered).O-robotOriginated from a bot.T-cloud-platformbranch-release-24.1Used to mark GA and release blockers, technical advisories, and bugs for 24.1

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions