Skip to content

ccl/sqlproxyccl: TestCancelQuery failed #131008

Closed
Closed
ccl/sqlproxyccl: TestCancelQuery failed#131008
Labels
C-test-failureBroken test (automatically or manually discovered).O-robotOriginated from a bot.T-cloud-platformbranch-release-24.1.5-rcrelease-blockerIndicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked.
Milestone
24.1
@cockroach-teamcity

Description

@cockroach-teamcity
cockroach-teamcity
opened on Sep 19, 2024

ccl/sqlproxyccl.TestCancelQuery failed on release-24.1.5-rc @ a2667b3be966a6b1ae79303f5b2db52a9dcb9132:

Fatal error:

panic: use of Span after Finish. Span: proxy-con-serve. Finish previously called at: <stack not captured. Set debugUseAfterFinish> [recovered]
	panic: use of Span after Finish. Span: proxy-con-serve. Finish previously called at: <stack not captured. Set debugUseAfterFinish>

Stack:

goroutine 4843 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xc00b414d30?, {0xe053738, 0xc0048ca6f0})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:231 +0x74
panic({0x9cab420?, 0xc0092680b0?})
	GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).detectUseAfterFinish(0xc005cc9008)
	github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:182 +0x274
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).RecordingType(0xc005cc9008)
	github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:442 +0x27
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).IsVerbose(...)
	github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:450
github.com/cockroachdb/cockroach/pkg/util/log.getSpan({0xe0535e0, 0xc005c28840})
	github.com/cockroachdb/cockroach/pkg/util/log/trace.go:27 +0x45
github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0xe0535e0, 0xc005c28840}, 0x2, 0x3, 0x1, 0x0, {0xa437439, 0x27}, {0xc00b415760, 0x2, ...})
	github.com/cockroachdb/cockroach/pkg/util/log/channels.go:104 +0x66b
github.com/cockroachdb/cockroach/pkg/util/log.logfDepth(...)
	github.com/cockroachdb/cockroach/pkg/util/log/channels.go:39
github.com/cockroachdb/cockroach/pkg/util/log.loggerOps.Errorf(...)
	github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/util/log/log_channels_generated.go:1281
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl.(*connector).dialTenantCluster(0xc0067f4580, {0xe0535e0, 0xc005c28840}, {0xe053490, 0xc00a5c4d20})
	github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/connector.go:246 +0xcd8
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl.(*connector).OpenTenantConnWithToken(0xc0067f4580, {0xe0535e0, 0xc005c28840}, {0xe053490, 0xc00a5c4d20}, {0xc0000c7040, 0x9c})
	github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/connector.go:109 +0x1ba
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl.transferConnection(0xc005c28840, {0xe053490, 0xc00a5c4d20}, 0xc0067f4580, 0xc00a111590, 0xc006b99c08, 0xc0058b5f38)
	github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/conn_migration.go:277 +0x3ed
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl.(*forwarder).TransferConnection(0xc00a5c4d20)
	github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/conn_migration.go:216 +0x509
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/balancer.(*Balancer).processQueue.func1.1({0xe053738, 0xc0048ca6f0})
	github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/balancer/balancer.go:374 +0x268
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x263
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 4681
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x69d
Log preceding fatal error 

=== RUN   TestCancelQuery
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestCancelQuery3813524590
    test_log_scope.go:81: use -show-logs to present logs inline
=== RUN   TestCancelQuery/cancel_after_migrating_a_session
=== RUN   TestCancelQuery/cancel_over_http
=== RUN   TestCancelQuery/cancel_over_sql
=== RUN   TestCancelQuery/forward_over_http
=== RUN   TestCancelQuery/ignore_unknown_secret_key
*
* ERROR: a panic has occurred!
* use of Span after Finish. Span: proxy-con-serve. Finish previously called at: <stack not captured. Set debugUseAfterFinish>
* (1) attached stack trace
*   -- stack trace:
*   | runtime.gopanic
*   | 	GOROOT/src/runtime/panic.go:770
*   | [...repeated from below...]
* Wraps: (2) assertion failure
* Wraps: (3) attached stack trace
*   -- stack trace:
*   | github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).detectUseAfterFinish
*   | 	github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:182
*   | github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).RecordingType
*   | 	github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:442
*   | github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).IsVerbose
*   | 	github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:450
*   | github.com/cockroachdb/cockroach/pkg/util/log.getSpan
*   | 	github.com/cockroachdb/cockroach/pkg/util/log/trace.go:27
*   | github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal
*   | 	github.com/cockroachdb/cockroach/pkg/util/log/channels.go:104
*   | github.com/cockroachdb/cockroach/pkg/util/log.logfDepth
*   | 	github.com/cockroachdb/cockroach/pkg/util/log/channels.go:39
*   | github.com/cockroachdb/cockroach/pkg/util/log.loggerOps.Errorf
*   | 	github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/util/log/log_channels_generated.go:1281
*   | github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl.(*connector).dialTenantCluster
*   | 	github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/connector.go:246
*   | github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl.(*connector).OpenTenantConnWithToken
*   | 	github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/connector.go:109
*   | github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl.transferConnection
*   | 	github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/conn_migration.go:277
*   | github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl.(*forwarder).TransferConnection
*   | 	github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/conn_migration.go:216
*   | github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/balancer.(*Balancer).processQueue.func1.1
*   | 	github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/balancer/balancer.go:374
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
*   | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485
*   | runtime.goexit
*   | 	src/runtime/asm_amd64.s:1695
* Wraps: (4) use of Span after Finish. Span: proxy-con-serve. Finish previously called at: <stack not captured. Set debugUseAfterFinish>
* Error types: (1) *withstack.withStack (2) *assert.withAssertionFailure (3) *withstack.withStack (4) *errutil.leafError
*

Parameters:

  • attempt=1
  • race=true
  • run=2
  • shard=7
Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/sql-foundations @cockroachdb/server

This test on roachdash | Improve this report!

Jira issue: CRDB-42335

Metadata

Metadata

Assignees

No one assigned

    Labels

    C-test-failureBroken test (automatically or manually discovered).O-robotOriginated from a bot.T-cloud-platformbranch-release-24.1.5-rcrelease-blockerIndicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked.

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions