kv/kvnemesis: TestKVNemesisMultiNode failed #108241
Description
kv/kvnemesis.TestKVNemesisMultiNode failed with artifacts on master @ 695d4ec85157a4da0c97237e6f16bf919eebf6c5:
Fatal error:
panic: use of Span after Finish. Span: /cockroach.roachpb.Internal/Batch. Finish previously called at: <stack not captured. Set debugUseAfterFinish> [recovered]
panic: use of Span after Finish. Span: /cockroach.roachpb.Internal/Batch. Finish previously called at: <stack not captured. Set debugUseAfterFinish>
Stack:
goroutine 6569871 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x1f81c6a?, {0x6976bc8, 0xc014ecf170})
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:230 +0x6a
panic({0x4b2d840, 0xc01b174160})
GOROOT/src/runtime/panic.go:884 +0x212
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).detectUseAfterFinish(0x6976bc8?)
github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:182 +0x12e
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).Tracer(...)
github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:225
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaDecoder).createTracingSpans(0xc00a3270d8, {0x6976bc8, 0xc0204de0f0})
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_decoder.go:154 +0x4f0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaDecoder).DecodeAndBind(0xc00a326c80?, {0x6976bc8, 0xc0204de0f0}, {0xc02851e210?, 0x3?, 0x10?})
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_decoder.go:63 +0x66
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).Decode(...)
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:142
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:878 +0x73a
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:728 +0x1b0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady(0xc0165eea80, 0x2?)
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:689 +0x14b
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker(0xc0085cbd50, {0x6976bc8, 0xc014ecf170}, {0x695c980, 0xc0165eea80}, 0xc0165ee000)
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:416 +0x274
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2({0x6976bc8?, 0xc014ecf170?})
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321 +0x45
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x146
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x43b
Log preceding fatal error
=== RUN TestKVNemesisMultiNode
test_log_scope.go:167: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode1784560352
test_log_scope.go:81: use -show-logs to present logs inline
kvnemesis_test.go:283: seed: 2381536854787156731
test_server_shim.go:98:
Test server was configured to route SQL queries to a secondary tenant (virtual cluster).
If you are only seeing a test failure when this message appears, there may be a problem
specific to cluster virtualization or multi-tenancy.
To investigate, consider using "COCKROACH_TEST_TENANT=true" to force-enable just
the secondary tenant in all runs (or, alternatively, "false" to force-disable), or use
"COCKROACH_INTERNAL_DISABLE_METAMORPHIC_TESTING=false" to disable all random test variables altogether.
*
* WARNING: test tenant requested by configuration, but code organization prevents start!
* OSS binaries do not include enterprise features
*
*
* WARNING: test tenant requested by configuration, but code organization prevents start!
* OSS binaries do not include enterprise features
*
*
* WARNING: test tenant requested by configuration, but code organization prevents start!
* OSS binaries do not include enterprise features
*
*
* WARNING: test tenant requested by configuration, but code organization prevents start!
* OSS binaries do not include enterprise features
*
kvnemesis_test.go:197: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis1734753383
Parameters: TAGS=bazel,gss,deadlock
, stress=true
This test on roachdash | Improve this report!
Jira issue: CRDB-30377