-
Notifications
You must be signed in to change notification settings - Fork 4.1k
ccl/serverccl: TestServiceShutdownUsesGracefulDrain failed #107856
Copy link
Copy link
Labels
C-test-failureBroken test (automatically or manually discovered).Broken test (automatically or manually discovered).O-robotOriginated from a bot.Originated from a bot.T-multitenantIssues owned by the multi-tenant virtual teamIssues owned by the multi-tenant virtual teambranch-masterFailures and bugs on the master branch.Failures and bugs on the master branch.db-cy-23
Milestone
Description
ccl/serverccl.TestServiceShutdownUsesGracefulDrain failed with artifacts on master @ f295bd861a3a427652b19c2254d2401ebb4a3c8e:
F230730 06:57:00.234543 28283619 server/server_sql.go:1682 [T2,n1] 1 !goroutine 27787625 [sync.Cond.Wait]:
F230730 06:57:00.234543 28283619 server/server_sql.go:1682 [T2,n1] 1 !sync.runtime_notifyListWait(0xc013189690, 0x20e)
F230730 06:57:00.234543 28283619 server/server_sql.go:1682 [T2,n1] 1 ! GOROOT/src/runtime/sema.go:517 +0x14c
F230730 06:57:00.234543 28283619 server/server_sql.go:1682 [T2,n1] 1 !sync.(*Cond).Wait(0x58c8f60?)
F230730 06:57:00.234543 28283619 server/server_sql.go:1682 [T2,n1] 1 ! GOROOT/src/sync/cond.go:70 +0x8c
F230730 06:57:00.234543 28283619 server/server_sql.go:1682 [T2,n1] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker(0xc0029f58f0, {0x79c1540, 0xc021bb24b0}, {0x79a4c20, 0xc003768a80}, 0xc003870a80)
F230730 06:57:00.234543 28283619 server/server_sql.go:1682 [T2,n1] 1 ! github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:378 +0x318
F230730 06:57:00.234543 28283619 server/server_sql.go:1682 [T2,n1] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2({0x79c1540?, 0xc021bb24b0?})
F230730 06:57:00.234543 28283619 server/server_sql.go:1682 [T2,n1] 1 ! github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321 +0x45
F230730 06:57:00.234543 28283619 server/server_sql.go:1682 [T2,n1] 1 !github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
F230730 06:57:00.234543 28283619 server/server_sql.go:1682 [T2,n1] 1 ! github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x146
F230730 06:57:00.234543 28283619 server/server_sql.go:1682 [T2,n1] 1 !created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
F230730 06:57:00.234543 28283619 server/server_sql.go:1682 [T2,n1] 1 ! github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x43b
F230730 06:57:00.234543 28283619 server/server_sql.go:1682 [T2,n1] 1 !
F230730 06:57:00.234543 28283619 server/server_sql.go:1682 [T2,n1] 1 !goroutine 27787367 [sync.Cond.Wait]:
F230730 06:57:00.234543 28283619 server/server_sql.go:1682 [T2,n1] 1 !sync.runtime_notifyListWait(0xc013189610, 0x356)
F230730 06:57:00.234543 28283619 server/server_sql.go:1682 [T2,n1] 1 ! GOROOT/src/runtime/sema.go:517 +0x14c
F230730 06:57:00.234543 28283619 server/server_sql.go:1682 [T2,n1] 1 !sync.(*Cond).Wait(0x58c8f60?)
F230730 06:57:00.234543 28283619 server/server_sql.go:1682 [T2,n1] 1 ! GOROOT/src/sync/cond.go:70 +0x8c
F230730 06:57:00.234543 28283619 server/server_sql.go:1682 [T2,n1] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker(0xc0029f56c0, {0x79c1540, 0xc02de2fad0}, {0x79a4c20, 0xc003768a80}, 0xc003870a80)
F230730 06:57:00.234543 28283619 server/server_sql.go:1682 [T2,n1] 1 ! github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:378 +0x318
F230730 06:57:00.234543 28283619 server/server_sql.go:1682 [T2,n1] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2({0x79c1540?, 0xc02de2fad0?})
F230730 06:57:00.234543 28283619 server/server_sql.go:1682 [T2,n1] 1 ! github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321 +0x45
F230730 06:57:00.234543 28283619 server/server_sql.go:1682 [T2,n1] 1 !github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
F230730 06:57:00.234543 28283619 server/server_sql.go:1682 [T2,n1] 1 ! github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x146
F230730 06:57:00.234543 28283619 server/server_sql.go:1682 [T2,n1] 1 !created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
F230730 06:57:00.234543 28283619 server/server_sql.go:1682 [T2,n1] 1 ! github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x43b
F230730 06:57:00.234543 28283619 server/server_sql.go:1682 [T2,n1] 1 !
F230730 06:57:00.234543 28283619 server/server_sql.go:1682 [T2,n1] 1 !For more context, check log files in: /artifacts/tmp/_tmp/c101b7a464a1afc1f5af0cd85792187e/logTestServiceShutdownUsesGracefulDrain2336992875
F230730 06:57:00.234543 28283619 server/server_sql.go:1682 [T2,n1] 1 !
F230730 06:57:00.234543 28283619 server/server_sql.go:1682 [T2,n1] 1 !
F230730 06:57:00.234543 28283619 server/server_sql.go:1682 [T2,n1] 1 !****************************************************************************
F230730 06:57:00.234543 28283619 server/server_sql.go:1682 [T2,n1] 1 !
F230730 06:57:00.234543 28283619 server/server_sql.go:1682 [T2,n1] 1 !This node experienced a fatal error (printed above), and as a result the
F230730 06:57:00.234543 28283619 server/server_sql.go:1682 [T2,n1] 1 !process is terminating.
F230730 06:57:00.234543 28283619 server/server_sql.go:1682 [T2,n1] 1 !
F230730 06:57:00.234543 28283619 server/server_sql.go:1682 [T2,n1] 1 !Fatal errors can occur due to faulty hardware (disks, memory, clocks) or a
F230730 06:57:00.234543 28283619 server/server_sql.go:1682 [T2,n1] 1 !problem in CockroachDB. With your help, the support team at Cockroach Labs
F230730 06:57:00.234543 28283619 server/server_sql.go:1682 [T2,n1] 1 !will try to determine the root cause, recommend next steps, and we can
F230730 06:57:00.234543 28283619 server/server_sql.go:1682 [T2,n1] 1 !improve CockroachDB based on your report.
F230730 06:57:00.234543 28283619 server/server_sql.go:1682 [T2,n1] 1 !
F230730 06:57:00.234543 28283619 server/server_sql.go:1682 [T2,n1] 1 !Please submit a crash report by following the instructions here:
F230730 06:57:00.234543 28283619 server/server_sql.go:1682 [T2,n1] 1 !
F230730 06:57:00.234543 28283619 server/server_sql.go:1682 [T2,n1] 1 ! https://github.com/cockroachdb/cockroach/issues/new/choose
F230730 06:57:00.234543 28283619 server/server_sql.go:1682 [T2,n1] 1 !
F230730 06:57:00.234543 28283619 server/server_sql.go:1682 [T2,n1] 1 !If you would rather not post publicly, please contact us directly at:
F230730 06:57:00.234543 28283619 server/server_sql.go:1682 [T2,n1] 1 !
F230730 06:57:00.234543 28283619 server/server_sql.go:1682 [T2,n1] 1 ! support@cockroachlabs.com
F230730 06:57:00.234543 28283619 server/server_sql.go:1682 [T2,n1] 1 !
F230730 06:57:00.234543 28283619 server/server_sql.go:1682 [T2,n1] 1 !The Cockroach Labs team appreciates your feedback.
Parameters: TAGS=bazel,gss,deadlock
, stress=true
This test on roachdash | Improve this report!
Jira issue: CRDB-30222
Epic: CRDB-26691
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
C-test-failureBroken test (automatically or manually discovered).Broken test (automatically or manually discovered).O-robotOriginated from a bot.Originated from a bot.T-multitenantIssues owned by the multi-tenant virtual teamIssues owned by the multi-tenant virtual teambranch-masterFailures and bugs on the master branch.Failures and bugs on the master branch.db-cy-23