Skip to content

*: tests flaking with "not enough time left on migration lease, terminating for safety" #21444

Closed
Closed
*: tests flaking with "not enough time left on migration lease, terminating for safety"#21444
Assignees
andreimatei
Labels
C-test-failureBroken test (automatically or manually discovered).O-robotOriginated from a bot.
Milestone
2.0
@cockroach-teamcity

Description

@cockroach-teamcity
cockroach-teamcity
opened on Jan 14, 2018

SHA: https://github.com/cockroachdb/cockroach/commits/3cd0224615394f7606be514ad42419c538e7bd7b

Parameters:

TAGS=
GOFLAGS=-race

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=478328&tab=buildLog


I180114 09:10:59.204680 7101 server/node.go:433  [n1] node=1: started with [<no-attributes>=<in-mem>] engine(s) and attributes []
I180114 09:10:59.205986 7101 sql/distsql_physical_planner.go:122  [n1] creating DistSQLPlanner with address {tcp 127.0.0.1:38367}
I180114 09:10:59.242933 7101 server/server.go:1164  [n1] starting https server at 127.0.0.1:37309
I180114 09:10:59.250728 7101 server/server.go:1165  [n1] starting grpc/postgres server at 127.0.0.1:38367
I180114 09:10:59.250922 7101 server/server.go:1166  [n1] advertising CockroachDB node at 127.0.0.1:38367
W180114 09:10:59.251407 7101 sql/jobs/registry.go:222  [n1] unable to get node liveness: node not in the liveness table
I180114 09:10:59.278089 19931 storage/replica_command.go:819  [split,n1,s1,r1/1:/M{in-ax}] initiating a split of this range at key /System/"" [r2]
I180114 09:10:59.864511 19931 storage/replica_command.go:819  [split,n1,s1,r2/1:/{System/-Max}] initiating a split of this range at key /System/NodeLiveness [r3]
I180114 09:11:00.041829 7101 sql/event_log.go:115  [n1] Event: "alter_table", target: 12, info: {TableName:eventlog Statement:ALTER TABLE system.eventlog ALTER COLUMN "uniqueID" SET DEFAULT uuid_v4() User:node MutationID:0 CascadeDroppedViews:[]}
I180114 09:11:00.193370 19931 storage/replica_command.go:819  [split,n1,s1,r3/1:/{System/NodeL…-Max}] initiating a split of this range at key /System/NodeLivenessMax [r4]
I180114 09:11:00.215689 7101 sql/lease.go:348  [n1] publish: descID=12 (eventlog) version=2 mtime=2018-01-14 09:11:00.213470356 +0000 UTC
I180114 09:11:00.375225 19931 storage/replica_command.go:819  [split,n1,s1,r4/1:/{System/NodeL…-Max}] initiating a split of this range at key /System/tsd [r5]
I180114 09:11:00.668523 19931 storage/replica_command.go:819  [split,n1,s1,r5/1:/{System/tsd-Max}] initiating a split of this range at key /System/"tse" [r6]
W180114 09:11:00.721658 21638 storage/intent_resolver.go:324  [n1,s1] failed to push during intent resolution: failed to push "sql txn implicit" id=991a93c3 key=/Table/SystemConfigSpan/Start rw=true pri=0.07748250 iso=SERIALIZABLE stat=PENDING epo=0 ts=1515921060.555862335,0 orig=1515921060.555862335,0 max=1515921060.555862335,0 wto=false rop=false seq=7
I180114 09:11:00.756464 7101 sql/event_log.go:115  [n1] Event: "set_cluster_setting", target: 0, info: {SettingName:diagnostics.reporting.enabled Value:true User:node}
W180114 09:11:00.785470 21179 storage/intent_resolver.go:324  [n1,s1] failed to push during intent resolution: failed to push "split" id=32b432d2 key=/Local/Range/System/tsd/RangeDescriptor rw=true pri=0.02501528 iso=SERIALIZABLE stat=PENDING epo=0 ts=1515921060.668708973,0 orig=1515921060.668708973,0 max=1515921060.668708973,0 wto=false rop=false seq=2
I180114 09:11:00.900855 19931 storage/replica_command.go:819  [split,n1,s1,r6/1:/{System/tse-Max}] initiating a split of this range at key /Table/SystemConfigSpan/Start [r7]
I180114 09:11:01.090602 19931 storage/replica_command.go:819  [split,n1,s1,r7/1:/{Table/System…-Max}] initiating a split of this range at key /Table/11 [r8]
I180114 09:11:01.320250 19931 storage/replica_command.go:819  [split,n1,s1,r8/1:/{Table/11-Max}] initiating a split of this range at key /Table/12 [r9]
I180114 09:11:01.362293 7101 sql/event_log.go:115  [n1] Event: "set_cluster_setting", target: 0, info: {SettingName:version Value:$1 User:node}
W180114 09:11:01.420940 21658 storage/intent_resolver.go:324  [n1,s1] failed to push during intent resolution: failed to push "split" id=84456346 key=/Local/Range/Table/11/RangeDescriptor rw=true pri=0.05185148 iso=SERIALIZABLE stat=PENDING epo=0 ts=1515921061.320475913,0 orig=1515921061.320475913,0 max=1515921061.320475913,0 wto=false rop=false seq=2
I180114 09:11:01.490427 19931 storage/replica_command.go:819  [split,n1,s1,r9/1:/{Table/12-Max}] initiating a split of this range at key /Table/13 [r10]
I180114 09:11:01.608132 7101 sql/event_log.go:115  [n1] Event: "set_cluster_setting", target: 0, info: {SettingName:trace.debug.enable Value:false User:node}
I180114 09:11:01.788154 19931 storage/replica_command.go:819  [split,n1,s1,r10/1:/{Table/13-Max}] initiating a split of this range at key /Table/14 [r11]
W180114 09:11:01.869457 21722 storage/intent_resolver.go:324  [n1,s1] failed to push during intent resolution: failed to push "split" id=2623bb5a key=/Local/Range/Table/13/RangeDescriptor rw=true pri=0.05554231 iso=SERIALIZABLE stat=PENDING epo=0 ts=1515921061.788492247,0 orig=1515921061.788492247,0 max=1515921061.788492247,0 wto=false rop=false seq=3
I180114 09:11:01.933590 19931 storage/replica_command.go:819  [split,n1,s1,r11/1:/{Table/14-Max}] initiating a split of this range at key /Table/15 [r12]
W180114 09:12:06.754764 21724 server/server.go:990  The server appears to be unable to contact the other nodes in the cluster. Please try

- starting the other nodes, if you haven't already
- double-checking that the '--join' and '--host' flags are set up correctly
- not using the '--background' flag.

If problems persist, please see https://www.cockroachlabs.com/docs/dev/cluster-setup-troubleshooting.html.
I180114 09:12:14.263542 21587 gossip/gossip.go:487  [n1] gossip status (ok, 1 node)
gossip client (0/3 cur/max conns)
gossip server (0/3 cur/max conns, infos 0/0 sent/received, bytes 0B/0B sent/received)
W180114 09:12:43.987880 21612 storage/node_liveness.go:426  [n1,hb] slow heartbeat took 93.6s
W180114 09:12:50.376704 21692 kv/txn_coord_sender.go:786  [split,s1,r11/1:/{Table/14-Max},n1] heartbeat to "split" id=c572730a key=/Local/Range/Table/14/RangeDescriptor rw=true pri=0.01850347 iso=SERIALIZABLE stat=PENDING epo=0 ts=1515921061.933860211,0 orig=1515921061.933860211,0 max=1515921062.433860211,0 wto=false rop=false seq=3 failed: context deadline exceeded
W180114 09:12:51.943648 21612 storage/node_liveness.go:365  [n1,hb] failed node liveness heartbeat: context deadline exceeded
I180114 09:13:00.901101 21589 server/status/runtime.go:219  [n1] runtime stats: 783 MiB RSS, 215 goroutines, 16 MiB/57 MiB/94 MiB GO alloc/idle/total, 37 MiB/84 MiB CGO alloc/total, 0.00cgo/sec, 0.00/0.00 %(u/s)time, 0.00 %gc (165x)
W180114 09:13:07.569460 19931 internal/client/txn.go:540  [split,n1,s1,r11/1:/{Table/14-Max}] failure aborting transaction: writing transaction timed out or ran on multiple coordinators; abort caused by: context deadline exceeded
W180114 09:13:14.909506 21565 storage/replica.go:1532  [n1,s1,r7/1:/Table/{SystemCon…-11}] have been waiting 1m0s attempting to acquire lease
W180114 09:13:15.392574 19932 storage/replica.go:1532  [replicate,n1,s1,r6/1:/{System/tse-Table/System…}] have been waiting 1m0s attempting to acquire lease
I180114 09:13:16.494163 21587 gossip/gossip.go:487  [n1] gossip status (stalled, 1 node)
gossip client (0/3 cur/max conns)
gossip server (0/3 cur/max conns, infos 0/0 sent/received, bytes 0B/0B sent/received)
W180114 09:13:27.245259 21564 storage/replica.go:1532  [n1,s1,r1/1:/{Min-System/}] have been waiting 1m0s attempting to acquire lease
W180114 09:13:27.245259 21566 storage/replica.go:1532  [n1,s1,r3/1:/System/NodeLiveness{-Max}] have been waiting 1m0s attempting to acquire lease
E180114 09:13:37.826736 19931 storage/queue.go:668  [split,n1,s1,r11/1:/{Table/14-Max}] unable to split [n1,s1,r11/1:/{Table/14-Max}] at key "/Table/15": split at key /Table/15 failed: context deadline exceeded
W180114 09:13:50.843243 21612 storage/node_liveness.go:426  [n1,hb] slow heartbeat took 53.2s
W180114 09:13:52.855733 21612 storage/node_liveness.go:365  [n1,hb] failed node liveness heartbeat: context deadline exceeded
I180114 09:14:02.399390 21587 gossip/gossip.go:487  [n1] gossip status (stalled, 1 node)
gossip client (0/3 cur/max conns)
gossip server (0/3 cur/max conns, infos 0/0 sent/received, bytes 0B/0B sent/received)
I180114 09:14:06.727621 21589 server/status/runtime.go:219  [n1] runtime stats: 783 MiB RSS, 216 goroutines, 20 MiB/54 MiB/94 MiB GO alloc/idle/total, 37 MiB/84 MiB CGO alloc/total, 0.30cgo/sec, 0.29/0.04 %(u/s)time, 0.04 %gc (1x)
W180114 09:14:06.949166 21612 storage/node_liveness.go:426  [n1,hb] slow heartbeat took 9.9s
W180114 09:14:13.679962 21612 storage/node_liveness.go:365  [n1,hb] failed node liveness heartbeat: context deadline exceeded
W180114 09:14:26.263723 21612 storage/node_liveness.go:426  [n1,hb] slow heartbeat took 8.9s
W180114 09:14:29.450402 21612 storage/node_liveness.go:365  [n1,hb] failed node liveness heartbeat: context deadline exceeded
I180114 09:14:30.724987 21589 server/status/runtime.go:219  [n1] runtime stats: 783 MiB RSS, 217 goroutines, 15 MiB/58 MiB/94 MiB GO alloc/idle/total, 37 MiB/84 MiB CGO alloc/total, 2.34cgo/sec, 0.01/0.01 %(u/s)time, 0.00 %gc (0x)
I180114 09:14:30.814952 21245 storage/replica_proposal.go:196  [n1,s1,r7/1:/Table/{SystemCon…-11}] new range lease repl=(n1,s1):1 seq=2 start=0.000000000,0 epo=1 pro=1515921121.444694966,0 following repl=(n1,s1):1 seq=1 start=0.000000000,0 exp=1515921068.164603412,0 pro=1515921059.164736054,0
W180114 09:14:39.238894 20143 storage/replica.go:2793  [n1,s1,r3/1:/System/NodeLiveness{-Max}] have been waiting 1m0s for proposing command RequestLease [/System/NodeLiveness,/Min)
W180114 09:14:39.238894 21628 storage/replica.go:2793  [n1,s1,r7/1:/Table/{SystemCon…-11}] have been waiting 1m0s for proposing command RequestLease [/Table/SystemConfigSpan/Start,/Min)
W180114 09:14:44.766951 21495 storage/store.go:3791  [n1,s1] handle raft ready: 66.0s [processed=1]
W180114 09:14:48.907784 21612 storage/node_liveness.go:426  [n1,hb] slow heartbeat took 13.6s
W180114 09:14:49.269652 21672 storage/replica.go:1532  [n1,s1,r11/1:/{Table/14-Max}] have been waiting 1m0s attempting to acquire lease
W180114 09:14:49.269728 21734 storage/replica.go:2793  [n1,s1,r1/1:/{Min-System/}] have been waiting 1m0s for proposing command RequestLease [/Min,/Min)
W180114 09:14:49.273370 21612 storage/node_liveness.go:365  [n1,hb] failed node liveness heartbeat: context deadline exceeded
W180114 09:14:49.486614 21417 storage/replica.go:2805  [n1,s1,r6/1:/{System/tse-Table/System…}] context cancellation after 54.9s of attempting command RequestLease [/System/"tse",/Min)
I180114 09:14:55.923112 21589 server/status/runtime.go:219  [n1] runtime stats: 784 MiB RSS, 217 goroutines, 17 MiB/57 MiB/94 MiB GO alloc/idle/total, 37 MiB/84 MiB CGO alloc/total, 0.98cgo/sec, 0.01/0.01 %(u/s)time, 0.00 %gc (0x)
W180114 09:14:59.904244 21245 storage/store.go:3791  [n1,s1] handle raft ready: 84.3s [processed=1]
W180114 09:14:59.904250 21507 storage/store.go:3791  [n1,s1] handle raft ready: 68.4s [processed=1]
W180114 09:15:14.360409 21612 storage/node_liveness.go:426  [n1,hb] slow heartbeat took 17.0s
W180114 09:15:20.833392 21612 storage/node_liveness.go:365  [n1,hb] failed node liveness heartbeat: context deadline exceeded
W180114 09:15:21.534283 19088 storage/replica.go:1532  [n1,s1,r4/1:/System/{NodeLive…-tsd}] have been waiting 1m0s attempting to acquire lease
I180114 09:15:22.022870 21587 gossip/gossip.go:487  [n1] gossip status (stalled, 1 node)
gossip client (0/3 cur/max conns)
gossip server (0/3 cur/max conns, infos 0/0 sent/received, bytes 0B/0B sent/received)
I180114 09:15:22.206248 21498 storage/replica_proposal.go:196  [n1,s1,r6/1:/{System/tse-Table/System…}] new range lease repl=(n1,s1):1 seq=2 start=0.000000000,0 epo=1 pro=1515921225.397078370,0 following repl=(n1,s1):1 seq=1 start=0.000000000,0 exp=1515921068.164603412,0 pro=1515921059.164736054,0
I180114 09:15:22.206640 21499 storage/replica_proposal.go:196  [n1,s1,r11/1:/{Table/14-Max}] new range lease repl=(n1,s1):1 seq=2 start=0.000000000,0 epo=1 pro=1515921225.397239416,0 following repl=(n1,s1):1 seq=1 start=0.000000000,0 exp=1515921068.164603412,0 pro=1515921059.164736054,0
I180114 09:15:22.282658 21589 server/status/runtime.go:219  [n1] runtime stats: 784 MiB RSS, 216 goroutines, 17 MiB/56 MiB/94 MiB GO alloc/idle/total, 37 MiB/84 MiB CGO alloc/total, 1.63cgo/sec, 0.01/0.01 %(u/s)time, 0.00 %gc (0x)
I180114 09:15:22.358804 21589 server/status/runtime.go:219  [n1] runtime stats: 765 MiB RSS, 218 goroutines, 18 MiB/30 MiB/68 MiB GO alloc/idle/total, 37 MiB/84 MiB CGO alloc/total, 48.36cgo/sec, 0.32/0.16 %(u/s)time, 0.00 %gc (0x)
I180114 09:15:22.360738 21503 storage/replica_proposal.go:196  [n1,s1,r4/1:/System/{NodeLive…-tsd}] new range lease repl=(n1,s1):1 seq=2 start=0.000000000,0 epo=1 pro=1515921256.684103499,0 following repl=(n1,s1):1 seq=1 start=0.000000000,0 exp=1515921068.164603412,0 pro=1515921059.164736054,0
I180114 09:15:22.361370 21508 storage/replica_proposal.go:196  [n1,s1,r5/1:/System/ts{d-e}] new range lease repl=(n1,s1):1 seq=2 start=0.000000000,0 epo=1 pro=1515921271.318173096,0 following repl=(n1,s1):1 seq=1 start=0.000000000,0 exp=1515921068.164603412,0 pro=1515921059.164736054,0
W180114 09:15:22.457378 21499 storage/store.go:3791  [n1,s1] handle raft ready: 56.9s [processed=1]
W180114 09:15:22.460938 21503 storage/store.go:3791  [n1,s1] handle raft ready: 0.9s [processed=1]
W180114 09:15:22.461478 21498 storage/store.go:3791  [n1,s1] handle raft ready: 56.9s [processed=1]
W180114 09:15:22.478681 21508 storage/store.go:3791  [n1,s1] handle raft ready: 0.8s [processed=1]
W180114 09:15:22.552003 21694 storage/node_liveness.go:426  [n1,s1,r7/1:/Table/{SystemCon…-11}] slow heartbeat took 20.1s
I180114 09:15:22.552287 21612 storage/node_liveness.go:362  [n1,hb] heartbeat failed on epoch increment; retrying
E180114 09:15:22.552804 21707 storage/replica_range_lease.go:256  [n1,s1,r11/1:/{Table/14-Max}] heartbeat failed on epoch increment
E180114 09:15:22.555221 21826 storage/replica_range_lease.go:256  [n1,s1,r4/1:/System/{NodeLive…-tsd}] heartbeat failed on epoch increment
I180114 09:15:22.555687 21536 storage/replica_proposal.go:196  [n1,s1,r7/1:/Table/{SystemCon…-11}] new range lease repl=(n1,s1):1 seq=2 start=0.000000000,0 epo=1 pro=1515921298.639229777,0 following repl=(n1,s1):1 seq=2 start=0.000000000,0 epo=1 pro=1515921121.444694966,0
E180114 09:15:22.563154 21680 storage/replica_range_lease.go:256  [n1,s1,r5/1:/System/ts{d-e}] heartbeat failed on epoch increment
I180114 09:15:22.591414 21545 storage/replica_proposal.go:196  [n1,s1,r4/1:/System/{NodeLive…-tsd}] new range lease repl=(n1,s1):1 seq=2 start=0.000000000,0 epo=1 pro=1515921322.556914823,0 following repl=(n1,s1):1 seq=2 start=0.000000000,0 epo=1 pro=1515921256.684103499,0
I180114 09:15:22.610310 21548 storage/replica_proposal.go:196  [n1,s1,r7/1:/Table/{SystemCon…-11}] new range lease repl=(n1,s1):1 seq=2 start=0.000000000,0 epo=1 pro=1515921322.563845401,0 following repl=(n1,s1):1 seq=2 start=0.000000000,0 epo=1 pro=1515921298.639229777,0
I180114 09:15:22.620390 21547 storage/replica_proposal.go:196  [n1,s1,r5/1:/System/ts{d-e}] new range lease repl=(n1,s1):1 seq=2 start=0.000000000,0 epo=1 pro=1515921322.571172594,0 following repl=(n1,s1):1 seq=2 start=0.000000000,0 epo=1 pro=1515921271.318173096,0
I180114 09:15:22.619674 21546 storage/replica_proposal.go:196  [n1,s1,r11/1:/{Table/14-Max}] new range lease repl=(n1,s1):1 seq=2 start=0.000000000,0 epo=1 pro=1515921322.554659498,0 following repl=(n1,s1):1 seq=2 start=0.000000000,0 epo=1 pro=1515921225.397239416,0
F180114 09:15:22.665344 19088 sqlmigrations/migrations.go:342  [n1] not enough time left on migration lease, terminating for safety
goroutine 19088 [running]:
github.com/cockroachdb/cockroach/pkg/util/log.getStacks(0x47f7101, 0x0, 0x0, 0x1b)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:871 +0xdd
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).outputLogEntry(0x3e2b640, 0xc400000004, 0x38e0067, 0x1b, 0x156, 0xc4203f2e10, 0x44)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:757 +0x6c8
github.com/cockroachdb/cockroach/pkg/util/log.addStructured(0x3c07c80, 0xc421822210, 0x4, 0x2, 0x0, 0x0, 0xc42064feb8, 0x1, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/structured.go:152 +0x324
github.com/cockroachdb/cockroach/pkg/util/log.logDepth(0x3c07c80, 0xc421822210, 0x1, 0xc400000004, 0x0, 0x0, 0xc42064feb8, 0x1, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:55 +0x9a
github.com/cockroachdb/cockroach/pkg/util/log.Fatal(0x3c07c80, 0xc421822210, 0xc42064feb8, 0x1, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:158 +0x88
github.com/cockroachdb/cockroach/pkg/sqlmigrations.(*Manager).EnsureMigrations.func2(0x3c07c80, 0xc421822210)
	/go/src/github.com/cockroachdb/cockroach/pkg/sqlmigrations/migrations.go:342 +0x37c
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc420276ab0, 0x3c07c80, 0xc421822210, 0xc422810c30, 0x29, 0x0, 0x0, 0xc421676d80)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:266 +0xf4
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:261 +0x14d

goroutine 1 [chan receive, 4 minutes]:
testing.(*T).Run(0xc4202e6690, 0x2982b9c, 0x1d, 0x2a434b0, 0xc42064fc01)
	/usr/local/go/src/testing/testing.go:790 +0x59b
testing.runTests.func1(0xc4202e6690)
	/usr/local/go/src/testing/testing.go:1004 +0xa8
testing.tRunner(0xc4202e6690, 0xc42064fd50)
	/usr/local/go/src/testing/testing.go:746 +0x16d
testing.runTests(0xc4204e4140, 0x3be06e0, 0xd1, 0xd1, 0xc4201981b0)
	/usr/local/go/src/testing/testing.go:1002 +0x522
testing.(*M).Run(0xc420fc5f20, 0x18)
	/usr/local/go/src/testing/testing.go:921 +0x207
github.com/cockroachdb/cockroach/pkg/sql_test.TestMain(0xc42064ff20)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/main_test.go:36 +0x15e
main.main()
	github.com/cockroachdb/cockroach/pkg/sql/_test/_testmain.go:464 +0x1ce

goroutine 19 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).flushDaemon(0x3e2b640)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1043 +0x8f
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:581 +0x13a

goroutine 20 [syscall, 7 minutes]:
os/signal.signal_recv(0x823b11)
	/usr/local/go/src/runtime/sigqueue.go:131 +0xa6
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:22 +0x30
created by os/signal.init.0
	/usr/local/go/src/os/signal/signal_unix.go:28 +0x4f

goroutine 21 [select, 4 minutes, locked to thread]:
runtime.gopark(0x2a46f00, 0x0, 0x295b7d7, 0x6, 0x18, 0x1)
	/usr/local/go/src/runtime/proc.go:277 +0xfd
runtime.selectgo(0xc42046b750, 0xc4204c4a80)
	/usr/local/go/src/runtime/select.go:395 +0x12c3
runtime.ensureSigM.func1()
	/usr/local/go/src/runtime/signal_unix.go:511 +0x1a7
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2337 +0x1

goroutine 21501 [semacquire]:
sync.runtime_notifyListWait(0xc421b30b90, 0x147)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc423e08390)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc423e08390)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c28630, 0xc420276ab0, 0xc422c28620)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21447 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc421b30b90, 0x10b)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc420e325a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc420e325a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4208a6260, 0xc420276ab0, 0xc4208a6250)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21527 [semacquire]:
sync.runtime_notifyListWait(0xc421b30b90, 0x150)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc423e085d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc423e085d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c28970, 0xc420276ab0, 0xc422c28960)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21453 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc421b30b90, 0x114)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc420e32900)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc420e32900)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4208a63a0, 0xc420276ab0, 0xc4208a6390)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21460 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc421b30b90, 0x11b)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc420e32b40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc420e32b40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4208a6510, 0xc420276ab0, 0xc4208a6500)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21276 [select, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).bootstrap.func1(0x3c07c80, 0xc422c39620)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1160 +0x623
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420eefad0, 0xc420276ab0, 0xc420eefac0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21458 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc421b30b90, 0x117)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc420e32ab0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc420e32ab0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4208a64d0, 0xc420276ab0, 0xc4208a64c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21589 [select]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).startSampleEnvironment.func1(0x3c07c80, 0xc420aec9f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1419 +0x1ab
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c29200, 0xc420276ab0, 0xc4204f8300)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21463 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc421b30b90, 0x11c)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc421512ba0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc421512ba0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4208a65d0, 0xc420276ab0, 0xc4208a6580)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21502 [semacquire]:
sync.runtime_notifyListWait(0xc421b30b90, 0x142)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc423e08270)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc423e08270)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c28650, 0xc420276ab0, 0xc422c28640)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21505 [semacquire]:
sync.runtime_notifyListWait(0xc421b30b90, 0x148)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc423e08360)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc423e08360)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c286b0, 0xc420276ab0, 0xc422c286a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21519 [semacquire]:
sync.runtime_notifyListWait(0xc421b30b90, 0x14d)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc423e08570)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc423e08570)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c28870, 0xc420276ab0, 0xc422c28860)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21603 [select, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x3c07c80, 0xc4213b6b40)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x1af
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c29460, 0xc420276ab0, 0xc422c29430)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21476 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc421b30b90, 0x122)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc4206620c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc4206620c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c282f0, 0xc420276ab0, 0xc422c282c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21532 [semacquire]:
sync.runtime_notifyListWait(0xc421b30b90, 0x166)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc423e087e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc423e087e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c28a10, 0xc420276ab0, 0xc422c28a00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21498 [semacquire]:
sync.runtime_notifyListWait(0xc421b30b90, 0x159)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc421512c00)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc421512c00)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c285d0, 0xc420276ab0, 0xc422c285c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21594 [select, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x3c07c80, 0xc420e33050)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x1af
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c292b0, 0xc420276ab0, 0xc422c292a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21564 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x3c07c80, 0xc42039f9e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1416 +0x4dd
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c28e10, 0xc420276ab0, 0xc420d425a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21244 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc421b30b90, 0x128)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc42049a5d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc42049a5d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c281d0, 0xc420276ab0, 0xc422c281c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21574 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/credentials.(*tlsCreds).ClientHandshake(0xc420b10078, 0x3c07c40, 0xc422963b60, 0xc420f48bb0, 0xf, 0x3c139a0, 0xc420b420a8, 0x0, 0x0, 0x45, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/credentials/credentials.go:149 +0x2df
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client(0x3c07c40, 0xc422963b60, 0x3c07bc0, 0xc421c48480, 0xc420f48bb0, 0xf, 0x0, 0x0, 0xc420f48bb0, 0xf, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:182 +0x2902
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.NewClientTransport(0x3c07c40, 0xc422963b60, 0x3c07bc0, 0xc421472c00, 0xc420f48bb0, 0xf, 0x0, 0x0, 0xc420f48bb0, 0xf, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:518 +0xd8
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).createTransport(0xc422432500, 0x8, 0x0, 0xbe8ee5c7424f8769, 0x6620f80490, 0x3e296e0, 0xbe8ee5cc0e0f285a, 0x6a9934a381, 0x3e296e0, 0xc421c48400, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1139 +0x3fa
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc422432500, 0x38, 0xc4203eb000)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1100 +0x7b6
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).transportMonitor(0xc422432500)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1273 +0x61a
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect.func1(0xc422432500)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:837 +0x269
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:828 +0x186

goroutine 21240 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc421b30b90, 0x136)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc420662210)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc420662210)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c28150, 0xc420276ab0, 0xc422c28140)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21477 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc421b30b90, 0x13c)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc4200e2060)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc4200e2060)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c28310, 0xc420276ab0, 0xc422c28300)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21543 [semacquire]:
sync.runtime_notifyListWait(0xc421b30b90, 0x171)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc42039f6e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc42039f6e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c28b70, 0xc420276ab0, 0xc422c28b60)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21551 [semacquire]:
sync.runtime_notifyListWait(0xc421b30b90, 0x16c)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc42039f5c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc42039f5c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c28c70, 0xc420276ab0, 0xc422c28c60)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21520 [semacquire]:
sync.runtime_notifyListWait(0xc421b30b90, 0x151)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc423e085a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc423e085a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c28890, 0xc420276ab0, 0xc422c28880)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21273 [chan receive]:
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*muxListener).Accept(0xc42140d760, 0x2a461a0, 0xc4216ea2c0, 0x3c13820, 0xc420df20a0)
	<autogenerated>:1 +0x8a
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).Serve(0xc4216ea2c0, 0x3c06e40, 0xc42140d760, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:504 +0x265
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func7(0x3c07c80, 0xc421566db0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:798 +0x8d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420eef8d0, 0xc420276ab0, 0xc42140d880)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21491 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc421b30b90, 0x137)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc420662240)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc420662240)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c284f0, 0xc420276ab0, 0xc422c284e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21517 [semacquire]:
sync.runtime_notifyListWait(0xc421b30b90, 0x15c)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc423e08540)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc423e08540)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c28830, 0xc420276ab0, 0xc422c28820)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21277 [select]:
github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).manage.func1(0x3c07c80, 0xc422c39650)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1189 +0x332
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420eefaf0, 0xc420276ab0, 0xc420eefae0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21508 [semacquire]:
sync.runtime_notifyListWait(0xc421b30b90, 0x162)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc423e083f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc423e083f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c28710, 0xc420276ab0, 0xc422c28700)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21492 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc421b30b90, 0x132)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc42039f500)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc42039f500)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c28510, 0xc420276ab0, 0xc422c28500)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21503 [semacquire]:
sync.runtime_notifyListWait(0xc421b30b90, 0x156)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc423e082d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc423e082d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c28670, 0xc420276ab0, 0xc422c28660)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21563 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).Start.func2(0x3c07c80, 0xc421512d80)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1282 +0x15d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c28e00, 0xc420276ab0, 0xc4204f8000)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 19930 [select, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x3c07c80, 0xc42039fb00)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:514 +0x1f8
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203ea140, 0xc420276ab0, 0xc42022b8e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21243 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc421b30b90, 0x13a)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc4206621e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc4206621e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c281b0, 0xc420276ab0, 0xc422c281a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21613 [select]:
github.com/cockroachdb/cockroach/pkg/sql/jobs.(*Registry).Start.func1(0x3c07c80, 0xc422c380c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/jobs/registry.go:194 +0x17e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4210ba330, 0xc420276ab0, 0xc421379f80)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 20929 [select, 4 minutes]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc420c80000)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:122 +0x1e7
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:113 +0x2f2

goroutine 21452 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc421b30b90, 0x113)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc420e32960)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc420e32960)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4208a6330, 0xc420276ab0, 0xc4208a6320)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21223 [select, 4 minutes]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccResolverWrapper).watcher(0xc421ca4cc0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/resolver_conn_wrapper.go:110 +0x22f
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccResolverWrapper).start
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/resolver_conn_wrapper.go:96 +0x4d

goroutine 21538 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc421b30b90, 0xfd)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc4200e2090)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc4200e2090)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c28ad0, 0xc420276ab0, 0xc422c28ac0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21454 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc421b30b90, 0x10d)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc420e32600)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc420e32600)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4208a63c0, 0xc420276ab0, 0xc4208a63b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 7105 [runnable]:
github.com/cockroachdb/cockroach/pkg/sql.TestTxnCanStillResolveOldName.func2(0xc420ded000, 0x24, 0x40)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/rename_test.go:148 +0x163
github.com/cockroachdb/cockroach/pkg/sql.(*LeaseManager).RefreshLeases.func1(0x3c07c80, 0xc421566cf0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/lease.go:1516 +0xd83
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420eef550, 0xc420276ab0, 0xc421823890)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21507 [semacquire]:
sync.runtime_notifyListWait(0xc421b30b90, 0x145)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc423e08150)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc423e08150)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c286f0, 0xc420276ab0, 0xc422c286e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21478 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc421b30b90, 0x121)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc4200e2030)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc4200e2030)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c28330, 0xc420276ab0, 0xc422c28320)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21533 [semacquire]:
sync.runtime_notifyListWait(0xc421b30b90, 0x15f)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc423e089c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc423e089c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c28a30, 0xc420276ab0, 0xc422c28a20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21609 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*Executor).Start.func1(0x3c07c80, 0xc4213b7620)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/executor.go:417 +0x15d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c29540, 0xc420276ab0, 0xc4204f8380)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21490 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc421b30b90, 0x134)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc421202000)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc421202000)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c284d0, 0xc420276ab0, 0xc422c284c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21611 [select, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*flowScheduler).Start.func1(0x3c07c80, 0xc422522000)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/flow_scheduler.go:125 +0x1b6
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4210ba160, 0xc420276ab0, 0xc4210ba140)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21226 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc421b30b90, 0xfe)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc420e32060)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc420e32060)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4208a60c0, 0xc420276ab0, 0xc4208a60b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21475 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc421b30b90, 0x135)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc421d00000)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc421d00000)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c282b0, 0xc420276ab0, 0xc422c282a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21449 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc421b30b90, 0x111)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc420e32840)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc420e32840)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4208a62a0, 0xc420276ab0, 0xc4208a6290)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21534 [semacquire]:
sync.runtime_notifyListWait(0xc421b30b90, 0x160)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc423e08990)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc423e08990)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c28a50, 0xc420276ab0, 0xc422c28a40)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21542 [semacquire]:
sync.runtime_notifyListWait(0xc421b30b90, 0x16d)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc42039f5f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc42039f5f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c28b50, 0xc420276ab0, 0xc422c28b40)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21275 [chan receive, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/gossip.(*server).start.func3(0x3c07c80, 0xc421566de0)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/server.go:371 +0x83
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420eefab0, 0xc420276ab0, 0xc42140d9e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21590 [runnable]:
github.com/cockroachdb/cockroach/pkg/roachpb.Span.Contains(0xc4201f7140, 0x2d, 0x30, 0xc4201f7140, 0x2e, 0x30, 0xc42178dd00, 0x24, 0x40, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/roachpb/data.go:1289 +0x2ed
github.com/cockroachdb/cockroach/pkg/storage/spanset.(*SpanSet).CheckAllowed(0xc42369daa0, 0x1, 0xc42178dd00, 0x24, 0x40, 0x0, 0x0, 0x0, 0xc4230e7688, 0xc4211bfc90)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/spanset/spanset.go:120 +0x248
github.com/cockroachdb/cockroach/pkg/storage/spanset.spanSetWriter.Merge(0x7f1194570ea0, 0xc420c7e000, 0xc42369daa0, 0xc42178dd00, 0x24, 0x40, 0x0, 0x0, 0xc4210cfa00, 0x40, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/spanset/batch.go:278 +0xc3
github.com/cockroachdb/cockroach/pkg/storage/spanset.(*spanSetBatch).Merge(0xc42261bb30, 0xc42178dd00, 0x24, 0x40, 0x0, 0x0, 0xc4210cfa00, 0x40, 0x40, 0xc4211c0a00, ...)
	<autogenerated>:1 +0xdb
github.com/cockroachdb/cockroach/pkg/storage/engine.MVCCMerge(0x3c07c80, 0xc420e58f30, 0x7f11909bf028, 0xc42261bb30, 0xc42165d6c0, 0xc42178dd00, 0x24, 0x40, 0x1509a269432def93, 0x0, ...)

Metadata

Metadata

Assignees

Labels

C-test-failureBroken test (automatically or manually discovered).O-robotOriginated from a bot.

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions