Skip to content

teamcity: failed tests on release-2.0: acceptance/TestJSONBUpgrade, testrace/TestBehaviorDuringLeaseTransfer #29189

Closed
Closed
teamcity: failed tests on release-2.0: acceptance/TestJSONBUpgrade, testrace/TestBehaviorDuringLeaseTransfer#29189
Assignees
a-robinson
Labels
C-test-failureBroken test (automatically or manually discovered).O-robotOriginated from a bot.
@cockroach-teamcity

Description

@cockroach-teamcity
cockroach-teamcity
opened on Aug 28, 2018

The following tests appear to have failed:

#866778:

--- FAIL: testrace/TestBehaviorDuringLeaseTransfer (0.090s)
panic: close of closed channel [recovered]
panic: close of closed channel [recovered]
panic: close of closed channel [recovered]
panic: close of closed channel
/usr/local/go/src/testing/testing.go:742 +0x567
/usr/local/go/src/runtime/panic.go:505 +0x24a
/go/src/github.com/cockroachdb/cockroach/pkg/util/leaktest/leaktest.go:89 +0x526
/usr/local/go/src/runtime/panic.go:505 +0x24a
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:181 +0x12e
/usr/local/go/src/runtime/panic.go:505 +0x24a
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470 +0x66c
/usr/local/go/src/runtime/panic.go:505 +0x24a
/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_test.go:660 +0x68
/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_test.go:557 +0x8b
/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_range_lease.go:516 +0xb08
/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:1443 +0x12c2
/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:1452 +0x261
/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2507 +0xdae
/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:1900 +0x31d
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/sender.go:176 +0x109
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/sender.go:103 +0x86
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/sender.go:152 +0x158
/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_test.go:270 +0xeb
/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_test.go:275 +0x96
/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_test.go:668 +0x100d
/usr/local/go/src/testing/testing.go:777 +0x16e
/usr/local/go/src/testing/testing.go:824 +0x565
/usr/local/go/src/testing/testing.go:825 +0x597
/usr/local/go/src/testing/testing.go:1063 +0xa5
/usr/local/go/src/testing/testing.go:777 +0x16e
/usr/local/go/src/testing/testing.go:1061 +0x4e2
/usr/local/go/src/testing/testing.go:978 +0x2ce
/go/src/github.com/cockroachdb/cockroach/pkg/storage/main_test.go:57 +0x2a9
_testmain.go:1048 +0x22b
/usr/local/go/src/runtime/sigqueue.go:139 +0xa6
/usr/local/go/src/os/signal/signal_unix.go:22 +0x30
/usr/local/go/src/os/signal/signal_unix.go:28 +0x4f
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1161 +0x132
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:592 +0x1b5
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:601 +0x16b
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:593 +0x1cd
/usr/local/go/src/runtime/proc.go:291 +0xf9
/usr/local/go/src/runtime/select.go:392 +0x11d4
/usr/local/go/src/runtime/signal_unix.go:549 +0x19f
/usr/local/go/src/runtime/asm_amd64.s:2361 +0x1
/usr/local/go/src/runtime/sema.go:71 +0x3d
/usr/local/go/src/sync/mutex.go:134 +0x172
/go/src/github.com/cockroachdb/cockroach/pkg/util/syncutil/mutex_sync.go:32 +0x3b
/usr/local/go/src/sync/cond.go:57 +0xa4
/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:635 +0x86
/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:623 +0x625
/go/src/github.com/cockroachdb/cockroach/pkg/util/syncutil/mutex_sync.go:37 +0x69
/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:1764 +0x997
/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:3623 +0x71e
/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:3404 +0xec
/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:4005 +0x104
/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:215 +0x344
/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x14b
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xbb
/usr/local/go/src/runtime/sema.go:56 +0x39
/usr/local/go/src/sync/waitgroup.go:129 +0xb3
/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:172 +0x3f
/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:4059 +0x56
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x14b
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xbb
/usr/local/go/src/runtime/sema.go:510 +0x11a
/usr/local/go/src/sync/cond.go:56 +0x8e
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:536 +0x271
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:462 +0x13d
/usr/local/go/src/context/context.go:325 +0x11c
/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2770 +0x52b
/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2600 +0xbf
/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:1897 +0x7ce
/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_range_lease.go:310 +0x6ac
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xf4
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x14d
/usr/local/go/src/runtime/sema.go:71 +0x3d
/usr/local/go/src/sync/mutex.go:134 +0x172
/go/src/github.com/cockroachdb/cockroach/pkg/util/syncutil/mutex_sync.go:32 +0x3b
/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:703 +0x1d8
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:467 +0x21d
------- Stdout: -------
I180828 17:11:23.434116 11550 storage/store.go:1379  [s1] [n1,s1]: failed initial metrics computation: [n1,s1]: system config not yet available
I180828 17:11:23.458776 12116 storage/replica_command.go:83  [s1,r1/1:/M{in-ax}] test injecting error: storage/replica_test.go:570: injected transfer error
I180828 17:11:23.467324 11824 util/stop/stopper.go:534  quiescing; tasks left:
1      [async] storage.pendingLeaseRequest: requesting lease
panic: close of closed channel [recovered]

goroutine 11550 [running]:
testing.tRunner.func1(0xc420a26000)
panic(0x2ad3900, 0x32de370)
github.com/cockroachdb/cockroach/pkg/util/leaktest.AfterTest.func1()
panic(0x2ad3900, 0x32de370)
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Recover(0xc4206a0870, 0x3307d20, 0xc4200cc058)
panic(0x2ad3900, 0x32de370)
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Stop(0xc4206a0870, 0x3307d20, 0xc4200cc058)
panic(0x2ad3900, 0x32de370)
github.com/cockroachdb/cockroach/pkg/storage.TestBehaviorDuringLeaseTransfer.func5(0x26f, 0xc)
github.com/cockroachdb/cockroach/pkg/storage.TestBehaviorDuringLeaseTransfer.func1(0x26f, 0xc)
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).requestLeaseLocked(0xc420456a80, 0x3307da0, 0xc420c3da10, 0x0, 0x0, 0xc420046d60, 0x100000001, 0x1, 0xc420046d70, 0xc420046d80, ...)
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).redirectOnOrAcquireLease.func1(0xc420456a80, 0x26f, 0xc40000000c, 0xc4204851f0, 0x3307da0, 0xc420c3da10, 0xc420cae538, 0x0, 0x0)
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).redirectOnOrAcquireLease(0xc420456a80, 0x3307da0, 0xc420c3da10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).executeReadOnlyBatch(0xc420456a80, 0x3307da0, 0xc420c3da10, 0x26f, 0x6, 0x0, 0x0, 0x1, 0x0, 0x0, ...)
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).Send(0xc420456a80, 0x3307da0, 0xc420c3da10, 0x26f, 0x6, 0x0, 0x0, 0x1, 0x0, 0x0, ...)
github.com/cockroachdb/cockroach/pkg/internal/client.Wrap.func1(0x3307d20, 0xc4200cc050, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
github.com/cockroachdb/cockroach/pkg/internal/client.SenderFunc.Send(0xc4203218e0, 0x3307d20, 0xc4200cc050, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
github.com/cockroachdb/cockroach/pkg/internal/client.SendWrappedWith(0x3307d20, 0xc4200cc050, 0x32e6c80, 0xc4203218e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
github.com/cockroachdb/cockroach/pkg/storage.(*testContext).SendWrappedWith(0xc420bcc660, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
github.com/cockroachdb/cockroach/pkg/storage.(*testContext).SendWrapped(0xc420bcc660, 0x332e0a0, 0xc420e4fcb0, 0x0, 0x0, 0xc420e51450)
github.com/cockroachdb/cockroach/pkg/storage.TestBehaviorDuringLeaseTransfer(0xc420a26000)
testing.tRunner(0xc420a26000, 0x2e7d4a8)
created by testing.(*T).Run

goroutine 1 [chan receive]:
testing.(*T).Run(0xc420238690, 0x2d80b27, 0x1f, 0x2e7d4a8, 0xc42060b901)
testing.runTests.func1(0xc420238690)
testing.tRunner(0xc420238690, 0xc42060bb20)
testing.runTests(0xc4202a4f40, 0x41bcee0, 0x1f0, 0x1f0, 0xc4202a3838)
testing.(*M).Run(0xc420546180, 0x0)
github.com/cockroachdb/cockroach/pkg/storage_test.TestMain(0xc420546180)
main.main()

goroutine 20 [syscall]:
os/signal.signal_recv(0x857081)
os/signal.loop()
created by os/signal.init.0

goroutine 4 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.flushDaemon()
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0

goroutine 5 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.signalFlusher()
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0

goroutine 29 [select, locked to thread]:
runtime.gopark(0x2e80aa0, 0x0, 0x2d52079, 0x6, 0x18, 0x1)
runtime.selectgo(0xc42007df50, 0xc4200b6300)
runtime.ensureSigM.func1()
runtime.goexit()

goroutine 11552 [semacquire]:
sync.runtime_SemacquireMutex(0xc420b62794, 0x900000000)
sync.(*Mutex).Lock(0xc420b62790)
github.com/cockroachdb/cockroach/pkg/util/syncutil.(*Mutex).Lock(0xc420b62790)
sync.(*Cond).Wait(0xc420b627a0)
github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).syncLoop(0xc420b62680)
created by github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).open

goroutine 12020 [runnable]:
github.com/cockroachdb/cockroach/pkg/util/syncutil.(*Mutex).Unlock(0xc420b62790)
github.com/cockroachdb/cockroach/pkg/storage/engine.(*rocksDBBatch).Commit(0xc42068ab40, 0xc4208c5901, 0x0, 0x0)
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked(0xc420456a80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReady(0xc420456a80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processReady(0xc420113b00, 0x3307da0, 0xc420c3c120, 0x1)
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4206a0900, 0x3307da0, 0xc420c3c120)
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3307da0, 0xc420c3c120)
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202a39c0, 0xc4206a0870, 0xc4202a39b0)
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker

goroutine 12082 [semacquire]:
sync.runtime_Semacquire(0xc4206a0984)
sync.(*WaitGroup).Wait(0xc4206a0978)
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Wait(0xc4206a0900, 0x3307da0, 0xc420c3cab0)
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Wait-fm(0x3307da0, 0xc420c3cab0)
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420344580, 0xc4206a0870, 0xc420344570)
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker

goroutine 11824 [semacquire]:
sync.runtime_notifyListWait(0xc420a1c010, 0xc400000000)
sync.(*Cond).Wait(0xc420a1c000)
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Quiesce(0xc4206a0870, 0x3307d20, 0xc4200cc058)
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Stop

goroutine 12091 [runnable]:
context.(*cancelCtx).Done(0xc42040da40, 0xc420bdd210)
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).tryExecuteWriteBatch(0xc420456a80, 0x3307da0, 0xc4207c2360, 0x26f, 0xa, 0x0, 0x0, 0x1, 0x0, 0x0, ...)
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).executeWriteBatch(0xc420456a80, 0x3307da0, 0xc4207c2360, 0x26f, 0xa, 0x0, 0x0, 0x1, 0x0, 0x0, ...)
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).Send(0xc420456a80, 0x3307da0, 0xc4207c2360, 0x26f, 0xa, 0x0, 0x0, 0x1, 0x0, 0x0, ...)
github.com/cockroachdb/cockroach/pkg/storage.(*pendingLeaseRequest).requestLeaseAsync.func2(0x3307da0, 0xc4207c2360)
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc4206a0870, 0x3307da0, 0xc4207c2360, 0xc420172ac0, 0x35, 0x3332da0, 0xc420233790, 0xc42050a750)
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask

goroutine 12131 [semacquire]:
sync.runtime_SemacquireMutex(0xc420b62794, 0x1100000000)
sync.(*Mutex).Lock(0xc420b62790)
github.com/cockroachdb/cockroach/pkg/util/syncutil.(*Mutex).Lock(0xc420b62790)
github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).Close(0xc420b62680)
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Stop
--- FAIL: acceptance/TestJSONBUpgrade (1.660s)
test_log_scope.go:81: test logs captured to: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/acceptance/logTestJSONBUpgrade654529018
test_log_scope.go:62: use -show-logs to present logs inline
--- FAIL: acceptance/TestJSONBUpgrade: TestJSONBUpgrade/runMode=local (1.650s)
util_cluster.go:214: dial tcp 127.0.0.1:5432: connect: connection refused
------- Stdout: -------
CockroachDB node starting at 2018-08-28 17:13:41.087456765 +0000 UTC (took 0.5s)
build:      CCL v1.1.8 @ 2018/04/23 17:25:48 (go1.8.3)
admin:      http://127.0.0.1:34199
sql:        postgresql://root@127.0.0.1:44399?application_name=cockroach&sslmode=disable
logs:       /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/acceptance/TestJSONBUpgrade/runMode=local/1
store[0]:   path=/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/acceptance/.localcluster343418129/1
status:     initialized new cluster
clusterID:  dcd288c5-8b72-438d-a4bd-2fa6edd5c061
nodeID:     1
test logs left over in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/acceptance/logTestJSONBUpgrade654529018
--- FAIL: acceptance/TestJSONBUpgrade (1.660s)
test_log_scope.go:81: test logs captured to: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/acceptance/logTestJSONBUpgrade654529018
test_log_scope.go:62: use -show-logs to present logs inline
--- FAIL: acceptance/TestJSONBUpgrade: TestJSONBUpgrade/runMode=local (1.650s)
util_cluster.go:214: dial tcp 127.0.0.1:5432: connect: connection refused
------- Stdout: -------
CockroachDB node starting at 2018-08-28 17:13:41.087456765 +0000 UTC (took 0.5s)
build:      CCL v1.1.8 @ 2018/04/23 17:25:48 (go1.8.3)
admin:      http://127.0.0.1:34199
sql:        postgresql://root@127.0.0.1:44399?application_name=cockroach&sslmode=disable
logs:       /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/acceptance/TestJSONBUpgrade/runMode=local/1
store[0]:   path=/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/acceptance/.localcluster343418129/1
status:     initialized new cluster
clusterID:  dcd288c5-8b72-438d-a4bd-2fa6edd5c061
nodeID:     1
test logs left over in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/acceptance/logTestJSONBUpgrade654529018
--- FAIL: testrace/TestBehaviorDuringLeaseTransfer (0.180s)
panic: close of closed channel [recovered]
panic: close of closed channel [recovered]
panic: close of closed channel [recovered]
panic: close of closed channel
/usr/local/go/src/testing/testing.go:742 +0x567
/usr/local/go/src/runtime/panic.go:505 +0x24a
/go/src/github.com/cockroachdb/cockroach/pkg/util/leaktest/leaktest.go:89 +0x526
/usr/local/go/src/runtime/panic.go:505 +0x24a
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:181 +0x12e
/usr/local/go/src/runtime/panic.go:505 +0x24a
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470 +0x66c
/usr/local/go/src/runtime/panic.go:505 +0x24a
/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_test.go:660 +0x68
/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_test.go:557 +0x8b
/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_range_lease.go:516 +0xb08
/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:1443 +0x12c2
/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:1452 +0x261
/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2507 +0xdae
/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:1900 +0x31d
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/sender.go:176 +0x109
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/sender.go:103 +0x86
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/sender.go:152 +0x158
/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_test.go:270 +0xeb
/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_test.go:275 +0x96
/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_test.go:668 +0x100d
/usr/local/go/src/testing/testing.go:777 +0x16e
/usr/local/go/src/testing/testing.go:824 +0x565
/usr/local/go/src/testing/testing.go:825 +0x597
/usr/local/go/src/testing/testing.go:1063 +0xa5
/usr/local/go/src/testing/testing.go:777 +0x16e
/usr/local/go/src/testing/testing.go:1061 +0x4e2
/usr/local/go/src/testing/testing.go:978 +0x2ce
/go/src/github.com/cockroachdb/cockroach/pkg/storage/main_test.go:57 +0x2a9
_testmain.go:1048 +0x22b
/usr/local/go/src/runtime/sigqueue.go:139 +0xa6
/usr/local/go/src/os/signal/signal_unix.go:22 +0x30
/usr/local/go/src/os/signal/signal_unix.go:28 +0x4f
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1161 +0x132
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:592 +0x1b5
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:601 +0x16b
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:593 +0x1cd
/usr/local/go/src/runtime/proc.go:291 +0xf9
/usr/local/go/src/runtime/select.go:392 +0x11d4
/usr/local/go/src/runtime/signal_unix.go:549 +0x19f
/usr/local/go/src/runtime/asm_amd64.s:2361 +0x1
/usr/local/go/src/runtime/sema.go:71 +0x3d
/usr/local/go/src/sync/mutex.go:134 +0x172
/go/src/github.com/cockroachdb/cockroach/pkg/util/syncutil/mutex_sync.go:32 +0x3b
/usr/local/go/src/sync/cond.go:57 +0xa4
/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:635 +0x86
/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:623 +0x625
/go/src/github.com/cockroachdb/cockroach/pkg/util/syncutil/mutex_sync.go:37 +0x69
/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:1764 +0x997
/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:3623 +0x71e
/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:3404 +0xec
/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:4005 +0x104
/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:215 +0x344
/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x14b
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xbb
/usr/local/go/src/runtime/sema.go:56 +0x39
/usr/local/go/src/sync/waitgroup.go:129 +0xb3
/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:172 +0x3f
/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:4059 +0x56
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x14b
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xbb
/usr/local/go/src/runtime/sema.go:510 +0x11a
/usr/local/go/src/sync/cond.go:56 +0x8e
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:536 +0x271
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:462 +0x13d
/usr/local/go/src/context/context.go:325 +0x11c
/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2770 +0x52b
/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2600 +0xbf
/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:1897 +0x7ce
/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_range_lease.go:310 +0x6ac
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xf4
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x14d
/usr/local/go/src/runtime/sema.go:71 +0x3d
/usr/local/go/src/sync/mutex.go:134 +0x172
/go/src/github.com/cockroachdb/cockroach/pkg/util/syncutil/mutex_sync.go:32 +0x3b
/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:703 +0x1d8
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:467 +0x21d
------- Stdout: -------
I180828 17:11:23.434116 11550 storage/store.go:1379  [s1] [n1,s1]: failed initial metrics computation: [n1,s1]: system config not yet available
I180828 17:11:23.458776 12116 storage/replica_command.go:83  [s1,r1/1:/M{in-ax}] test injecting error: storage/replica_test.go:570: injected transfer error
I180828 17:11:23.467324 11824 util/stop/stopper.go:534  quiescing; tasks left:
1      [async] storage.pendingLeaseRequest: requesting lease
panic: close of closed channel [recovered]

goroutine 11550 [running]:
testing.tRunner.func1(0xc420a26000)
panic(0x2ad3900, 0x32de370)
github.com/cockroachdb/cockroach/pkg/util/leaktest.AfterTest.func1()
panic(0x2ad3900, 0x32de370)
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Recover(0xc4206a0870, 0x3307d20, 0xc4200cc058)
panic(0x2ad3900, 0x32de370)
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Stop(0xc4206a0870, 0x3307d20, 0xc4200cc058)
panic(0x2ad3900, 0x32de370)
github.com/cockroachdb/cockroach/pkg/storage.TestBehaviorDuringLeaseTransfer.func5(0x26f, 0xc)
github.com/cockroachdb/cockroach/pkg/storage.TestBehaviorDuringLeaseTransfer.func1(0x26f, 0xc)
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).requestLeaseLocked(0xc420456a80, 0x3307da0, 0xc420c3da10, 0x0, 0x0, 0xc420046d60, 0x100000001, 0x1, 0xc420046d70, 0xc420046d80, ...)
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).redirectOnOrAcquireLease.func1(0xc420456a80, 0x26f, 0xc40000000c, 0xc4204851f0, 0x3307da0, 0xc420c3da10, 0xc420cae538, 0x0, 0x0)
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).redirectOnOrAcquireLease(0xc420456a80, 0x3307da0, 0xc420c3da10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).executeReadOnlyBatch(0xc420456a80, 0x3307da0, 0xc420c3da10, 0x26f, 0x6, 0x0, 0x0, 0x1, 0x0, 0x0, ...)
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).Send(0xc420456a80, 0x3307da0, 0xc420c3da10, 0x26f, 0x6, 0x0, 0x0, 0x1, 0x0, 0x0, ...)
github.com/cockroachdb/cockroach/pkg/internal/client.Wrap.func1(0x3307d20, 0xc4200cc050, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
github.com/cockroachdb/cockroach/pkg/internal/client.SenderFunc.Send(0xc4203218e0, 0x3307d20, 0xc4200cc050, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
github.com/cockroachdb/cockroach/pkg/internal/client.SendWrappedWith(0x3307d20, 0xc4200cc050, 0x32e6c80, 0xc4203218e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
github.com/cockroachdb/cockroach/pkg/storage.(*testContext).SendWrappedWith(0xc420bcc660, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
github.com/cockroachdb/cockroach/pkg/storage.(*testContext).SendWrapped(0xc420bcc660, 0x332e0a0, 0xc420e4fcb0, 0x0, 0x0, 0xc420e51450)
github.com/cockroachdb/cockroach/pkg/storage.TestBehaviorDuringLeaseTransfer(0xc420a26000)
testing.tRunner(0xc420a26000, 0x2e7d4a8)
created by testing.(*T).Run

goroutine 1 [chan receive]:
testing.(*T).Run(0xc420238690, 0x2d80b27, 0x1f, 0x2e7d4a8, 0xc42060b901)
testing.runTests.func1(0xc420238690)
testing.tRunner(0xc420238690, 0xc42060bb20)
testing.runTests(0xc4202a4f40, 0x41bcee0, 0x1f0, 0x1f0, 0xc4202a3838)
testing.(*M).Run(0xc420546180, 0x0)
github.com/cockroachdb/cockroach/pkg/storage_test.TestMain(0xc420546180)
main.main()

goroutine 20 [syscall]:
os/signal.signal_recv(0x857081)
os/signal.loop()
created by os/signal.init.0

goroutine 4 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.flushDaemon()
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0

goroutine 5 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.signalFlusher()
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0

goroutine 29 [select, locked to thread]:
runtime.gopark(0x2e80aa0, 0x0, 0x2d52079, 0x6, 0x18, 0x1)
runtime.selectgo(0xc42007df50, 0xc4200b6300)
runtime.ensureSigM.func1()
runtime.goexit()

goroutine 11552 [semacquire]:
sync.runtime_SemacquireMutex(0xc420b62794, 0x900000000)
sync.(*Mutex).Lock(0xc420b62790)
github.com/cockroachdb/cockroach/pkg/util/syncutil.(*Mutex).Lock(0xc420b62790)
sync.(*Cond).Wait(0xc420b627a0)
github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).syncLoop(0xc420b62680)
created by github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).open

goroutine 12020 [runnable]:
github.com/cockroachdb/cockroach/pkg/util/syncutil.(*Mutex).Unlock(0xc420b62790)
github.com/cockroachdb/cockroach/pkg/storage/engine.(*rocksDBBatch).Commit(0xc42068ab40, 0xc4208c5901, 0x0, 0x0)
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked(0xc420456a80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReady(0xc420456a80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processReady(0xc420113b00, 0x3307da0, 0xc420c3c120, 0x1)
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4206a0900, 0x3307da0, 0xc420c3c120)
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3307da0, 0xc420c3c120)
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202a39c0, 0xc4206a0870, 0xc4202a39b0)
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker

goroutine 12082 [semacquire]:
sync.runtime_Semacquire(0xc4206a0984)
sync.(*WaitGroup).Wait(0xc4206a0978)
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Wait(0xc4206a0900, 0x3307da0, 0xc420c3cab0)
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Wait-fm(0x3307da0, 0xc420c3cab0)
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420344580, 0xc4206a0870, 0xc420344570)
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker

goroutine 11824 [semacquire]:
sync.runtime_notifyListWait(0xc420a1c010, 0xc400000000)
sync.(*Cond).Wait(0xc420a1c000)
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Quiesce(0xc4206a0870, 0x3307d20, 0xc4200cc058)
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Stop

goroutine 12091 [runnable]:
context.(*cancelCtx).Done(0xc42040da40, 0xc420bdd210)
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).tryExecuteWriteBatch(0xc420456a80, 0x3307da0, 0xc4207c2360, 0x26f, 0xa, 0x0, 0x0, 0x1, 0x0, 0x0, ...)
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).executeWriteBatch(0xc420456a80, 0x3307da0, 0xc4207c2360, 0x26f, 0xa, 0x0, 0x0, 0x1, 0x0, 0x0, ...)
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).Send(0xc420456a80, 0x3307da0, 0xc4207c2360, 0x26f, 0xa, 0x0, 0x0, 0x1, 0x0, 0x0, ...)
github.com/cockroachdb/cockroach/pkg/storage.(*pendingLeaseRequest).requestLeaseAsync.func2(0x3307da0, 0xc4207c2360)
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc4206a0870, 0x3307da0, 0xc4207c2360, 0xc420172ac0, 0x35, 0x3332da0, 0xc420233790, 0xc42050a750)
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask

goroutine 12131 [semacquire]:
sync.runtime_SemacquireMutex(0xc420b62794, 0x1100000000)
sync.(*Mutex).Lock(0xc420b62790)
github.com/cockroachdb/cockroach/pkg/util/syncutil.(*Mutex).Lock(0xc420b62790)
github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).Close(0xc420b62680)
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Stop
------- Stdout: -------
I180828 17:02:01.980526 12041 storage/store.go:1379  [s1] [n1,s1]: failed initial metrics computation: [n1,s1]: system config not yet available
I180828 17:02:01.984895 12121 storage/replica_command.go:83  [s1,r1/1:/M{in-ax}] test injecting error: storage/replica_test.go:570: injected transfer error

Please assign, take a look and update the issue accordingly.

Metadata

Metadata

Assignees

Labels

C-test-failureBroken test (automatically or manually discovered).O-robotOriginated from a bot.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions