-
Notifications
You must be signed in to change notification settings - Fork 4.1k
storage: TestStoreRangeSplitBackpressureWrites/splitErr=bad_error failed under stress #25142
Copy link
Copy link
Closed
Labels
A-kv-replicationRelating to Raft, consensus, and coordination.Relating to Raft, consensus, and coordination.C-test-failureBroken test (automatically or manually discovered).Broken test (automatically or manually discovered).O-robotOriginated from a bot.Originated from a bot.
Milestone
Description
SHA: https://github.com/cockroachdb/cockroach/commits/0eea534c4919d0298473674edfe62ffe291a5d27
Parameters:
TAGS=
GOFLAGS=
Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=628948&tab=buildLog
I180428 06:03:08.614318 96867 storage/store.go:1385 [s1] [n1,s1]: failed initial metrics computation: [n1,s1]: system config not yet available
I180428 06:03:08.616525 96854 storage/replica_command.go:862 [split,s1,r1/1:/M{in-ax}] initiating a split of this range at key /System/"" [r2]
I180428 06:03:08.619867 96982 storage/replica_command.go:862 [split,s1,r2/1:/{System/-Max}] initiating a split of this range at key /System/NodeLiveness [r3]
E180428 06:03:08.620340 96963 storage/queue.go:778 [replicate,s1,r1/1:/{Min-System/}] no removable replicas from range that needs a removal: [1*:15]
W180428 06:03:08.621466 96982 storage/stores.go:222 range not contained in one range: [/Meta2/System/NodeLiveness,/System/NodeLiveness/NULL), but have [/Min,/System/"")
W180428 06:03:08.623002 96961 storage/intent_resolver.go:320 [s1] failed to push during intent resolution: failed to push "split" id=1398a269 key=/Local/Range/System/""/RangeDescriptor rw=true pri=0.00146455 iso=SERIALIZABLE stat=PENDING epo=0 ts=1524895388.619879003,0 orig=1524895388.619879003,0 max=1524895388.619879004,0 wto=false rop=false seq=2
I180428 06:03:08.625212 96984 storage/replica_command.go:862 [split,s1,r3/1:/{System/NodeL…-Max}] initiating a split of this range at key /System/NodeLivenessMax [r4]
E180428 06:03:08.625798 96985 storage/queue.go:778 [replicate,s1,r2/1:/System/{-NodeLive…}] no removable replicas from range that needs a removal: [1*:14]
E180428 06:03:08.631451 96966 storage/queue.go:778 [replicate,s1,r3/1:/System/NodeLiveness{-Max}] no removable replicas from range that needs a removal: [1*:15]
I180428 06:03:08.631994 96986 storage/replica_command.go:862 [split,s1,r4/1:/{System/NodeL…-Max}] initiating a split of this range at key /System/tsd [r5]
I180428 06:03:08.635168 96967 storage/replica_command.go:862 [split,s1,r5/1:/{System/tsd-Max}] initiating a split of this range at key /System/"tse" [r6]
E180428 06:03:08.635415 96990 storage/queue.go:778 [replicate,s1,r4/1:/System/{NodeLive…-tsd}] no removable replicas from range that needs a removal: [1*:14]
I180428 06:03:08.638376 96969 storage/replica_command.go:862 [split,s1,r6/1:/{System/tse-Max}] initiating a split of this range at key /Table/SystemConfigSpan/Start [r7]
E180428 06:03:08.639933 96992 storage/queue.go:778 [replicate,s1,r5/1:/System/ts{d-e}] no removable replicas from range that needs a removal: [1*:14]
I180428 06:03:08.644023 96807 storage/replica_command.go:862 [split,s1,r7/1:/{Table/System…-Max}] initiating a split of this range at key /Table/11 [r8]
E180428 06:03:08.644517 96808 storage/queue.go:778 [replicate,s1,r6/1:/{System/tse-Table/System…}] no removable replicas from range that needs a removal: [1*:14]
E180428 06:03:08.649493 96858 storage/queue.go:778 [replicate,s1,r7/1:/Table/{SystemCon…-11}] no removable replicas from range that needs a removal: [1*:14]
I180428 06:03:08.649522 96809 storage/replica_command.go:862 [split,s1,r8/1:/{Table/11-Max}] initiating a split of this range at key /Table/12 [r9]
I180428 06:03:08.652491 96998 storage/replica_command.go:862 [split,s1,r9/1:/{Table/12-Max}] initiating a split of this range at key /Table/13 [r10]
E180428 06:03:08.652584 96860 storage/queue.go:778 [replicate,s1,r8/1:/Table/1{1-2}] no removable replicas from range that needs a removal: [1*:14]
E180428 06:03:08.655531 96974 storage/queue.go:778 [replicate,s1,r9/1:/Table/1{2-3}] no removable replicas from range that needs a removal: [1*:14]
I180428 06:03:08.655606 96862 storage/replica_command.go:862 [split,s1,r10/1:/{Table/13-Max}] initiating a split of this range at key /Table/14 [r11]
E180428 06:03:08.658773 96864 storage/queue.go:778 [replicate,s1,r10/1:/Table/1{3-4}] no removable replicas from range that needs a removal: [1*:14]
I180428 06:03:08.658847 96863 storage/replica_command.go:862 [split,s1,r11/1:/{Table/14-Max}] initiating a split of this range at key /Table/15 [r12]
W180428 06:03:08.659953 96863 storage/stores.go:222 range not contained in one range: [/Meta2/Table/15,/Table/15/NULL), but have [/Min,/System/"")
E180428 06:03:08.662427 96999 storage/queue.go:778 [replicate,s1,r11/1:/Table/1{4-5}] no removable replicas from range that needs a removal: [1*:14]
I180428 06:03:08.662553 96821 storage/replica_command.go:862 [split,s1,r12/1:/{Table/15-Max}] initiating a split of this range at key /Table/16 [r13]
I180428 06:03:08.665572 96822 storage/replica_command.go:862 [split,s1,r13/1:/{Table/16-Max}] initiating a split of this range at key /Table/17 [r14]
E180428 06:03:08.665584 96816 storage/queue.go:778 [replicate,s1,r12/1:/Table/1{5-6}] no removable replicas from range that needs a removal: [1*:14]
I180428 06:03:08.668810 96367 storage/replica_command.go:862 [split,s1,r14/1:/{Table/17-Max}] initiating a split of this range at key /Table/18 [r15]
E180428 06:03:08.668963 97027 storage/queue.go:778 [replicate,s1,r13/1:/Table/1{6-7}] no removable replicas from range that needs a removal: [1*:14]
I180428 06:03:08.671830 96368 storage/replica_command.go:862 [split,s1,r15/1:/{Table/18-Max}] initiating a split of this range at key /Table/19 [r16]
E180428 06:03:08.671860 96827 storage/queue.go:778 [replicate,s1,r14/1:/Table/1{7-8}] no removable replicas from range that needs a removal: [1*:14]
I180428 06:03:08.675695 97001 storage/replica_command.go:862 [split,s1,r16/1:/{Table/19-Max}] initiating a split of this range at key /Table/20 [r17]
E180428 06:03:08.675979 97002 storage/queue.go:778 [replicate,s1,r15/1:/Table/1{8-9}] no removable replicas from range that needs a removal: [1*:14]
I180428 06:03:08.679622 97042 storage/replica_command.go:862 [split,s1,r17/1:/{Table/20-Max}] initiating a split of this range at key /Table/21 [r18]
E180428 06:03:08.681345 97058 storage/queue.go:778 [replicate,s1,r16/1:/Table/{19-20}] no removable replicas from range that needs a removal: [1*:15]
I180428 06:03:08.683376 97021 storage/replica_command.go:862 [split,s1,r18/1:/{Table/21-Max}] initiating a split of this range at key /Table/22 [r19]
E180428 06:03:08.683688 97022 storage/queue.go:778 [replicate,s1,r17/1:/Table/2{0-1}] no removable replicas from range that needs a removal: [1*:14]
I180428 06:03:08.687497 97030 storage/replica_command.go:862 [split,s1,r19/1:/{Table/22-Max}] initiating a split of this range at key /Table/23 [r20]
E180428 06:03:08.687916 97046 storage/queue.go:778 [replicate,s1,r18/1:/Table/2{1-2}] no removable replicas from range that needs a removal: [1*:14]
E180428 06:03:08.690249 97047 storage/queue.go:778 [replicate,s1,r19/1:/Table/2{2-3}] no removable replicas from range that needs a removal: [1*:14]
E180428 06:03:08.690498 97033 storage/queue.go:778 [replicate,s1,r20/1:/{Table/23-Max}] no removable replicas from range that needs a removal: [1*:11]
I180428 06:03:08.691006 96867 storage/replica_command.go:862 [s1,r20/1:/{Table/23-Max}] initiating a split of this range at key /Table/50 [r21]
E180428 06:03:08.694360 97075 storage/queue.go:778 [replicate,s1,r20/1:/Table/{23-50}] no removable replicas from range that needs a removal: [1*:14]
E180428 06:03:08.694463 97076 storage/queue.go:778 [replicate,s1,r21/1:/{Table/50-Max}] no removable replicas from range that needs a removal: [1*:11]
I180428 06:03:08.808657 96831 storage/replica_command.go:862 [split,s1,r21/1:/{Table/50-Max}] initiating a split of this range at key /Table/50/"MJoCeJamymgBLZmoqFENrdUGORsKxiLbkDPKcphWdsymNQCJotMVPNjFRDVvoNTVfuyLxxmgVRmraONHxfIZpXaFJXoUacsRzUdk" [r22]
I180428 06:03:08.910123 96830 storage/replica_command.go:862 [split,s1,r21/1:/{Table/50-Max}] initiating a split of this range at key /Table/50/"MJoCeJamymgBLZmoqFENrdUGORsKxiLbkDPKcphWdsymNQCJotMVPNjFRDVvoNTVfuyLxxmgVRmraONHxfIZpXaFJXoUacsRzUdk" [r23]
panic: 1 leaked iterators [recovered]
panic: 1 leaked iterators [recovered]
panic: 1 leaked iterators
goroutine 96867 [running]:
testing.tRunner.func1(0xc421697590)
/usr/local/go/src/testing/testing.go:742 +0x29d
panic(0x2056b80, 0xc4202a2600)
/usr/local/go/src/runtime/panic.go:505 +0x229
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Recover(0xc422d9d9e0, 0x2655040, 0xc4200ca058)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:178 +0x11f
panic(0x2056b80, 0xc4202a2600)
/usr/local/go/src/runtime/panic.go:505 +0x229
github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).Close(0xc422a1b1e0)
/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:709 +0x32b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Stop(0xc422d9d9e0, 0x2655040, 0xc4200ca058)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:422 +0x29d
github.com/cockroachdb/cockroach/pkg/storage_test.TestStoreRangeSplitBackpressureWrites.func1(0xc421697590)
/go/src/github.com/cockroachdb/cockroach/pkg/storage/client_split_test.go:1039 +0x780
testing.tRunner(0xc421697590, 0xc4203f25b0)
/usr/local/go/src/testing/testing.go:777 +0xd0
created by testing.(*T).Run
/usr/local/go/src/testing/testing.go:824 +0x2e0
goroutine 1 [chan receive]:
testing.(*T).Run(0xc4251132c0, 0x2311439, 0x25, 0x2400590, 0x83fd01)
/usr/local/go/src/testing/testing.go:825 +0x301
testing.runTests.func1(0xc420236960)
/usr/local/go/src/testing/testing.go:1063 +0x64
testing.tRunner(0xc420236960, 0xc4206e5b90)
/usr/local/go/src/testing/testing.go:777 +0xd0
testing.runTests(0xc420524300, 0x341f640, 0x1f2, 0x1f2, 0x20b2e80)
/usr/local/go/src/testing/testing.go:1061 +0x2c4
testing.(*M).Run(0xc4204d0100, 0x0)
/usr/local/go/src/testing/testing.go:978 +0x171
github.com/cockroachdb/cockroach/pkg/storage_test.TestMain(0xc4204d0100)
/go/src/github.com/cockroachdb/cockroach/pkg/storage/main_test.go:57 +0x1e3
main.main()
_testmain.go:1052 +0x151
goroutine 20 [syscall, 1 minutes]:
os/signal.signal_recv(0x0)
/usr/local/go/src/runtime/sigqueue.go:139 +0xa6
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.0
/usr/local/go/src/os/signal/signal_unix.go:28 +0x41
goroutine 50 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.flushDaemon()
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1158 +0xf1
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:591 +0x110
goroutine 51 [chan receive, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.signalFlusher()
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:598 +0xab
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:592 +0x128
goroutine 22 [select, 1 minutes, locked to thread]:
runtime.gopark(0x24026b0, 0x0, 0x22d27e6, 0x6, 0x18, 0x1)
/usr/local/go/src/runtime/proc.go:291 +0x11a
runtime.selectgo(0xc4204a7f50, 0xc4200b42a0)
/usr/local/go/src/runtime/select.go:392 +0xe50
runtime.ensureSigM.func1()
/usr/local/go/src/runtime/signal_unix.go:549 +0x1f4
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:2361 +0x1
goroutine 87887 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4235c9680)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1210 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:82 +0x2ab
goroutine 88496 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc423042a20)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1210 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:82 +0x2ab
goroutine 90147 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421dec000)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1210 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:82 +0x2ab
goroutine 39752 [chan receive, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42064ad80)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1210 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:82 +0x2ab
goroutine 96014 [chan receive]:
testing.(*T).Run(0xc421697590, 0xc424fe8a60, 0x12, 0xc4203f25b0, 0x1)
/usr/local/go/src/testing/testing.go:825 +0x301
github.com/cockroachdb/cockroach/pkg/storage_test.TestStoreRangeSplitBackpressureWrites(0xc4251132c0)
/go/src/github.com/cockroachdb/cockroach/pkg/storage/client_split_test.go:951 +0x295
testing.tRunner(0xc4251132c0, 0x2400590)
/usr/local/go/src/testing/testing.go:777 +0xd0
created by testing.(*T).Run
/usr/local/go/src/testing/testing.go:824 +0x2e0
goroutine 90894 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4235c9560)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1210 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:82 +0x2ab
goroutine 90895 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4235c98c0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1210 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:82 +0x2ab
goroutine 39751 [chan receive, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42064ac60)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1210 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:82 +0x2ab
goroutine 89636 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4202225a0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1210 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:82 +0x2ab
goroutine 87648 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc420f8fc20)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1210 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:82 +0x2ab
goroutine 88497 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc423042b40)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1210 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:82 +0x2ab
goroutine 90619 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc420dc38c0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1210 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:82 +0x2ab
goroutine 96830 [runnable]:
syscall.Syscall(0x13e, 0xc4200ca080, 0x10, 0x0, 0x10, 0x0, 0x0)
/usr/local/go/src/syscall/asm_linux_amd64.s:18 +0x5
internal/syscall/unix.GetRandom(0xc4200ca080, 0x10, 0x10, 0x0, 0x7fffffffffffffff, 0xed276059c, 0xc4215ad0b8)
/usr/local/go/src/internal/syscall/unix/getrandom_linux.go:38 +0x6d
crypto/rand.getRandomLinux(0xc4200ca080, 0x10, 0x10, 0x1c)
/usr/local/go/src/crypto/rand/rand_linux.go:46 +0x48
crypto/rand.batched.func1(0xc4200ca080, 0x10, 0x10, 0x7e2)
/usr/local/go/src/crypto/rand/rand_linux.go:34 +0xf1
crypto/rand.(*devReader).Read(0xc4200c22a0, 0xc4200ca080, 0x10, 0x10, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/rand/rand_unix.go:48 +0x3b8
io.ReadAtLeast(0x2631480, 0xc4200c22a0, 0xc4200ca080, 0x10, 0x10, 0x10, 0x7f38221d4858, 0xc4215ad201, 0x1010000007e5ea9)
/usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x2631480, 0xc4200c22a0, 0xc4200ca080, 0x10, 0x10, 0x0, 0x3, 0xc42005c570)
/usr/local/go/src/io/io.go:327 +0x58
crypto/rand.Read(0xc4200ca080, 0x10, 0x10, 0x21a5820, 0xc4215ad301, 0xc4200ca080)
/usr/local/go/src/crypto/rand/rand.go:23 +0x57
github.com/cockroachdb/cockroach/vendor/github.com/satori/go%2euuid.(*generator).safeRandom(0xc4200c62d0, 0xc4200ca080, 0x10, 0x10)
/go/src/github.com/cockroachdb/cockroach/vendor/github.com/satori/go.uuid/generator.go:200 +0x3f
github.com/cockroachdb/cockroach/vendor/github.com/satori/go%2euuid.(*generator).NewV4(0xc4200c62d0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/vendor/github.com/satori/go.uuid/generator.go:153 +0x63
github.com/cockroachdb/cockroach/vendor/github.com/satori/go%2euuid.NewV4(0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/vendor/github.com/satori/go.uuid/generator.go:65 +0x3d
github.com/cockroachdb/cockroach/pkg/util/uuid.MakeV4(0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/uuid/uuid.go:101 +0x2a
github.com/cockroachdb/cockroach/pkg/roachpb.MakeTransaction(0x22d4b8f, 0x7, 0x0, 0x0, 0x0, 0x3ff0000000000000, 0x0, 0x15298446e79ebb4c, 0xc400000000, 0x1, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/roachpb/data.go:712 +0x55
github.com/cockroachdb/cockroach/pkg/internal/client.NewTxn(0xc420d515f0, 0xc400000000, 0x1, 0xc4257fa000)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:116 +0xbd
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Txn(0xc420d515f0, 0x2655000, 0xc4257fa000, 0xc420df6000, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:509 +0x9d
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).adminSplitWithDescriptor(0xc423090380, 0x2655080, 0xc4229749c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_command.go:865 +0x860
github.com/cockroachdb/cockroach/pkg/storage.(*splitQueue).processAttempt(0xc4203f2db0, 0x2655080, 0xc4229749c0, 0xc423090380, 0xc420b4d000, 0x22, 0x40, 0xd6c7f2, 0x3)
/go/src/github.com/cockroachdb/cockroach/pkg/storage/split_queue.go:137 +0x415
github.com/cockroachdb/cockroach/pkg/storage.(*splitQueue).process(0xc4203f2db0, 0x2655080, 0xc4229749c0, 0xc423090380, 0xc420b4d000, 0x22, 0x40, 0x1, 0xc4232bae70)
/go/src/github.com/cockroachdb/cockroach/pkg/storage/split_queue.go:94 +0x77
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processReplica(0xc42201dc00, 0x26550c0, 0xc421af50b0, 0xc423090380, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:723 +0x35d
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).DrainQueue(0xc42201dc00, 0xc422d9d9e0)
/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:953 +0xef
github.com/cockroachdb/cockroach/pkg/storage.forceScanAndProcess(0xc420a73680, 0xc42201dc00)
/go/src/github.com/cockroachdb/cockroach/pkg/storage/helpers_test.go:81 +0x9d
github.com/cockroachdb/cockroach/pkg/storage.(*Store).ForceSplitScanAndProcess(0xc420a73680)
/go/src/github.com/cockroachdb/cockroach/pkg/storage/helpers_test.go:99 +0x3a
github.com/cockroachdb/cockroach/pkg/storage_test.TestStoreRangeSplitBackpressureWrites.func1.2(0xc420a73680)
/go/src/github.com/cockroachdb/cockroach/pkg/storage/client_split_test.go:1004 +0x3e
created by github.com/cockroachdb/cockroach/pkg/storage_test.TestStoreRangeSplitBackpressureWrites.func1
/go/src/github.com/cockroachdb/cockroach/pkg/storage/client_split_test.go:1002 +0x59c
goroutine 90371 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc423043200)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1210 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:82 +0x2ab
goroutine 96868 [semacquire]:
sync.runtime_notifyListWait(0xc422a1b308, 0xc40000029f)
/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc422a1b2f8)
/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).syncLoop(0xc422a1b1e0)
/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:647 +0x9c
created by github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).open
/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:634 +0x439
goroutine 89637 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc420222b40)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1210 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:82 +0x2ab
goroutine 87649 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc420f8fe60)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1210 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:82 +0x2ab
goroutine 90370 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4230430e0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1210 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:82 +0x2ab
goroutine 88141 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc420daa6c0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1210 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:82 +0x2ab
goroutine 87175 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42126ac60)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1210 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:82 +0x2ab
goroutine 88140 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc420daa5a0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1210 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:82 +0x2ab
goroutine 87888 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4235c97a0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1210 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:82 +0x2ab
goroutine 90618 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc420dc37a0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1210 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:82 +0x2ab
goroutine 90146 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc422403e60)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1210 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:82 +0x2ab
goroutine 87176 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42126ad80)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1210 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:82 +0x2ab
ERROR: exit status 2
context canceled
make: *** [stress] Error 1
25 runs completed, 1 failures, over 7m45s
Makefile:867: recipe for target 'stress' failed
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
A-kv-replicationRelating to Raft, consensus, and coordination.Relating to Raft, consensus, and coordination.C-test-failureBroken test (automatically or manually discovered).Broken test (automatically or manually discovered).O-robotOriginated from a bot.Originated from a bot.