-
Notifications
You must be signed in to change notification settings - Fork 4.1k
storage: TestStoreRangeSplitBackpressureWrites/splitErr=result_is_ambiguous_() failed under stress #24974
Copy link
Copy link
Closed
Labels
A-kv-replicationRelating to Raft, consensus, and coordination.Relating to Raft, consensus, and coordination.C-test-failureBroken test (automatically or manually discovered).Broken test (automatically or manually discovered).O-robotOriginated from a bot.Originated from a bot.
Milestone
Description
SHA: https://github.com/cockroachdb/cockroach/commits/9da732ca6a06c12149213c97fd8215ca23067eba
Parameters:
TAGS=
GOFLAGS=
Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=617679&tab=buildLog
I180422 10:39:07.348791 96348 storage/store.go:1385 [s1] [n1,s1]: failed initial metrics computation: [n1,s1]: system config not yet available
I180422 10:39:07.350910 96457 storage/replica_command.go:862 [split,s1,r1/1:/M{in-ax}] initiating a split of this range at key /System/"" [r2]
E180422 10:39:07.353876 96359 storage/queue.go:778 [replicate,s1,r1/1:/{Min-System/}] no removable replicas from range that needs a removal: [1*:15]
I180422 10:39:07.353990 96266 storage/replica_command.go:862 [split,s1,r2/1:/{System/-Max}] initiating a split of this range at key /System/NodeLiveness [r3]
W180422 10:39:07.354785 96266 storage/stores.go:222 range not contained in one range: [/Meta2/System/NodeLiveness,/System/NodeLiveness/NULL), but have [/Min,/System/"")
W180422 10:39:07.356152 96467 storage/intent_resolver.go:320 [s1] failed to push during intent resolution: failed to push "split" id=49d4e0f4 key=/Local/Range/System/""/RangeDescriptor rw=true pri=0.02329685 iso=SERIALIZABLE stat=PENDING epo=0 ts=1524393547.354036162,0 orig=1524393547.354036162,0 max=1524393547.354036163,0 wto=false rop=false seq=2
I180422 10:39:07.358554 96470 storage/replica_command.go:862 [split,s1,r3/1:/{System/NodeL…-Max}] initiating a split of this range at key /System/NodeLivenessMax [r4]
E180422 10:39:07.358707 96445 storage/queue.go:778 [replicate,s1,r2/1:/System/{-NodeLive…}] no removable replicas from range that needs a removal: [1*:14]
I180422 10:39:07.362495 96482 storage/replica_command.go:862 [split,s1,r4/1:/{System/NodeL…-Max}] initiating a split of this range at key /System/tsd [r5]
E180422 10:39:07.362552 96446 storage/queue.go:778 [replicate,s1,r3/1:/System/NodeLiveness{-Max}] no removable replicas from range that needs a removal: [1*:15]
I180422 10:39:07.365427 96360 storage/replica_command.go:862 [split,s1,r5/1:/{System/tsd-Max}] initiating a split of this range at key /System/"tse" [r6]
E180422 10:39:07.365614 96448 storage/queue.go:778 [replicate,s1,r4/1:/System/{NodeLive…-tsd}] no removable replicas from range that needs a removal: [1*:14]
I180422 10:39:07.368813 96268 storage/replica_command.go:862 [split,s1,r6/1:/{System/tse-Max}] initiating a split of this range at key /Table/SystemConfigSpan/Start [r7]
E180422 10:39:07.368885 96476 storage/queue.go:778 [replicate,s1,r5/1:/System/ts{d-e}] no removable replicas from range that needs a removal: [1*:14]
I180422 10:39:07.371753 96272 storage/replica_command.go:862 [split,s1,r7/1:/{Table/System…-Max}] initiating a split of this range at key /Table/11 [r8]
E180422 10:39:07.374740 96515 storage/queue.go:778 [replicate,s1,r6/1:/{System/tse-Table/System…}] no removable replicas from range that needs a removal: [1*:14]
E180422 10:39:07.374834 96516 storage/queue.go:778 [replicate,s1,r7/1:/Table/{SystemCon…-11}] no removable replicas from range that needs a removal: [1*:14]
I180422 10:39:07.374978 96518 storage/replica_command.go:862 [split,s1,r8/1:/{Table/11-Max}] initiating a split of this range at key /Table/12 [r9]
I180422 10:39:07.378465 96530 storage/replica_command.go:862 [split,s1,r9/1:/{Table/12-Max}] initiating a split of this range at key /Table/13 [r10]
E180422 10:39:07.378560 96520 storage/queue.go:778 [replicate,s1,r8/1:/Table/1{1-2}] no removable replicas from range that needs a removal: [1*:14]
E180422 10:39:07.381332 96484 storage/queue.go:778 [replicate,s1,r9/1:/Table/1{2-3}] no removable replicas from range that needs a removal: [1*:14]
I180422 10:39:07.381468 96503 storage/replica_command.go:862 [split,s1,r10/1:/{Table/13-Max}] initiating a split of this range at key /Table/14 [r11]
E180422 10:39:07.384059 96504 storage/queue.go:778 [replicate,s1,r10/1:/Table/1{3-4}] no removable replicas from range that needs a removal: [1*:14]
I180422 10:39:07.384072 96533 storage/replica_command.go:862 [split,s1,r11/1:/{Table/14-Max}] initiating a split of this range at key /Table/15 [r12]
I180422 10:39:07.387498 96537 storage/replica_command.go:862 [split,s1,r12/1:/{Table/15-Max}] initiating a split of this range at key /Table/16 [r13]
E180422 10:39:07.388102 96479 storage/queue.go:778 [replicate,s1,r11/1:/Table/1{4-5}] no removable replicas from range that needs a removal: [1*:14]
I180422 10:39:07.390415 96480 storage/replica_command.go:862 [split,s1,r13/1:/{Table/16-Max}] initiating a split of this range at key /Table/17 [r14]
E180422 10:39:07.390564 96525 storage/queue.go:778 [replicate,s1,r12/1:/Table/1{5-6}] no removable replicas from range that needs a removal: [1*:14]
I180422 10:39:07.393218 96487 storage/replica_command.go:862 [split,s1,r14/1:/{Table/17-Max}] initiating a split of this range at key /Table/18 [r15]
E180422 10:39:07.393473 96528 storage/queue.go:778 [replicate,s1,r13/1:/Table/1{6-7}] no removable replicas from range that needs a removal: [1*:14]
E180422 10:39:07.396872 96562 storage/queue.go:778 [replicate,s1,r14/1:/Table/1{7-8}] no removable replicas from range that needs a removal: [1*:14]
I180422 10:39:07.396873 96362 storage/replica_command.go:862 [split,s1,r15/1:/{Table/18-Max}] initiating a split of this range at key /Table/19 [r16]
E180422 10:39:07.399382 96363 storage/queue.go:778 [replicate,s1,r15/1:/Table/1{8-9}] no removable replicas from range that needs a removal: [1*:14]
I180422 10:39:07.399451 96126 storage/replica_command.go:862 [split,s1,r16/1:/{Table/19-Max}] initiating a split of this range at key /Table/20 [r17]
W180422 10:39:07.400332 96126 storage/stores.go:222 range not contained in one range: [/Meta2/Table/20,/Table/20/NULL), but have [/Min,/System/"")
I180422 10:39:07.402953 96549 storage/replica_command.go:862 [split,s1,r17/1:/{Table/20-Max}] initiating a split of this range at key /Table/21 [r18]
E180422 10:39:07.403001 96548 storage/queue.go:778 [replicate,s1,r16/1:/Table/{19-20}] no removable replicas from range that needs a removal: [1*:14]
E180422 10:39:07.405814 96570 storage/queue.go:778 [replicate,s1,r17/1:/Table/2{0-1}] no removable replicas from range that needs a removal: [1*:14]
I180422 10:39:07.405884 96128 storage/replica_command.go:862 [split,s1,r18/1:/{Table/21-Max}] initiating a split of this range at key /Table/22 [r19]
E180422 10:39:07.408905 96364 storage/queue.go:778 [replicate,s1,r18/1:/Table/2{1-2}] no removable replicas from range that needs a removal: [1*:14]
I180422 10:39:07.408989 96552 storage/replica_command.go:862 [split,s1,r19/1:/{Table/22-Max}] initiating a split of this range at key /Table/23 [r20]
E180422 10:39:07.412094 96492 storage/queue.go:778 [replicate,s1,r19/1:/Table/2{2-3}] no removable replicas from range that needs a removal: [1*:14]
E180422 10:39:07.412270 96572 storage/queue.go:778 [replicate,s1,r20/1:/{Table/23-Max}] no removable replicas from range that needs a removal: [1*:11]
I180422 10:39:07.412988 96348 storage/replica_command.go:862 [s1,r20/1:/{Table/23-Max}] initiating a split of this range at key /Table/50 [r21]
E180422 10:39:07.415811 96573 storage/queue.go:778 [replicate,s1,r20/1:/Table/{23-50}] no removable replicas from range that needs a removal: [1*:14]
E180422 10:39:07.415960 96574 storage/queue.go:778 [replicate,s1,r21/1:/{Table/50-Max}] no removable replicas from range that needs a removal: [1*:11]
I180422 10:39:07.531269 96509 storage/replica_command.go:862 [split,s1,r21/1:/{Table/50-Max}] initiating a split of this range at key /Table/50/"MJoCeJamymgBLZmoqFENrdUGORsKxiLbkDPKcphWdsymNQCJotMVPNjFRDVvoNTVfuyLxxmgVRmraONHxfIZpXaFJXoUacsRzUdk" [r22]
W180422 10:39:07.531490 96510 storage/replica_backpressure.go:135 [s1,r21/1:/{Table/50-Max}] applying backpressure to limit range growth on batch Put [/Table/50,/Min)
I180422 10:39:07.631979 96508 storage/replica_command.go:862 [split,s1,r21/1:/{Table/50-Max}] initiating a split of this range at key /Table/50/"MJoCeJamymgBLZmoqFENrdUGORsKxiLbkDPKcphWdsymNQCJotMVPNjFRDVvoNTVfuyLxxmgVRmraONHxfIZpXaFJXoUacsRzUdk" [r23]
I180422 10:39:07.632349 96348 util/stop/stopper.go:471 quiescing; tasks left:
1 [async] kv.TxnCoordSender: aborting txn
panic: 1 leaked iterators [recovered]
panic: 1 leaked iterators [recovered]
panic: 1 leaked iterators
goroutine 96348 [running]:
testing.tRunner.func1(0xc421d621e0)
/usr/local/go/src/testing/testing.go:742 +0x29d
panic(0x204d8c0, 0xc423276ab0)
/usr/local/go/src/runtime/panic.go:505 +0x229
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Recover(0xc4252ca990, 0x2649ca0, 0xc42003e138)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:178 +0x11f
panic(0x204d8c0, 0xc423276ab0)
/usr/local/go/src/runtime/panic.go:505 +0x229
github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).Close(0xc4204a2820)
/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:709 +0x32b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Stop(0xc4252ca990, 0x2649ca0, 0xc42003e138)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:422 +0x29d
github.com/cockroachdb/cockroach/pkg/storage_test.TestStoreRangeSplitBackpressureWrites.func1(0xc421d621e0)
/go/src/github.com/cockroachdb/cockroach/pkg/storage/client_split_test.go:1039 +0x780
testing.tRunner(0xc421d621e0, 0xc423fb0c30)
/usr/local/go/src/testing/testing.go:777 +0xd0
created by testing.(*T).Run
/usr/local/go/src/testing/testing.go:824 +0x2e0
goroutine 1 [chan receive]:
testing.(*T).Run(0xc420213860, 0x230751f, 0x25, 0x23f6338, 0x83fd01)
/usr/local/go/src/testing/testing.go:825 +0x301
testing.runTests.func1(0xc420212a50)
/usr/local/go/src/testing/testing.go:1063 +0x64
testing.tRunner(0xc420212a50, 0xc42001fb90)
/usr/local/go/src/testing/testing.go:777 +0xd0
testing.runTests(0xc4204cc300, 0x340f6a0, 0x1f2, 0x1f2, 0x20a9900)
/usr/local/go/src/testing/testing.go:1061 +0x2c4
testing.(*M).Run(0xc42046e180, 0x0)
/usr/local/go/src/testing/testing.go:978 +0x171
github.com/cockroachdb/cockroach/pkg/storage_test.TestMain(0xc42046e180)
/go/src/github.com/cockroachdb/cockroach/pkg/storage/main_test.go:57 +0x1e3
main.main()
_testmain.go:1052 +0x151
goroutine 6 [syscall, 1 minutes]:
os/signal.signal_recv(0x0)
/usr/local/go/src/runtime/sigqueue.go:139 +0xa6
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.0
/usr/local/go/src/os/signal/signal_unix.go:28 +0x41
goroutine 13 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.flushDaemon()
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1128 +0xf1
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:591 +0x110
goroutine 14 [chan receive, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.signalFlusher()
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:598 +0xab
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:592 +0x128
goroutine 19 [select, 1 minutes, locked to thread]:
runtime.gopark(0x23f8468, 0x0, 0x22c8946, 0x6, 0x18, 0x1)
/usr/local/go/src/runtime/proc.go:291 +0x11a
runtime.selectgo(0xc420068f50, 0xc4205ea240)
/usr/local/go/src/runtime/select.go:392 +0xe50
runtime.ensureSigM.func1()
/usr/local/go/src/runtime/signal_unix.go:549 +0x1f4
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:2361 +0x1
goroutine 95899 [chan receive]:
testing.(*T).Run(0xc421d621e0, 0xc4202e46c0, 0x1f, 0xc423fb0c30, 0x1)
/usr/local/go/src/testing/testing.go:825 +0x301
github.com/cockroachdb/cockroach/pkg/storage_test.TestStoreRangeSplitBackpressureWrites(0xc420213860)
/go/src/github.com/cockroachdb/cockroach/pkg/storage/client_split_test.go:951 +0x295
testing.tRunner(0xc420213860, 0x23f6338)
/usr/local/go/src/testing/testing.go:777 +0xd0
created by testing.(*T).Run
/usr/local/go/src/testing/testing.go:824 +0x2e0
goroutine 88111 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc423a3f440)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1180 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x2ab
goroutine 87648 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc422327c20)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1180 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x2ab
goroutine 39748 [chan receive, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4201f6900)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1180 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x2ab
goroutine 96349 [semacquire]:
sync.runtime_notifyListWait(0xc4204a2948, 0xc40000029f)
/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc4204a2938)
/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).syncLoop(0xc4204a2820)
/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:647 +0x9c
created by github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).open
/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:634 +0x439
goroutine 39749 [chan receive, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4201f6a20)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1180 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x2ab
goroutine 90172 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42411d320)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1180 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x2ab
goroutine 87161 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421bd5200)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1180 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x2ab
goroutine 90930 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4236e4360)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1180 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x2ab
goroutine 90171 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42411d200)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1180 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x2ab
goroutine 90375 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4222279e0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1180 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x2ab
goroutine 90929 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4236e4240)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1180 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x2ab
goroutine 88112 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc423a3f560)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1180 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x2ab
goroutine 88499 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421d498c0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1180 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x2ab
goroutine 89656 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc423a3f680)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1180 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x2ab
goroutine 88500 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4238b66c0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1180 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x2ab
goroutine 89657 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc423a3f7a0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1180 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x2ab
goroutine 87893 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc422a17c20)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1180 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x2ab
goroutine 90645 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc422a16b40)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1180 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x2ab
goroutine 96508 [runnable]:
github.com/cockroachdb/cockroach/pkg/roachpb.(*Error).setGoError(0xc4210c3040, 0x26264e0, 0xc420381a40)
/go/src/github.com/cockroachdb/cockroach/pkg/roachpb/errors.go:151 +0xd6
github.com/cockroachdb/cockroach/pkg/roachpb.NewError(0x26264e0, 0xc420381a40, 0x2)
/go/src/github.com/cockroachdb/cockroach/pkg/roachpb/errors.go:70 +0xda
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).tryExecuteWriteBatch(0xc4258a2e00, 0x2649d20, 0xc42264b7a0, 0x1527bbdae080bb00, 0x0, 0x100000001, 0x1, 0x15, 0x0, 0xc42236e100, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2880 +0xaaf
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).executeWriteBatch(0xc4258a2e00, 0x2649d20, 0xc42264b7a0, 0x1527bbdae080bb00, 0x0, 0x100000001, 0x1, 0x15, 0x0, 0xc42236e100, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2652 +0xae
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).Send(0xc4258a2e00, 0x2649d20, 0xc42264b7a0, 0x1527bbdae080bb00, 0x0, 0x100000001, 0x1, 0x15, 0x0, 0xc42236e100, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:1891 +0x69d
github.com/cockroachdb/cockroach/pkg/storage.(*Store).Send(0xc4228de000, 0x2649d20, 0xc42264b740, 0x1527bbdae080bb00, 0x0, 0x100000001, 0x1, 0x15, 0x0, 0xc42236e100, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:2811 +0x623
github.com/cockroachdb/cockroach/pkg/storage.(*Stores).Send(0xc42265dce0, 0x2649d20, 0xc42264b5c0, 0x0, 0x0, 0x100000001, 0x1, 0x15, 0x0, 0xc42236e000, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/storage/stores.go:192 +0x1fc
github.com/cockroachdb/cockroach/pkg/kv.(*senderTransport).SendNext(0xc422756820, 0x2649d20, 0xc42264b5c0, 0xc4222bc000)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:411 +0x1c9
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendToReplicas(0xc421d62ff0, 0x2649d20, 0xc42264b4a0, 0xc421d63040, 0x10, 0xc420381720, 0x1, 0x1, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1292 +0x386
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendRPC(0xc421d62ff0, 0x2649d20, 0xc42264b4a0, 0x10, 0xc420381720, 0x1, 0x1, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:396 +0x242
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendSingleRange(0xc421d62ff0, 0x2649d20, 0xc42264b4a0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc42236e000, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:463 +0x21a
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendPartialBatch(0xc421d62ff0, 0x2649d20, 0xc42264b4a0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc42236e000, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1070 +0x325
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).divideAndSendBatchToRanges(0xc421d62ff0, 0x2649d20, 0xc42264b4a0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc42236e000, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:742 +0x15e6
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).Send(0xc421d62ff0, 0x2649d20, 0xc42264b4a0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc42230ff00, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:658 +0x415
github.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSender).Send(0xc4260b2c40, 0x2649d20, 0xc42264b380, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc42230ff00, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:490 +0x1ec
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).sendUsingSender(0xc426ee0150, 0x2649c60, 0xc421c87640, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc42230ff00, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:553 +0x118
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Send(0xc4206f5040, 0x2649c60, 0xc421c87640, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:959 +0x3a8
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Send-fm(0x2649c60, 0xc421c87640, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:505 +0x76
github.com/cockroachdb/cockroach/pkg/internal/client.sendAndFill(0x2649c60, 0xc421c87640, 0xc4210f5340, 0xc420a7dc00, 0x17, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:468 +0xf8
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Run(0xc4206f5040, 0x2649c60, 0xc421c87640, 0xc420a7dc00, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:505 +0xc7
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).adminSplitWithDescriptor.func1(0x2649c60, 0xc421c87640, 0xc4206f5040, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_command.go:892 +0x27b
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Txn.func1(0x2649c60, 0xc421c87640, 0xc4206f5040, 0xc42030bd38, 0x1527bbdae080bb01, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:516 +0x43
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Exec(0xc4206f5040, 0x2649c60, 0xc421c87640, 0xc4206f0101, 0xc42332a130, 0x1, 0xc4210f57a8)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:753 +0x10c
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Txn(0xc426ee0150, 0x2649c60, 0xc421c87640, 0xc42264b260, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:515 +0x133
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).adminSplitWithDescriptor(0xc4258a2e00, 0x2649ce0, 0xc421ef9d40, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_command.go:865 +0x860
github.com/cockroachdb/cockroach/pkg/storage.(*splitQueue).processAttempt(0xc423fb1140, 0x2649ce0, 0xc421ef9d40, 0xc4258a2e00, 0xc422c63000, 0x22, 0x40, 0xd6c682, 0x3)
/go/src/github.com/cockroachdb/cockroach/pkg/storage/split_queue.go:137 +0x415
github.com/cockroachdb/cockroach/pkg/storage.(*splitQueue).process(0xc423fb1140, 0x2649ce0, 0xc421ef9d40, 0xc4258a2e00, 0xc422c63000, 0x22, 0x40, 0x1, 0xc4245eaa00)
/go/src/github.com/cockroachdb/cockroach/pkg/storage/split_queue.go:94 +0x77
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processReplica(0xc425394e00, 0x2649d20, 0xc423c66a20, 0xc4258a2e00, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:723 +0x35d
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).DrainQueue(0xc425394e00, 0xc4252ca990)
/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:953 +0xef
github.com/cockroachdb/cockroach/pkg/storage.forceScanAndProcess(0xc4228de000, 0xc425394e00)
/go/src/github.com/cockroachdb/cockroach/pkg/storage/helpers_test.go:81 +0x9d
github.com/cockroachdb/cockroach/pkg/storage.(*Store).ForceSplitScanAndProcess(0xc4228de000)
/go/src/github.com/cockroachdb/cockroach/pkg/storage/helpers_test.go:99 +0x3a
github.com/cockroachdb/cockroach/pkg/storage_test.TestStoreRangeSplitBackpressureWrites.func1.2(0xc4228de000)
/go/src/github.com/cockroachdb/cockroach/pkg/storage/client_split_test.go:1004 +0x3e
created by github.com/cockroachdb/cockroach/pkg/storage_test.TestStoreRangeSplitBackpressureWrites.func1
/go/src/github.com/cockroachdb/cockroach/pkg/storage/client_split_test.go:1002 +0x59c
goroutine 90646 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc422a16c60)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1180 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x2ab
goroutine 87894 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc422a17d40)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1180 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x2ab
goroutine 87162 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421bd5320)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1180 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x2ab
goroutine 90376 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc422227b00)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1180 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x2ab
goroutine 87649 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc422327d40)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1180 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x2ab
ERROR: exit status 2
context canceled
make: *** [stress] Error 1
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
A-kv-replicationRelating to Raft, consensus, and coordination.Relating to Raft, consensus, and coordination.C-test-failureBroken test (automatically or manually discovered).Broken test (automatically or manually discovered).O-robotOriginated from a bot.Originated from a bot.