Skip to content

teamcity: failed tests on master: test/TestSplitSnapshotRace_SnapshotWins #16251

@cockroach-teamcity

Description

@cockroach-teamcity

The following tests appear to have failed:

#261589:

--- FAIL: test/TestSplitSnapshotRace_SnapshotWins (0.000s)
Test ended in panic.

------- Stdout: -------
I170531 22:03:39.031246 61568 storage/store.go:1269  [n1,s1]: failed initial metrics computation: [n1,s1]: system config not yet available
I170531 22:03:39.031312 61568 gossip/gossip.go:297  [n1] NodeDescriptor set to node_id:1 address:<network_field:"tcp" address_field:"127.0.0.1:38914" > attrs:<> locality:<> 
W170531 22:03:39.035270 61568 gossip/gossip.go:1196  [n2] no incoming or outgoing connections
I170531 22:03:39.035366 61568 storage/store.go:1269  [n2,s2]: failed initial metrics computation: [n2,s2]: system config not yet available
I170531 22:03:39.035402 61568 gossip/gossip.go:297  [n2] NodeDescriptor set to node_id:2 address:<network_field:"tcp" address_field:"127.0.0.1:34844" > attrs:<> locality:<> 
I170531 22:03:39.035585 62371 gossip/client.go:131  [n2] started gossip client to 127.0.0.1:38914
W170531 22:03:39.054995 61568 gossip/gossip.go:1196  [n3] no incoming or outgoing connections
I170531 22:03:39.055349 61568 storage/store.go:1269  [n3,s3]: failed initial metrics computation: [n3,s3]: system config not yet available
I170531 22:03:39.055386 61568 gossip/gossip.go:297  [n3] NodeDescriptor set to node_id:3 address:<network_field:"tcp" address_field:"127.0.0.1:48267" > attrs:<> locality:<> 
I170531 22:03:39.055847 62388 gossip/client.go:131  [n3] started gossip client to 127.0.0.1:38914
W170531 22:03:39.075927 61568 gossip/gossip.go:1196  [n4] no incoming or outgoing connections
I170531 22:03:39.076099 61568 storage/store.go:1269  [n4,s4]: failed initial metrics computation: [n4,s4]: system config not yet available
I170531 22:03:39.076142 62612 gossip/client.go:131  [n4] started gossip client to 127.0.0.1:38914
I170531 22:03:39.076207 61568 gossip/gossip.go:297  [n4] NodeDescriptor set to node_id:4 address:<network_field:"tcp" address_field:"127.0.0.1:40581" > attrs:<> locality:<> 
W170531 22:03:39.078599 61568 gossip/gossip.go:1196  [n5] no incoming or outgoing connections
I170531 22:03:39.078723 61568 storage/store.go:1269  [n5,s5]: failed initial metrics computation: [n5,s5]: system config not yet available
I170531 22:03:39.078755 61568 gossip/gossip.go:297  [n5] NodeDescriptor set to node_id:5 address:<network_field:"tcp" address_field:"127.0.0.1:49130" > attrs:<> locality:<> 
I170531 22:03:39.078918 62758 gossip/client.go:131  [n5] started gossip client to 127.0.0.1:38914
I170531 22:03:39.079074 62618 gossip/server.go:285  [n1] refusing gossip from node 5 (max 3 conns); forwarding to 2 ({tcp 127.0.0.1:34844})
I170531 22:03:39.079386 62758 gossip/client.go:136  [n5] closing client to node 1 (127.0.0.1:38914): received forward from node 1 to 2 (127.0.0.1:34844)
I170531 22:03:39.079427 62775 gossip/gossip.go:1210  [n5] node has connected to cluster via gossip
I170531 22:03:39.079523 62873 gossip/client.go:131  [n5] started gossip client to 127.0.0.1:34844
W170531 22:03:39.092822 61568 gossip/gossip.go:1196  [n6] no incoming or outgoing connections
I170531 22:03:39.092953 61568 storage/store.go:1269  [n6,s6]: failed initial metrics computation: [n6,s6]: system config not yet available
I170531 22:03:39.092994 61568 gossip/gossip.go:297  [n6] NodeDescriptor set to node_id:6 address:<network_field:"tcp" address_field:"127.0.0.1:45044" > attrs:<> locality:<> 
I170531 22:03:39.093119 62910 gossip/client.go:131  [n6] started gossip client to 127.0.0.1:38914
I170531 22:03:39.093301 62913 gossip/server.go:285  [n1] refusing gossip from node 6 (max 3 conns); forwarding to 4 ({tcp 127.0.0.1:40581})
I170531 22:03:39.093451 62910 gossip/client.go:136  [n6] closing client to node 1 (127.0.0.1:38914): received forward from node 1 to 4 (127.0.0.1:40581)
I170531 22:03:39.093525 62915 gossip/gossip.go:1210  [n6] node has connected to cluster via gossip
I170531 22:03:39.093610 63038 gossip/client.go:131  [n6] started gossip client to 127.0.0.1:40581
I170531 22:03:39.095649 61568 storage/replica_command.go:2663  [s1,r1/1:/M{in-ax}] initiating a split of this range at key /System/Max [r2]
I170531 22:03:39.098202 63044 storage/replica_raftstorage.go:416  [s1,r2/1:/{System/Max-Max}] generated preemptive snapshot 0db9d70a at index 11
I170531 22:03:39.098664 63044 storage/store.go:3370  [s1,r2/1:/{System/Max-Max}] streamed snapshot to (n2,s2):?: kv pairs: 29, log entries: 1, rate-limit: 2.0 MiB/sec, 0ms
I170531 22:03:39.098828 63076 storage/replica_raftstorage.go:597  [s2,r2/?:{-}] applying preemptive snapshot at index 11 (id=0db9d70a, encoded size=3585, 1 rocksdb batches, 1 log entries)
I170531 22:03:39.099045 63076 storage/replica_raftstorage.go:605  [s2,r2/?:/{System/Max-Max}] applied preemptive snapshot in 0ms [clear=0ms batch=0ms entries=0ms commit=0ms]
I170531 22:03:39.099354 63044 storage/replica_command.go:3568  [s1,r2/1:/{System/Max-Max}] change replicas (ADD_REPLICA (n2,s2):2): read existing descriptor r2:/{System/Max-Max} [(n1,s1):1, next=2]
W170531 22:03:39.099944 63002 storage/stores.go:218  range not contained in one range: [/Meta2/Max,"\x05\x00"), but have [/Min,/System/Max)
I170531 22:03:39.100783 63006 storage/replica.go:2865  [s1,r2/1:/{System/Max-Max}] proposing ADD_REPLICA (n2,s2):2: [(n1,s1):1 (n2,s2):2]
I170531 22:03:39.101378 63008 storage/replica_raftstorage.go:416  [s1,r2/1:/{System/Max-Max}] generated preemptive snapshot 6d067a23 at index 13
I170531 22:03:39.101847 63008 storage/store.go:3370  [s1,r2/1:/{System/Max-Max}] streamed snapshot to (n3,s3):?: kv pairs: 31, log entries: 3, rate-limit: 2.0 MiB/sec, 0ms
I170531 22:03:39.102011 63078 storage/replica_raftstorage.go:597  [s3,r2/?:{-}] applying preemptive snapshot at index 13 (id=6d067a23, encoded size=5100, 1 rocksdb batches, 3 log entries)
I170531 22:03:39.102269 63078 storage/replica_raftstorage.go:605  [s3,r2/?:/{System/Max-Max}] applied preemptive snapshot in 0ms [clear=0ms batch=0ms entries=0ms commit=0ms]
I170531 22:03:39.102596 63008 storage/replica_command.go:3568  [s1,r2/1:/{System/Max-Max}] change replicas (ADD_REPLICA (n3,s3):3): read existing descriptor r2:/{System/Max-Max} [(n1,s1):1, (n2,s2):2, next=3]
I170531 22:03:39.103435 63081 storage/raft_transport.go:436  raft transport stream to node 1 established
I170531 22:03:39.106134 63093 storage/replica.go:2865  [s1,r2/1:/{System/Max-Max}] proposing ADD_REPLICA (n3,s3):3: [(n1,s1):1 (n2,s2):2 (n3,s3):3]
I170531 22:03:39.107295 63095 storage/replica_raftstorage.go:416  [s1,r2/1:/{System/Max-Max}] generated preemptive snapshot 07c91ffc at index 16
I170531 22:03:39.107714 63095 storage/store.go:3370  [s1,r2/1:/{System/Max-Max}] streamed snapshot to (n4,s4):?: kv pairs: 33, log entries: 6, rate-limit: 2.0 MiB/sec, 0ms
I170531 22:03:39.107873 63024 storage/replica_raftstorage.go:597  [s4,r2/?:{-}] applying preemptive snapshot at index 16 (id=07c91ffc, encoded size=7015, 1 rocksdb batches, 6 log entries)
I170531 22:03:39.108215 63024 storage/replica_raftstorage.go:605  [s4,r2/?:/{System/Max-Max}] applied preemptive snapshot in 0ms [clear=0ms batch=0ms entries=0ms commit=0ms]
I170531 22:03:39.108550 63095 storage/replica_command.go:3568  [s1,r2/1:/{System/Max-Max}] change replicas (ADD_REPLICA (n4,s4):4): read existing descriptor r2:/{System/Max-Max} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4]
I170531 22:03:39.111505 63057 storage/replica.go:2865  [s1,r2/1:/{System/Max-Max}] proposing ADD_REPLICA (n4,s4):4: [(n1,s1):1 (n2,s2):2 (n3,s3):3 (n4,s4):4]
I170531 22:03:39.250532 62444 storage/replica_proposal.go:385  [s2,r2/2:/{System/Max-Max}] new range lease repl=(n2,s2):2 start=0.000000123,392 epo=1 pro=0.000000123,393 following repl=(n1,s1):1 start=0.000000000,0 exp=0.900000123,6 pro=0.000000123,7
I170531 22:03:39.270264 63113 storage/replica_command.go:3568  [s2,r2/2:/{System/Max-Max}] change replicas (REMOVE_REPLICA (n1,s1):1): read existing descriptor r2:/{System/Max-Max} [(n1,s1):1, (n2,s2):2, (n3,s3):3, (n4,s4):4, next=5]
I170531 22:03:39.273754 63171 storage/replica.go:2865  [s2,r2/2:/{System/Max-Max}] proposing REMOVE_REPLICA (n1,s1):1: [(n4,s4):4 (n2,s2):2 (n3,s3):3]
I170531 22:03:39.282010 63055 storage/store.go:3177  [s1,r2/1:/{System/Max-Max}] added to replica GC queue (peer suggestion)
I170531 22:03:39.578717 62254 storage/store.go:2159  [replicaGC,s1,r2/1:/{System/Max-Max}] removing replica
I170531 22:03:39.578860 62254 storage/replica.go:714  [replicaGC,s1,r2/1:/{System/Max-Max}] removed 33 (21+12) keys in 0ms [clear=0ms commit=0ms]
I170531 22:03:39.816171 61568 storage/client_test.go:1177  test clock advanced to: 1.800000125,0
W170531 22:03:39.817954 62398 storage/raft_transport.go:257  unable to accept Raft message from (n2,s2):2: no handler registered for (n4,s4):4
W170531 22:03:39.818059 63122 storage/store.go:3181  [s2] raft error: node 4 claims to not contain store 4 for replica (n4,s4):4: store 4 was not found
W170531 22:03:39.818078 62396 storage/raft_transport.go:442  raft transport stream to node 4 failed: store 4 was not found
I170531 22:03:39.818371 62477 storage/replica_proposal.go:385  [s2,r2/2:/{System/Max-Max}] new range lease repl=(n2,s2):2 start=0.000000123,392 epo=1 pro=1.800000125,5 following repl=(n2,s2):2 start=0.000000123,392 epo=1 pro=0.000000123,393
I170531 22:03:39.818559 63145 storage/raft_transport.go:436  raft transport stream to node 4 established
W170531 22:03:39.818732 63234 storage/raft_transport.go:257  unable to accept Raft message from (n2,s2):2: no handler registered for (n4,s4):4
W170531 22:03:39.818841 63200 storage/store.go:3181  [s2] raft error: node 4 claims to not contain store 4 for replica (n4,s4):4: store 4 was not found
I170531 22:03:39.818990 63133 storage/replica_command.go:2663  [s2,r2/2:/{System/Max-Max}] initiating a split of this range at key "m" [r11]
I170531 22:03:39.819570 63149 storage/raft_transport.go:436  raft transport stream to node 4 established
W170531 22:03:39.819763 63251 storage/raft_transport.go:257  unable to accept Raft message from (n2,s2):2: no handler registered for (n4,s4):4
W170531 22:03:39.819877 63151 storage/store.go:3181  [s2] raft error: node 4 claims to not contain store 4 for replica (n4,s4):4: store 4 was not found
I170531 22:03:39.820204 63212 storage/raft_transport.go:436  raft transport stream to node 4 established
W170531 22:03:39.821093 63226 storage/raft_transport.go:257  unable to accept Raft message from (n2,s2):2: no handler registered for (n4,s4):4
W170531 22:03:39.821148 63226 storage/raft_transport.go:257  unable to accept Raft message from (n2,s2):2: no handler registered for (n4,s4):4
W170531 22:03:39.821219 63253 storage/store.go:3181  [s2] raft error: node 4 claims to not contain store 4 for replica (n4,s4):4: store 4 was not found
I170531 22:03:39.821851 63216 storage/raft_transport.go:436  raft transport stream to node 4 established
W170531 22:03:39.821973 63269 storage/raft_transport.go:257  unable to accept Raft message from (n2,s2):2: no handler registered for (n4,s4):4
W170531 22:03:39.822064 63267 storage/store.go:3181  [s2] raft error: node 4 claims to not contain store 4 for replica (n4,s4):4: store 4 was not found
I170531 22:03:39.822431 63254 storage/raft_transport.go:436  raft transport stream to node 4 established
W170531 22:03:39.822554 63230 storage/raft_transport.go:257  unable to accept Raft message from (n2,s2):2: no handler registered for (n4,s4):4
W170531 22:03:39.822848 63228 storage/store.go:3181  [s2] raft error: node 4 claims to not contain store 4 for replica (n4,s4):4: store 4 was not found
I170531 22:03:39.823467 62483 storage/replica_proposal.go:385  [s2,r2/2:/{System/Max-Max}] new range lease repl=(n2,s2):2 start=0.000000123,392 epo=1 pro=1.800000125,5 following repl=(n0,s0):? start=0.000000000,0 exp=0.000000000,0
I170531 22:03:39.827448 63152 storage/replica_raftstorage.go:416  [s2,r11/2:{m-/Max}] generated preemptive snapshot 85beb522 at index 10
I170531 22:03:39.828051 63152 storage/store.go:3370  [s2,r11/2:{m-/Max}] streamed snapshot to (n5,s5):?: kv pairs: 28, log entries: 0, rate-limit: 2.0 MiB/sec, 0ms
I170531 22:03:39.828413 63283 storage/replica_raftstorage.go:597  [s5,r11/?:{-}] applying preemptive snapshot at index 10 (id=85beb522, encoded size=3570, 1 rocksdb batches, 0 log entries)
I170531 22:03:39.828762 63283 storage/replica_raftstorage.go:605  [s5,r11/?:{m-/Max}] applied preemptive snapshot in 0ms [clear=0ms batch=0ms entries=0ms commit=0ms]
I170531 22:03:39.829120 63152 storage/replica_command.go:3568  [s2,r11/2:{m-/Max}] change replicas (ADD_REPLICA (n5,s5):4): read existing descriptor r11:{m-/Max} [(n4,s4):1, (n2,s2):2, (n3,s3):3, next=4]
I170531 22:03:39.829612 63257 storage/raft_transport.go:436  raft transport stream to node 4 established
W170531 22:03:39.829732 63261 storage/raft_transport.go:257  unable to accept Raft message from (n2,s2):2: no handler registered for (n4,s4):1
W170531 22:03:39.829823 63259 storage/store.go:3181  [s2] raft error: node 4 claims to not contain store 4 for replica (n4,s4):1: store 4 was not found
I170531 22:03:39.830481 63232 storage/raft_transport.go:436  raft transport stream to node 4 established
W170531 22:03:39.830587 63263 storage/raft_transport.go:257  unable to accept Raft message from (n2,s2):2: no handler registered for (n4,s4):1
W170531 22:03:39.830684 63298 storage/store.go:3181  [s2] raft error: node 4 claims to not contain store 4 for replica (n4,s4):1: store 4 was not found
I170531 22:03:39.832306 63302 storage/raft_transport.go:436  raft transport stream to node 4 established
W170531 22:03:39.832403 63265 storage/raft_transport.go:257  unable to accept Raft message from (n2,s2):2: no handler registered for (n4,s4):4
W170531 22:03:39.832533 63304 storage/store.go:3181  [s2] raft error: node 4 claims to not contain store 4 for replica (n4,s4):4: store 4 was not found
I170531 22:03:39.832897 63305 storage/raft_transport.go:436  raft transport stream to node 4 established
W170531 22:03:39.833239 63073 storage/raft_transport.go:257  unable to accept Raft message from (n2,s2):2: no handler registered for (n4,s4):4
W170531 22:03:39.833397 63285 storage/store.go:3181  [s2] raft error: node 4 claims to not contain store 4 for replica (n4,s4):4: store 4 was not found
I170531 22:03:39.833720 63307 storage/replica.go:2865  [s2,r11/2:{m-/Max}] proposing ADD_REPLICA (n5,s5):4: [(n4,s4):1 (n2,s2):2 (n3,s3):3 (n5,s5):4]
I170531 22:03:39.834740 63315 storage/replica_raftstorage.go:416  [s2,r11/2:{m-/Max}] generated preemptive snapshot e2cef84a at index 14
I170531 22:03:39.835233 63315 storage/store.go:3370  [s2,r11/2:{m-/Max}] streamed snapshot to (n6,s6):?: kv pairs: 30, log entries: 4, rate-limit: 2.0 MiB/sec, 0ms
I170531 22:03:39.835411 63331 storage/replica_raftstorage.go:597  [s6,r11/?:{-}] applying preemptive snapshot at index 14 (id=e2cef84a, encoded size=5910, 1 rocksdb batches, 4 log entries)
I170531 22:03:39.835696 63331 storage/replica_raftstorage.go:605  [s6,r11/?:{m-/Max}] applied preemptive snapshot in 0ms [clear=0ms batch=0ms entries=0ms commit=0ms]
I170531 22:03:39.836019 63315 storage/replica_command.go:3568  [s2,r11/2:{m-/Max}] change replicas (ADD_REPLICA (n6,s6):5): read existing descriptor r11:{m-/Max} [(n4,s4):1, (n2,s2):2, (n3,s3):3, (n5,s5):4, next=5]
I170531 22:03:39.840834 63348 storage/replica.go:2865  [s2,r11/2:{m-/Max}] proposing ADD_REPLICA (n6,s6):5: [(n4,s4):1 (n2,s2):2 (n3,s3):3 (n5,s5):4 (n6,s6):5]
I170531 22:03:39.885752 63270 storage/raft_transport.go:436  raft transport stream to node 4 established
W170531 22:03:39.885862 63272 storage/raft_transport.go:257  unable to accept Raft message from (n2,s2):?: no handler registered for (n4,s4):?
W170531 22:03:39.885988 63287 storage/store.go:3181  [s2] raft error: node 4 claims to not contain store 4 for replica (n4,s4):?: store 4 was not found
I170531 22:03:39.985794 63333 storage/raft_transport.go:436  raft transport stream to node 4 established
W170531 22:03:39.986023 63289 storage/raft_transport.go:257  unable to accept Raft message from (n2,s2):?: no handler registered for (n4,s4):?
W170531 22:03:39.986172 63337 storage/store.go:3181  [s2] raft error: node 4 claims to not contain store 4 for replica (n4,s4):?: store 4 was not found
I170531 22:03:40.085737 63320 storage/raft_transport.go:436  raft transport stream to node 4 established
W170531 22:03:40.085893 63158 storage/raft_transport.go:257  unable to accept Raft message from (n2,s2):?: no handler registered for (n4,s4):?
W170531 22:03:40.086054 63339 storage/store.go:3181  [s2] raft error: node 4 claims to not contain store 4 for replica (n4,s4):?: store 4 was not found
I170531 22:03:40.112543 63160 storage/replica_command.go:3568  [s2,r11/2:{m-/Max}] change replicas (REMOVE_REPLICA (n3,s3):3): read existing descriptor r11:{m-/Max} [(n4,s4):1, (n2,s2):2, (n3,s3):3, (n5,s5):4, (n6,s6):5, next=6]
I170531 22:03:40.115635 63164 storage/replica.go:2865  [s2,r11/2:{m-/Max}] proposing REMOVE_REPLICA (n3,s3):3: [(n4,s4):1 (n2,s2):2 (n6,s6):5 (n5,s5):4]
I170531 22:03:40.116768 63050 storage/store.go:3177  [s3,r11/3:{m-/Max}] added to replica GC queue (peer suggestion)
I170531 22:03:40.118140 62808 storage/replica_proposal.go:385  [s5,r11/4:{m-/Max}] new range lease repl=(n5,s5):4 start=1.800000125,278 epo=1 pro=1.800000125,279 following repl=(n2,s2):2 start=0.000000123,392 epo=1 pro=1.800000125,5
I170531 22:03:40.118258 62636 storage/store.go:2159  [replicaGC,s3,r11/3:{m-/Max}] removing replica
I170531 22:03:40.118380 62636 storage/replica.go:714  [replicaGC,s3,r11/3:{m-/Max}] removed 31 (20+11) keys in 0ms [clear=0ms commit=0ms]
I170531 22:03:40.118522 63291 storage/raft_transport.go:436  raft transport stream to node 4 established
W170531 22:03:40.118656 63176 storage/raft_transport.go:257  unable to accept Raft message from (n5,s5):4: no handler registered for (n4,s4):1
W170531 22:03:40.118759 63174 storage/store.go:3181  [s5] raft error: node 4 claims to not contain store 4 for replica (n4,s4):1: store 4 was not found
W170531 22:03:40.118843 63176 storage/raft_transport.go:257  unable to accept Raft message from (n5,s5):4: no handler registered for (n4,s4):1
I170531 22:03:40.139798 63362 storage/replica_command.go:3568  [s5,r11/4:{m-/Max}] change replicas (REMOVE_REPLICA (n2,s2):2): read existing descriptor r11:{m-/Max} [(n4,s4):1, (n2,s2):2, (n6,s6):5, (n5,s5):4, next=6]
I170531 22:03:40.141467 63169 storage/replica.go:2865  [s5,r11/4:{m-/Max}] proposing REMOVE_REPLICA (n2,s2):2: [(n4,s4):1 (n5,s5):4 (n6,s6):5]
I170531 22:03:40.142786 63312 storage/store.go:3177  [s2,r11/2:{m-/Max}] added to replica GC queue (peer suggestion)
I170531 22:03:40.144050 62502 storage/store.go:2159  [replicaGC,s2,r11/2:{m-/Max}] removing replica
I170531 22:03:40.144188 62502 storage/replica.go:714  [replicaGC,s2,r11/2:{m-/Max}] removed 32 (20+12) keys in 0ms [clear=0ms commit=0ms]
I170531 22:03:40.548083 63181 storage/raft_transport.go:436  raft transport stream to node 4 established
I170531 22:03:40.549119 63400 storage/replica_raftstorage.go:416  [raftsnapshot,s5,r11/4:{m-/Max}] generated Raft snapshot 7fa34f9e at index 27
E170531 22:03:40.549706 63400 storage/queue.go:636  [raftsnapshot,s5,r11/4:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range; initiated GC: [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:40.549983 63248 storage/client_test.go:1177  [replicaGC,r2/4:/{System/Max-Max},s1] test clock advanced to: 3.600000127,0
I170531 22:03:40.550328 63521 storage/replica_proposal.go:385  [s5,r11/4:{m-/Max}] new range lease repl=(n5,s5):4 start=1.800000125,278 epo=1 pro=1.800000125,528 following repl=(n5,s5):4 start=1.800000125,278 epo=1 pro=1.800000125,279
I170531 22:03:40.550915 63672 storage/client_test.go:1177  test clock advanced to: 5.400000129,0
I170531 22:03:40.551012 63616 storage/replica_proposal.go:385  [s6,r11/5:{m-/Max}] new range lease repl=(n6,s6):5 start=1.800000125,608 epo=2 pro=1.800000125,610 following repl=(n5,s5):4 start=1.800000125,278 epo=1 pro=1.800000125,528
W170531 22:03:40.552326 63579 storage/store.go:1356  [s5,r11/4:{m-/Max}] could not gossip system config: [NotLeaseHolderError] r11: replica (n5,s5):4 not lease holder; lease holder unknown
I170531 22:03:40.553799 63612 storage/replica_proposal.go:385  [s6,r11/5:{m-/Max}] new range lease repl=(n6,s6):5 start=1.800000125,608 epo=2 pro=5.400000129,26 following repl=(n6,s6):5 start=1.800000125,608 epo=2 pro=1.800000125,610
I170531 22:03:40.555264 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot e1962939 at index 34
E170531 22:03:40.555580 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range; initiated GC: [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:40.746972 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot 718b05d1 at index 34
E170531 22:03:40.747422 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range; initiated GC: [n4,s4,r2/4:/{System/Max-Max}]
W170531 22:03:40.846924 63088 storage/raft_transport.go:257  unable to accept Raft message from (n4,s4):4: no handler registered for (n3,s3):3
W170531 22:03:40.847037 63080 storage/raft_transport.go:257  unable to accept Raft message from (n4,s4):4: no handler registered for (n2,s2):2
W170531 22:03:40.847054 63055 storage/store.go:3181  [s4] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
W170531 22:03:40.847074 63053 storage/raft_transport.go:442  raft transport stream to node 3 failed: store 3 was not found
W170531 22:03:40.847151 63050 storage/store.go:3181  [s4] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
W170531 22:03:40.847169 63048 storage/raft_transport.go:442  raft transport stream to node 2 failed: store 2 was not found
I170531 22:03:40.847214 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot f415113a at index 34
E170531 22:03:40.847584 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range; initiated GC: [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:40.946883 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot 9da78984 at index 34
E170531 22:03:40.947284 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:41.046914 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot 13c0da48 at index 34
E170531 22:03:41.047322 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:41.146890 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot 1de6fe70 at index 34
E170531 22:03:41.147392 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:41.246876 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot 64f26c7c at index 34
E170531 22:03:41.247287 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:41.346817 63767 storage/raft_transport.go:436  raft transport stream to node 2 established
I170531 22:03:41.346903 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot 18c8219c at index 34
W170531 22:03:41.346926 63722 storage/raft_transport.go:257  unable to accept Raft message from (n4,s4):4: no handler registered for (n2,s2):2
I170531 22:03:41.346974 63766 storage/raft_transport.go:436  raft transport stream to node 3 established
W170531 22:03:41.347047 63772 storage/raft_transport.go:257  unable to accept Raft message from (n4,s4):4: no handler registered for (n3,s3):3
W170531 22:03:41.347099 63769 storage/store.go:3181  [s4] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
W170531 22:03:41.347178 63723 storage/store.go:3181  [s4] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
E170531 22:03:41.347381 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:41.447007 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot 28c4ff56 at index 34
E170531 22:03:41.447942 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:41.497001 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot 69613aeb at index 34
E170531 22:03:41.497422 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:41.646985 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot 1a3e2116 at index 34
E170531 22:03:41.647431 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:41.746910 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot dc7e09ed at index 34
E170531 22:03:41.747350 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:41.846960 63816 storage/raft_transport.go:436  raft transport stream to node 3 established
W170531 22:03:41.847053 63781 storage/raft_transport.go:257  unable to accept Raft message from (n4,s4):4: no handler registered for (n3,s3):3
W170531 22:03:41.847192 63503 storage/store.go:3181  [s4] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
I170531 22:03:41.847207 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot 37ab6d65 at index 34
I170531 22:03:41.847343 63817 storage/raft_transport.go:436  raft transport stream to node 2 established
W170531 22:03:41.847470 63755 storage/raft_transport.go:257  unable to accept Raft message from (n4,s4):4: no handler registered for (n2,s2):2
E170531 22:03:41.847538 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
W170531 22:03:41.847565 63819 storage/store.go:3181  [s4] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
I170531 22:03:41.896895 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot 36f531c2 at index 34
E170531 22:03:41.897272 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:42.046845 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot 1637cc3a at index 34
E170531 22:03:42.047218 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:42.096872 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot a44861db at index 34
E170531 22:03:42.097225 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:42.146817 63835 storage/raft_transport.go:436  raft transport stream to node 3 established
I170531 22:03:42.146934 63834 storage/raft_transport.go:436  raft transport stream to node 2 established
W170531 22:03:42.147106 63860 storage/raft_transport.go:257  unable to accept Raft message from (n4,s4):4: no handler registered for (n2,s2):2
W170531 22:03:42.147229 63862 storage/raft_transport.go:257  unable to accept Raft message from (n4,s4):4: no handler registered for (n3,s3):3
W170531 22:03:42.147342 63837 storage/store.go:3181  [s4] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
W170531 22:03:42.147369 63839 storage/store.go:3181  [s4] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
I170531 22:03:42.246902 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot 73435dd3 at index 34
E170531 22:03:42.247342 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:42.347009 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot 18352a22 at index 34
E170531 22:03:42.347387 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:42.446904 63785 storage/raft_transport.go:436  raft transport stream to node 3 established
W170531 22:03:42.447011 63893 storage/raft_transport.go:257  unable to accept Raft message from (n4,s4):4: no handler registered for (n3,s3):3
I170531 22:03:42.447019 63784 storage/raft_transport.go:436  raft transport stream to node 2 established
I170531 22:03:42.447064 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot 3a4fb072 at index 34
W170531 22:03:42.447138 63787 storage/store.go:3181  [s4] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
W170531 22:03:42.447246 63792 storage/raft_transport.go:257  unable to accept Raft message from (n4,s4):4: no handler registered for (n2,s2):2
E170531 22:03:42.447508 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
W170531 22:03:42.447820 63789 storage/store.go:3181  [s4] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
I170531 22:03:42.546980 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot e39dd40d at index 34
E170531 22:03:42.547381 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:42.646905 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot 66adb93e at index 34
E170531 22:03:42.647277 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:42.696889 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot 9e482902 at index 34
E170531 22:03:42.697275 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:42.746830 63879 storage/raft_transport.go:436  raft transport stream to node 3 established
I170531 22:03:42.746911 63878 storage/raft_transport.go:436  raft transport stream to node 2 established
W170531 22:03:42.747024 63911 storage/raft_transport.go:257  unable to accept Raft message from (n4,s4):4: no handler registered for (n2,s2):2
W170531 22:03:42.747033 63895 storage/raft_transport.go:257  unable to accept Raft message from (n4,s4):4: no handler registered for (n3,s3):3
W170531 22:03:42.747158 63925 storage/store.go:3181  [s4] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
W170531 22:03:42.747181 63881 storage/store.go:3181  [s4] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
I170531 22:03:42.846889 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot 658dbfa7 at index 34
E170531 22:03:42.847370 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:42.946856 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot 5d5cf547 at index 34
E170531 22:03:42.947260 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:43.046923 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot b72839b6 at index 34
E170531 22:03:43.047318 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:43.146922 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot 9b575621 at index 34
E170531 22:03:43.147319 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:43.247061 63961 storage/raft_transport.go:436  raft transport stream to node 3 established
I170531 22:03:43.247371 63960 storage/raft_transport.go:436  raft transport stream to node 2 established
W170531 22:03:43.247553 63929 storage/raft_transport.go:257  unable to accept Raft message from (n4,s4):4: no handler registered for (n2,s2):2
W170531 22:03:43.247602 63760 storage/raft_transport.go:257  unable to accept Raft message from (n4,s4):4: no handler registered for (n3,s3):3
W170531 22:03:43.247731 63963 storage/store.go:3181  [s4] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
W170531 22:03:43.247825 63965 storage/store.go:3181  [s4] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
I170531 22:03:43.247863 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot 871bd75e at index 34
E170531 22:03:43.248230 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:43.346884 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot 2e0048fe at index 34
E170531 22:03:43.347253 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:43.446879 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot 6a4e3375 at index 34
E170531 22:03:43.447284 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:43.546854 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot 1ea12de7 at index 34
E170531 22:03:43.547262 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:43.646893 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot 5baeef15 at index 34
E170531 22:03:43.647285 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:43.746901 63918 storage/raft_transport.go:436  raft transport stream to node 3 established
I170531 22:03:43.746956 63917 storage/raft_transport.go:436  raft transport stream to node 2 established
I170531 22:03:43.746969 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot 197c00b1 at index 34
W170531 22:03:43.747057 63745 storage/raft_transport.go:257  unable to accept Raft message from (n4,s4):4: no handler registered for (n3,s3):3
W170531 22:03:43.747119 64002 storage/raft_transport.go:257  unable to accept Raft message from (n4,s4):4: no handler registered for (n2,s2):2
W170531 22:03:43.747183 63920 storage/store.go:3181  [s4] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
W170531 22:03:43.747228 63977 storage/store.go:3181  [s4] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
E170531 22:03:43.747293 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:43.846869 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot 788ec532 at index 34
E170531 22:03:43.847275 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:43.946900 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot 8026f632 at index 34
E170531 22:03:43.947304 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:44.046890 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot 3d23dd3d at index 34
E170531 22:03:44.047324 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:44.146851 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot c55bc2b2 at index 34
E170531 22:03:44.147261 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:44.246841 63995 storage/raft_transport.go:436  raft transport stream to node 2 established
I170531 22:03:44.246882 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot c0d0b65a at index 34
I170531 22:03:44.246905 63996 storage/raft_transport.go:436  raft transport stream to node 3 established
W170531 22:03:44.247137 64056 storage/raft_transport.go:257  unable to accept Raft message from (n4,s4):4: no handler registered for (n2,s2):2
W170531 22:03:44.247175 63807 storage/raft_transport.go:257  unable to accept Raft message from (n4,s4):4: no handler registered for (n3,s3):3
W170531 22:03:44.247298 64053 storage/store.go:3181  [s4] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
W170531 22:03:44.247308 64054 storage/store.go:3181  [s4] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
E170531 22:03:44.247347 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:44.346905 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot dc2faa23 at index 34
E170531 22:03:44.347279 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:44.446850 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot 397915d5 at index 34
E170531 22:03:44.447234 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:44.547786 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot 67c36480 at index 34
E170531 22:03:44.548202 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:44.647341 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot 76663052 at index 34
E170531 22:03:44.647771 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:44.747094 64026 storage/raft_transport.go:436  raft transport stream to node 3 established
I170531 22:03:44.747136 64025 storage/raft_transport.go:436  raft transport stream to node 2 established
I170531 22:03:44.747225 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot 65c70813 at index 34
W170531 22:03:44.747257 64030 storage/raft_transport.go:257  unable to accept Raft message from (n4,s4):4: no handler registered for (n2,s2):2
W170531 22:03:44.747307 64087 storage/raft_transport.go:257  unable to accept Raft message from (n4,s4):4: no handler registered for (n3,s3):3
W170531 22:03:44.747400 64085 storage/store.go:3181  [s4] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
W170531 22:03:44.747414 64028 storage/store.go:3181  [s4] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
E170531 22:03:44.749627 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:44.848243 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot 8211694c at index 34
E170531 22:03:44.848660 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:44.946979 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot 9443ff0e at index 34
E170531 22:03:44.947371 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:45.047835 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot dd696f9e at index 34
E170531 22:03:45.048223 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:45.147090 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot d1be5b30 at index 34
E170531 22:03:45.147819 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:45.246871 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot c50f3c7b at index 34
I170531 22:03:45.246891 64146 storage/raft_transport.go:436  raft transport stream to node 3 established
I170531 22:03:45.247038 64049 storage/raft_transport.go:436  raft transport stream to node 2 established
W170531 22:03:45.247075 64163 storage/raft_transport.go:257  unable to accept Raft message from (n4,s4):4: no handler registered for (n3,s3):3
W170531 22:03:45.247174 64111 storage/raft_transport.go:257  unable to accept Raft message from (n4,s4):4: no handler registered for (n2,s2):2
W170531 22:03:45.247220 64148 storage/store.go:3181  [s4] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
E170531 22:03:45.247257 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
W170531 22:03:45.247273 64109 storage/store.go:3181  [s4] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
I170531 22:03:45.346904 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot 1a216059 at index 34
E170531 22:03:45.347314 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:45.451654 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot 09a11391 at index 34
E170531 22:03:45.452053 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:45.546851 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot 24d6c533 at index 34
E170531 22:03:45.547221 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:45.646834 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot 1acaeacc at index 34
E170531 22:03:45.647235 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:45.746889 64128 storage/raft_transport.go:436  raft transport stream to node 3 established
I170531 22:03:45.747003 64127 storage/raft_transport.go:436  raft transport stream to node 2 established
W170531 22:03:45.747167 64157 storage/raft_transport.go:257  unable to accept Raft message from (n4,s4):4: no handler registered for (n2,s2):2
W170531 22:03:45.747272 64159 storage/raft_transport.go:257  unable to accept Raft message from (n4,s4):4: no handler registered for (n3,s3):3
I170531 22:03:45.747578 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot b5f77780 at index 34
W170531 22:03:45.747773 64196 storage/store.go:3181  [s4] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
W170531 22:03:45.747816 64194 storage/store.go:3181  [s4] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
E170531 22:03:45.747943 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:45.846860 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot 73c55706 at index 34
E170531 22:03:45.847278 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:45.946804 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot 1f898c01 at index 34
E170531 22:03:45.947186 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:46.046917 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot f6d5530c at index 34
E170531 22:03:46.047314 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:46.147161 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot a516b0e9 at index 34
E170531 22:03:46.147576 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:46.246820 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot e0efd0b4 at index 34
I170531 22:03:46.246929 64190 storage/raft_transport.go:436  raft transport stream to node 3 established
I170531 22:03:46.247020 64189 storage/raft_transport.go:436  raft transport stream to node 2 established
W170531 22:03:46.247114 64253 storage/raft_transport.go:257  unable to accept Raft message from (n4,s4):4: no handler registered for (n3,s3):3
W170531 22:03:46.247136 64259 storage/raft_transport.go:257  unable to accept Raft message from (n4,s4):4: no handler registered for (n2,s2):2
W170531 22:03:46.247245 64193 storage/store.go:3181  [s4] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
W170531 22:03:46.247260 64234 storage/store.go:3181  [s4] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
E170531 22:03:46.247270 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:46.346880 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot c4d47d11 at index 34
E170531 22:03:46.347320 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:46.447042 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot b0b686a4 at index 34
E170531 22:03:46.447439 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:46.546799 64171 storage/raft_transport.go:436  raft transport stream to node 3 established
I170531 22:03:46.546854 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot 87f65a0f at index 34
I170531 22:03:46.546875 64170 storage/raft_transport.go:436  raft transport stream to node 2 established
W170531 22:03:46.546905 64255 storage/raft_transport.go:257  unable to accept Raft message from (n4,s4):4: no handler registered for (n3,s3):3
W170531 22:03:46.546965 64175 storage/raft_transport.go:257  unable to accept Raft message from (n4,s4):4: no handler registered for (n2,s2):2
W170531 22:03:46.547037 64173 storage/store.go:3181  [s4] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
W170531 22:03:46.547077 64240 storage/store.go:3181  [s4] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
E170531 22:03:46.547200 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:46.646804 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot 28175522 at index 34
E170531 22:03:46.647158 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:46.696837 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot 9157106e at index 34
E170531 22:03:46.697185 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:46.849777 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot 17158f8c at index 34
E170531 22:03:46.850216 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:46.946898 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot 0d5ac623 at index 34
E170531 22:03:46.947288 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:47.047673 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot 15582b1c at index 34
I170531 22:03:47.048195 64265 storage/raft_transport.go:436  raft transport stream to node 2 established
W170531 22:03:47.048934 64223 storage/raft_transport.go:257  unable to accept Raft message from (n4,s4):4: no handler registered for (n2,s2):2
E170531 22:03:47.049027 63371 storage/queue.go:636  [raftsnapshot,s6,r11/5:{m-/Max}] snapshot failed: (n4,s4):1: remote couldn't accept snapshot with error: [n4,s4],r11: cannot apply snapshot: snapshot intersects existing range [n4,s4,r2/4:/{System/Max-Max}]
I170531 22:03:47.049194 64266 storage/raft_transport.go:436  raft transport stream to node 3 established
W170531 22:03:47.049290 64221 storage/store.go:3181  [s4] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
W170531 22:03:47.049314 64225 storage/raft_transport.go:257  unable to accept Raft message from (n4,s4):4: no handler registered for (n3,s3):3
W170531 22:03:47.049440 64326 storage/store.go:3181  [s4] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
I170531 22:03:47.096985 63371 storage/replica_raftstorage.go:416  [raftsnapshot,s6,r11/5:{m-/Max}] generated Raft snapshot e0ca53e3 at index 34
E170531 22:03:47.097356 63371 storage/queue.go:636  

Please assign, take a look and update the issue accordingly.

Metadata

Metadata

Assignees

Labels

C-test-failureBroken test (automatically or manually discovered).O-robotOriginated from a bot.

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions