replica has lost quorum, recovering: r2:/System/NodeLiveness{-Max} [(n1,s1):1, (n3,s3):2, (n2,s2):3, (n4,s4):4, next=5, gen=6] -> r2:/System/NodeLiveness{-Max} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r3:/System/{NodeLivenessMax-tsd} [(n1,s1):1, (n3,s3):2, (n4,s4):3, (n2,s2):4, next=5, gen=6] -> r3:/System/{NodeLivenessMax-tsd} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r4:/System{/tsd-tse} [(n1,s1):1, (n3,s3):2, (n2,s2):3, (n4,s4):4, next=5, gen=6] -> r4:/System{/tsd-tse} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r5:/{Systemtse-Table/SystemConfigSpan/Start} [(n1,s1):1, (n4,s4):2, (n3,s3):3, (n2,s2):4, next=5, gen=6] -> r5:/{Systemtse-Table/SystemConfigSpan/Start} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r6:/Table/{SystemConfigSpan/Start-11} [(n3,s3):4, (n4,s4):2, (n2,s2):3, (n1,s1):5, next=6, gen=10] -> r6:/Table/{SystemConfigSpan/Start-11} [(n2,s2):6, next=7, gen=10]
replica has lost quorum, recovering: r7:/Table/1{1-2} [(n1,s1):1, (n4,s4):2, (n2,s2):3, (n3,s3):4, next=5, gen=6] -> r7:/Table/1{1-2} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r8:/Table/1{2-3} [(n3,s3):4, (n2,s2):2, (n4,s4):3, (n1,s1):5, next=6, gen=10] -> r8:/Table/1{2-3} [(n2,s2):6, next=7, gen=10]
replica has lost quorum, recovering: r9:/Table/1{3-4} [(n1,s1):1, (n2,s2):2, (n3,s3):3, (n4,s4):4, next=5, gen=6] -> r9:/Table/1{3-4} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r10:/Table/1{4-5} [(n1,s1):1, (n3,s3):2, (n2,s2):3, (n4,s4):4, next=5, gen=6] -> r10:/Table/1{4-5} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r11:/Table/1{5-6} [(n1,s1):1, (n2,s2):2, (n3,s3):3, (n4,s4):4, next=5, gen=6] -> r11:/Table/1{5-6} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r12:/Table/1{6-7} [(n1,s1):1, (n4,s4):2, (n3,s3):3, (n2,s2):4, next=5, gen=6] -> r12:/Table/1{6-7} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r13:/Table/1{7-8} [(n1,s1):1, (n4,s4):2, (n3,s3):3, (n2,s2):4, next=5, gen=6] -> r13:/Table/1{7-8} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r14:/Table/1{8-9} [(n1,s1):1, (n3,s3):2, (n2,s2):3, (n4,s4):4, next=5, gen=6] -> r14:/Table/1{8-9} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r15:/Table/{19-20} [(n1,s1):1, (n4,s4):2, (n2,s2):3, (n3,s3):4, next=5, gen=6] -> r15:/Table/{19-20} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r16:/Table/2{0-1} [(n1,s1):1, (n3,s3):2, (n4,s4):3, (n2,s2):4, next=5, gen=6] -> r16:/Table/2{0-1} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r17:/Table/2{1-2} [(n1,s1):1, (n3,s3):2, (n2,s2):3, (n4,s4):4, next=5, gen=6] -> r17:/Table/2{1-2} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r18:/Table/2{2-3} [(n1,s1):1, (n2,s2):2, (n4,s4):3, (n3,s3):4, next=5, gen=6] -> r18:/Table/2{2-3} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r19:/Table/2{3-4} [(n1,s1):1, (n4,s4):2, (n3,s3):3, (n2,s2):4, next=5, gen=6] -> r19:/Table/2{3-4} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r20:/Table/2{4-5} [(n1,s1):1, (n3,s3):2, (n4,s4):3, (n2,s2):4, next=5, gen=6] -> r20:/Table/2{4-5} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r21:/Table/2{5-6} [(n1,s1):1, (n4,s4):2, (n2,s2):3, (n3,s3):4, next=5, gen=6] -> r21:/Table/2{5-6} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r22:/Table/2{6-7} [(n1,s1):1, (n4,s4):2, (n2,s2):3, (n3,s3):4, next=5, gen=6] -> r22:/Table/2{6-7} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r23:/Table/2{7-8} [(n1,s1):1, (n4,s4):2, (n3,s3):3, (n2,s2):4, next=5, gen=6] -> r23:/Table/2{7-8} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r24:/Table/2{8-9} [(n3,s3):4, (n2,s2):2, (n4,s4):3, (n1,s1):5, next=6, gen=10] -> r24:/Table/2{8-9} [(n2,s2):6, next=7, gen=10]
replica has lost quorum, recovering: r25:/{Table/29-NamespaceTable/30} [(n1,s1):1, (n4,s4):2, (n2,s2):3, (n3,s3):4, next=5, gen=6] -> r25:/{Table/29-NamespaceTable/30} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r26:/NamespaceTable/{30-Max} [(n1,s1):1, (n4,s4):2, (n2,s2):3, (n3,s3):4, next=5, gen=6] -> r26:/NamespaceTable/{30-Max} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r27:/{NamespaceTable/Max-Table/32} [(n1,s1):1, (n2,s2):2, (n4,s4):3, (n3,s3):4, next=5, gen=6] -> r27:/{NamespaceTable/Max-Table/32} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r28:/Table/3{2-3} [(n1,s1):1, (n2,s2):2, (n3,s3):3, (n4,s4):4, next=5, gen=6] -> r28:/Table/3{2-3} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r29:/Table/3{3-4} [(n1,s1):1, (n3,s3):2, (n2,s2):3, (n4,s4):4, next=5, gen=6] -> r29:/Table/3{3-4} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r30:/Table/3{4-5} [(n1,s1):1, (n2,s2):2, (n4,s4):3, (n3,s3):4, next=5, gen=6] -> r30:/Table/3{4-5} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r31:/Table/3{5-6} [(n1,s1):1, (n3,s3):2, (n2,s2):3, (n4,s4):4, next=5, gen=6] -> r31:/Table/3{5-6} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r32:/Table/3{6-7} [(n1,s1):1, (n4,s4):2, (n2,s2):3, (n3,s3):4, next=5, gen=6] -> r32:/Table/3{6-7} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r33:/Table/3{7-8} [(n1,s1):1, (n4,s4):2, (n2,s2):3, (n3,s3):4, next=5, gen=6] -> r33:/Table/3{7-8} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r34:/Table/3{8-9} [(n1,s1):1, (n2,s2):2, (n3,s3):3, (n4,s4):4, next=5, gen=6] -> r34:/Table/3{8-9} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r35:/Table/{39-40} [(n1,s1):1, (n2,s2):2, (n3,s3):3, (n4,s4):4, next=5, gen=6] -> r35:/Table/{39-40} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r36:/Table/4{0-1} [(n1,s1):1, (n4,s4):2, (n2,s2):3, (n3,s3):4, next=5, gen=6] -> r36:/Table/4{0-1} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r37:/Table/4{1-2} [(n1,s1):1, (n4,s4):2, (n3,s3):3, (n2,s2):4, next=5, gen=6] -> r37:/Table/4{1-2} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r38:/Table/4{2-3} [(n1,s1):1, (n3,s3):2, (n4,s4):3, (n2,s2):4, next=5, gen=6] -> r38:/Table/4{2-3} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r39:/Table/4{3-4} [(n1,s1):1, (n2,s2):2, (n3,s3):3, (n4,s4):4, next=5, gen=6] -> r39:/Table/4{3-4} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r40:/Table/4{4-5} [(n2,s2):4, (n4,s4):2, (n3,s3):3, (n1,s1):5, next=6, gen=10] -> r40:/Table/4{4-5} [(n2,s2):6, next=7, gen=10]
replica has lost quorum, recovering: r41:/Table/4{5-6} [(n1,s1):1, (n4,s4):2, (n3,s3):3, (n2,s2):4, next=5, gen=6] -> r41:/Table/4{5-6} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r42:/Table/4{6-7} [(n1,s1):1, (n3,s3):2, (n4,s4):3, (n2,s2):4, next=5, gen=6] -> r42:/Table/4{6-7} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r43:/{Table/47-Max} [(n1,s1):1, (n3,s3):2, (n4,s4):3, (n2,s2):4, next=5, gen=6] -> r43:/{Table/47-Max} [(n2,s2):5, next=6, gen=6]
aborting intent: /Local/Range/System/tsd/RangeDescriptor (txn 4435d938-2f83-426c-a270-6aff4c966650)
Scanning replicas on store cluster_id:00578a7c-aef7-4181-b427-3f8d8aaa401a node_id:2 store_id:2 for dead peers []
id is_live replicas is_decommissioning membership is_draining
3 false 43 true decommissioning false
4 false 43 true decommissioning false
debug_test.go:391: expected replicas on {1,2,5,6} but got {1,2,5}
--- FAIL: TestRemoveDeadReplicas/2/4/r=4 (26.39s)
cli.TestRemoveDeadReplicas failed with artifacts on master @ 912964e02ddd951c77d4f71981ae18b3894e9084:
Help
See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:
This test on roachdash | Improve this report!
Jira issue: CRDB-12480