roachtest: acceptance/version-upgrade failed #38428
Description
See #38380 (comment)
Diagnosis is made more difficult by roachtest bugs, but I believe the root cause is that a server isn't running after returning from c.Start and hitting
cockroach/pkg/cmd/roachtest/upgrade.go
Line 431 in 7e2ceae
build/builder.sh env COCKROACH_DEV_LICENSE=crl-0-EJ7ZyugFGAIiI0NvY2tyb2FjaCBMYWJzIC0gUHJvZHVjdGlvbiBUZXN0aW5n stdbuf -oL -eL ./bin/roachtest run (acceptance|kv/splits|cdc/bank) --local --cockroach cockroach --roachprod bin/roachprod --workload bin/workload --artifacts artifacts --teamcity
##teamcity[testStarted name='acceptance' flowId='acceptance']
22:29:37 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod create local -n 4 --local-ssd-no-ext4-barrier
Creating cluster local with 4 nodes
##teamcity[testStarted name='acceptance/bank/cluster-recovery' flowId='acceptance/bank/cluster-recovery']
22:29:37 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod wipe local
local: stopping and waiting
local: wiping
22:29:37 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod put local /go/src/github.com/cockroachdb/cockroach/cockroach ./cockroach
local: putting /go/src/github.com/cockroachdb/cockroach/cockroach ./cockroach
1: done
2: done
3: done
4: done
22:29:37 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local
local: starting.
local: initializing cluster settings
SET CLUSTER SETTING
22:29:39 bank.go:450: monkey starts (seed -2785650561259812330)
22:29:40 bank.go:245: round 1: restarting nodes [3 1]
22:29:40 bank.go:250: round 1: restarting 3
22:29:40 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:3
local: stopping and waiting22:29:40 bank.go:420: round 1: client counts: (42, 43, 38, 58)
.
22:29:41 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:3
local: starting
22:29:42 bank.go:250: round 1: restarting 1
22:29:42 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:1
local: stopping and waiting.
22:29:43 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:1
local: starting
local: initializing cluster settings
22:29:43 bank.go:278: round 1: monkey sleeping while cluster recovers...
22:29:43 bank.go:420: round 1: client counts: (42, 161, 38, 282)
22:29:44 bank.go:270: round 1: progress made by client 1
22:29:44 bank.go:288: round 1: cluster recovered
22:29:44 bank.go:420: round 2: client counts: (42, 537, 38, 1017)
22:29:45 bank.go:420: round 2: client counts: (42, 961, 38, 1799)
22:29:46 bank.go:420: round 2: client counts: (42, 1436, 38, 2653)
22:29:47 bank.go:420: round 2: client counts: (42, 1919, 38, 3498)
22:29:48 bank.go:245: round 2: restarting nodes [2 3 4]
22:29:48 bank.go:250: round 2: restarting 2
22:29:48 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:2
local: stopping and waiting22:29:48 bank.go:420: round 2: client counts: (43, 2297, 40, 4172)
.
22:29:49 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:2
local: starting
22:29:50 bank.go:250: round 2: restarting 3
22:29:50 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:3
local: stopping and waiting.
22:29:51 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:3
local: starting
22:29:51 bank.go:250: round 2: restarting 4
22:29:51 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:4
local: stopping and waiting.
22:29:52 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:4
local: starting
22:29:53 bank.go:278: round 2: monkey sleeping while cluster recovers...
22:29:53 bank.go:420: round 2: client counts: (114, 2297, 108, 4292)
22:29:54 bank.go:270: round 2: progress made by client 0
22:29:54 bank.go:288: round 2: cluster recovered
22:29:54 bank.go:245: round 3: restarting nodes [4]
22:29:54 bank.go:250: round 3: restarting 4
22:29:54 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:4
local: stopping and waiting22:29:54 bank.go:420: round 3: client counts: (255, 2298, 534, 5030)
.
22:29:55 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:4
local: starting
22:29:56 bank.go:278: round 3: monkey sleeping while cluster recovers...
22:29:56 bank.go:420: round 3: client counts: (255, 2330, 534, 5085)
22:29:57 bank.go:270: round 3: progress made by client 0
22:29:57 bank.go:288: round 3: cluster recovered
22:29:57 bank.go:420: round 4: client counts: (420, 2500, 534, 5884)
22:29:58 bank.go:245: round 4: restarting nodes [4 1 2 3]
22:29:58 bank.go:250: round 4: restarting 4
22:29:58 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:4
local: stopping and waiting22:29:58 bank.go:420: round 4: client counts: (420, 2500, 535, 5925)
.
22:29:59 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:4
local: starting
22:29:59 bank.go:250: round 4: restarting 1
22:29:59 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:1
local: stopping and waiting.
22:30:00 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:1
local: starting
local: initializing cluster settings
22:30:00 bank.go:250: round 4: restarting 2
22:30:00 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:2
local: stopping and waiting.
22:30:01 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:2
local: starting
22:30:02 bank.go:250: round 4: restarting 3
22:30:02 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:3
local: stopping and waiting.
22:30:03 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:3
local: starting
22:30:03 bank.go:278: round 4: monkey sleeping while cluster recovers...
22:30:07 bank.go:270: round 4: progress made by client 1
22:30:07 bank.go:288: round 4: cluster recovered
22:30:07 bank.go:245: round 5: restarting nodes [2]
22:30:07 bank.go:250: round 5: restarting 2
22:30:07 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:2
local: stopping and waiting22:30:07 bank.go:420: round 5: client counts: (421, 2574, 612, 6221)
.
22:30:08 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:2
local: starting
22:30:09 bank.go:278: round 5: monkey sleeping while cluster recovers...
22:30:09 bank.go:420: round 5: client counts: (661, 2612, 861, 6667)
22:30:10 bank.go:270: round 5: progress made by client 0
22:30:10 bank.go:288: round 5: cluster recovered
22:30:10 bank.go:245: round 6: restarting nodes [4 3]
22:30:10 bank.go:250: round 6: restarting 4
22:30:10 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:4
local: stopping and waiting22:30:10 bank.go:420: round 6: client counts: (819, 2761, 1015, 6949)
.
22:30:11 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:4
local: starting
22:30:11 bank.go:250: round 6: restarting 3
22:30:11 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:3
local: stopping and waiting.
22:30:12 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:3
local: starting
22:30:13 bank.go:278: round 6: monkey sleeping while cluster recovers...
22:30:13 bank.go:420: round 6: client counts: (1146, 3092, 1015, 6949)
22:30:14 bank.go:270: round 6: progress made by client 0
22:30:14 bank.go:288: round 6: cluster recovered
22:30:14 bank.go:420: round 7: client counts: (1287, 3230, 1015, 6949)
22:30:17 bank.go:245: round 7: restarting nodes [3]
22:30:17 bank.go:250: round 7: restarting 3
22:30:17 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:3
local: stopping and waiting22:30:18 bank.go:420: round 7: client counts: (1287, 3230, 1016, 6956)
.
22:30:18 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:3
local: starting
22:30:19 bank.go:278: round 7: monkey sleeping while cluster recovers...
22:30:20 bank.go:420: round 7: client counts: (1633, 3581, 1016, 7560)
22:30:20 bank.go:270: round 7: progress made by client 0
22:30:20 bank.go:288: round 7: cluster recovered
22:30:21 bank.go:420: round 8: client counts: (1717, 3665, 1016, 8375)
22:30:22 bank.go:420: round 8: client counts: (1717, 3665, 1016, 9248)
22:30:23 bank.go:420: round 8: client counts: (1717, 3665, 1016, 10106)
22:30:24 bank.go:420: round 8: client counts: (1717, 3665, 1016, 10975)
22:30:25 bank.go:420: round 8: client counts: (1717, 3665, 1016, 11829)
22:30:26 bank.go:420: round 8: client counts: (1717, 3665, 1016, 12702)
22:30:27 bank.go:420: round 8: client counts: (1717, 3665, 1016, 13611)
22:30:28 bank.go:420: round 8: client counts: (1717, 3665, 1016, 14484)
22:30:29 bank.go:420: round 8: client counts: (1717, 3665, 1016, 15394)
22:30:29 bank.go:245: round 8: restarting nodes [2 4 3 1]
22:30:29 bank.go:250: round 8: restarting 2
22:30:29 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:2
local: stopping and waiting22:30:30 bank.go:420: round 8: client counts: (1717, 3665, 1017, 16026)
.
22:30:30 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:2
local: starting
22:30:31 bank.go:250: round 8: restarting 4
22:30:31 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:4
local: stopping and waiting.
22:30:32 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:4
local: starting
22:30:32 bank.go:250: round 8: restarting 3
22:30:32 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:3
local: stopping and waiting.
22:30:33 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:3
local: starting
22:30:34 bank.go:250: round 8: restarting 1
22:30:34 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:1
local: stopping and waiting.
22:30:35 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:1
local: starting
local: initializing cluster settings
22:30:35 bank.go:278: round 8: monkey sleeping while cluster recovers...
[ 1] acceptance/bank/cluster-recovery: ??? (0s)
22:30:39 bank.go:270: round 8: progress made by client 0
22:30:39 bank.go:288: round 8: cluster recovered
22:30:39 bank.go:167: client 4 shutting down
22:30:39 bank.go:167: client 3 shutting down
22:30:39 bank.go:245: round 9: restarting nodes [3 2 4]
22:30:39 bank.go:278: round 9: monkey sleeping while cluster recovers...
22:30:39 bank.go:283: round 9: not waiting for recovery due to signal that we're done
22:30:39 bank.go:167: client 1 shutting down
22:30:39 bank.go:167: client 2 shutting down
22:30:39 bank.go:471: 22567 transfers (375.8/sec) in 60.0s
22:30:39 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor local --oneshot --ignore-empty-nodes
22:30:40 cluster.go:1102: running (fast) consistency checks on node 1
22:30:40 cluster.go:949: fetching logs
22:30:40 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod get local logs artifacts/acceptance/bank/cluster-recovery/logs
local: getting logs artifacts/acceptance/bank/cluster-recovery/logs
1: done
2: done
3: done
4: done
--- PASS: acceptance/bank/cluster-recovery (63.13s)
##teamcity[testFinished name='acceptance/bank/cluster-recovery' flowId='acceptance/bank/cluster-recovery']
##teamcity[testStarted name='acceptance/bank/node-restart' flowId='acceptance/bank/node-restart']
22:30:40 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod wipe local
local: stopping and waiting.
local: wiping
22:30:42 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod put local /go/src/github.com/cockroachdb/cockroach/cockroach ./cockroach
local: putting /go/src/github.com/cockroachdb/cockroach/cockroach ./cockroach
1: done
2: done
3: done
4: done
22:30:42 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local
local: starting.
local: initializing cluster settings
SET CLUSTER SETTING
22:30:43 bank.go:496: monkey starts (seed -6525498360904207697)
22:30:43 bank.go:245: round 1: restarting nodes [1]
22:30:43 bank.go:250: round 1: restarting 1
22:30:43 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:1
local: stopping and waiting22:30:44 bank.go:420: round 1: client counts: (338)
.
22:30:45 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:1
local: starting
local: initializing cluster settings
22:30:45 bank.go:278: round 1: monkey sleeping while cluster recovers...
22:30:45 bank.go:420: round 1: client counts: (711)
22:30:46 bank.go:270: round 1: progress made by client 0
22:30:46 bank.go:288: round 1: cluster recovered
22:30:46 bank.go:245: round 2: restarting nodes [3]
22:30:46 bank.go:250: round 2: restarting 3
22:30:46 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:3
local: stopping and waiting22:30:46 bank.go:420: round 2: client counts: (939)
.
22:30:47 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:3
local: starting
22:30:47 bank.go:278: round 2: monkey sleeping while cluster recovers...
22:30:53 bank.go:420: round 2: client counts: (1316)
22:30:53 bank.go:270: round 2: progress made by client 0
22:30:53 bank.go:288: round 2: cluster recovered
22:30:53 bank.go:245: round 3: restarting nodes [1]
22:30:53 bank.go:250: round 3: restarting 1
22:30:53 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:1
local: stopping and waiting22:30:54 bank.go:420: round 3: client counts: (1732)
.
22:30:54 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:1
local: starting
local: initializing cluster settings
22:30:55 bank.go:278: round 3: monkey sleeping while cluster recovers...
22:30:55 bank.go:420: round 3: client counts: (2129)
22:30:56 bank.go:270: round 3: progress made by client 0
22:30:56 bank.go:288: round 3: cluster recovered
22:30:56 bank.go:245: round 4: restarting nodes [1]
22:30:56 bank.go:250: round 4: restarting 1
22:30:56 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:1
local: stopping and waiting22:30:56 bank.go:420: round 4: client counts: (2547)
.
22:30:57 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:1
local: starting
local: initializing cluster settings
22:30:57 bank.go:278: round 4: monkey sleeping while cluster recovers...
22:30:57 bank.go:420: round 4: client counts: (2972)
22:30:58 bank.go:270: round 4: progress made by client 0
22:30:58 bank.go:288: round 4: cluster recovered
22:30:58 bank.go:245: round 5: restarting nodes [4]
22:30:58 bank.go:250: round 5: restarting 4
22:30:58 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:4
22:30:58 bank.go:420: round 5: client counts: (3396)
local: stopping and waiting22:30:59 bank.go:420: round 5: client counts: (3405)
.
22:30:59 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:4
local: starting
22:31:00 bank.go:278: round 5: monkey sleeping while cluster recovers...
22:31:04 bank.go:420: round 5: client counts: (3602)
22:31:05 bank.go:270: round 5: progress made by client 0
22:31:05 bank.go:288: round 5: cluster recovered
22:31:05 bank.go:245: round 6: restarting nodes [1]
22:31:05 bank.go:250: round 6: restarting 1
22:31:05 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:1
local: stopping and waiting22:31:05 bank.go:420: round 6: client counts: (4015)
.
22:31:06 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:1
local: starting
local: initializing cluster settings
22:31:06 bank.go:278: round 6: monkey sleeping while cluster recovers...
22:31:06 bank.go:420: round 6: client counts: (4421)
22:31:07 bank.go:270: round 6: progress made by client 0
22:31:07 bank.go:288: round 6: cluster recovered
22:31:07 bank.go:245: round 7: restarting nodes [2]
22:31:07 bank.go:250: round 7: restarting 2
22:31:07 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:2
local: stopping and waiting22:31:07 bank.go:420: round 7: client counts: (4815)
.
22:31:08 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:2
local: starting
22:31:09 bank.go:278: round 7: monkey sleeping while cluster recovers...
22:31:13 bank.go:270: round 7: progress made by client 0
22:31:13 bank.go:288: round 7: cluster recovered
22:31:13 bank.go:245: round 8: restarting nodes [2]
22:31:13 bank.go:250: round 8: restarting 2
22:31:13 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:2
local: stopping and waiting22:31:13 bank.go:420: round 8: client counts: (5572)
.
22:31:14 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:2
local: starting
22:31:14 bank.go:278: round 8: monkey sleeping while cluster recovers...
22:31:14 bank.go:420: round 8: client counts: (6376)
22:31:15 bank.go:270: round 8: progress made by client 0
22:31:15 bank.go:288: round 8: cluster recovered
22:31:15 bank.go:245: round 9: restarting nodes [2]
22:31:15 bank.go:250: round 9: restarting 2
22:31:15 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:2
local: stopping and waiting22:31:15 bank.go:420: round 9: client counts: (7190)
.
22:31:16 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:2
local: starting22:31:16 bank.go:420: round 9: client counts: (8002)
22:31:17 bank.go:278: round 9: monkey sleeping while cluster recovers...
22:31:17 bank.go:420: round 9: client counts: (8786)
22:31:18 bank.go:270: round 9: progress made by client 0
22:31:18 bank.go:288: round 9: cluster recovered
22:31:18 bank.go:245: round 10: restarting nodes [1]
22:31:18 bank.go:250: round 10: restarting 1
22:31:18 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:1
local: stopping and waiting22:31:18 bank.go:420: round 10: client counts: (9572)
.
22:31:19 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:1
local: starting
local: initializing cluster settings
22:31:19 bank.go:278: round 10: monkey sleeping while cluster recovers...
22:31:19 bank.go:420: round 10: client counts: (10376)
22:31:20 bank.go:270: round 10: progress made by client 0
22:31:20 bank.go:288: round 10: cluster recovered
22:31:20 bank.go:245: round 11: restarting nodes [4]
22:31:20 bank.go:250: round 11: restarting 4
22:31:20 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:4
local: stopping and waiting22:31:20 bank.go:420: round 11: client counts: (11013)
.
22:31:21 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:4
local: starting
22:31:22 bank.go:278: round 11: monkey sleeping while cluster recovers...
22:31:23 bank.go:420: round 11: client counts: (11362)
22:31:24 bank.go:270: round 11: progress made by client 0
22:31:24 bank.go:288: round 11: cluster recovered
22:31:24 bank.go:245: round 12: restarting nodes [2]
22:31:24 bank.go:250: round 12: restarting 2
22:31:24 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:2
local: stopping and waiting22:31:24 bank.go:420: round 12: client counts: (12159)
.
22:31:25 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:2
local: starting
22:31:25 bank.go:278: round 12: monkey sleeping while cluster recovers...
22:31:25 bank.go:420: round 12: client counts: (12938)
22:31:26 bank.go:270: round 12: progress made by client 0
22:31:26 bank.go:288: round 12: cluster recovered
22:31:26 bank.go:245: round 13: restarting nodes [1]
22:31:26 bank.go:250: round 13: restarting 1
22:31:26 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:1
local: stopping and waiting22:31:26 bank.go:420: round 13: client counts: (13725)
.
22:31:27 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:1
local: starting22:31:27 bank.go:420: round 13: client counts: (14512)
local: initializing cluster settings
22:31:28 bank.go:278: round 13: monkey sleeping while cluster recovers...
22:31:28 bank.go:420: round 13: client counts: (15309)
22:31:29 bank.go:270: round 13: progress made by client 0
22:31:29 bank.go:288: round 13: cluster recovered
22:31:29 bank.go:245: round 14: restarting nodes [3]
22:31:29 bank.go:250: round 14: restarting 3
22:31:29 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:3
local: stopping and waiting22:31:29 bank.go:420: round 14: client counts: (16109)
.
22:31:30 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:3
local: starting
22:31:30 bank.go:278: round 14: monkey sleeping while cluster recovers...
22:31:30 bank.go:420: round 14: client counts: (16876)
22:31:31 bank.go:270: round 14: progress made by client 0
22:31:31 bank.go:288: round 14: cluster recovered
22:31:31 bank.go:245: round 15: restarting nodes [4]
22:31:31 bank.go:250: round 15: restarting 4
22:31:31 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:4
local: stopping and waiting22:31:31 bank.go:420: round 15: client counts: (17373)
.
22:31:32 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:4
local: starting
22:31:32 bank.go:278: round 15: monkey sleeping while cluster recovers...
22:31:36 bank.go:420: round 15: client counts: (17702)
22:31:36 bank.go:270: round 15: progress made by client 0
22:31:36 bank.go:288: round 15: cluster recovered
22:31:36 bank.go:245: round 16: restarting nodes [2]
22:31:36 bank.go:250: round 16: restarting 2
22:31:36 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:2
local: stopping and waiting[ 2] acceptance/bank/node-restart: stopping cluster (1s)
22:31:37 bank.go:420: round 16: client counts: (18471)
.
22:31:37 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:2
local: starting
22:31:38 bank.go:278: round 16: monkey sleeping while cluster recovers...
22:31:38 bank.go:420: round 16: client counts: (19233)
22:31:39 bank.go:270: round 16: progress made by client 0
22:31:39 bank.go:288: round 16: cluster recovered
22:31:39 bank.go:245: round 17: restarting nodes [3]
22:31:39 bank.go:250: round 17: restarting 3
22:31:39 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:3
local: stopping and waiting22:31:39 bank.go:420: round 17: client counts: (19693)
.
22:31:40 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:3
local: starting
22:31:40 bank.go:278: round 17: monkey sleeping while cluster recovers...
22:31:40 bank.go:420: round 17: client counts: (19801)
22:31:41 bank.go:270: round 17: progress made by client 0
22:31:41 bank.go:288: round 17: cluster recovered
22:31:41 bank.go:245: round 18: restarting nodes [1]
22:31:41 bank.go:250: round 18: restarting 1
22:31:41 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:1
local: stopping and waiting22:31:41 bank.go:420: round 18: client counts: (20571)
.
22:31:42 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:1
local: starting22:31:42 bank.go:420: round 18: client counts: (21357)
local: initializing cluster settings
22:31:43 bank.go:278: round 18: monkey sleeping while cluster recovers...
22:31:43 bank.go:167: client 1 shutting down
22:31:43 bank.go:509: 22031 transfers (367.2/sec) in 60.0s
22:31:44 bank.go:283: round 18: not waiting for recovery due to signal that we're done
22:31:44 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor local --oneshot --ignore-empty-nodes
22:31:49 cluster.go:1102: running (fast) consistency checks on node 1
22:31:49 cluster.go:949: fetching logs
22:31:49 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod get local logs artifacts/acceptance/bank/node-restart/logs
local: getting logs artifacts/acceptance/bank/node-restart/logs
1: done
2: done
3: done
4: done
--- PASS: acceptance/bank/node-restart (68.20s)
##teamcity[testFinished name='acceptance/bank/node-restart' flowId='acceptance/bank/node-restart']
##teamcity[testStarted name='acceptance/bank/zerosum-splits' flowId='acceptance/bank/zerosum-splits']
##teamcity[testIgnored name='acceptance/bank/zerosum-splits' message='https://github.com/cockroachdb/cockroach/issues/33683 (runs into various errors during its rebalances, see isExpectedRelocateError)']
--- SKIP: acceptance/bank/zerosum-splits (0.00s)
https://github.com/cockroachdb/cockroach/issues/33683 (runs into various errors during its rebalances, see isExpectedRelocateError)
##teamcity[testFinished name='acceptance/bank/zerosum-splits' flowId='acceptance/bank/zerosum-splits']
##teamcity[testStarted name='acceptance/build-info' flowId='acceptance/build-info']
22:31:49 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod wipe local
local: stopping and waiting.
local: wiping
22:31:50 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod put local /go/src/github.com/cockroachdb/cockroach/cockroach ./cockroach
local: putting /go/src/github.com/cockroachdb/cockroach/cockroach ./cockroach
1: done
2: done
3: done
4: done
22:31:50 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local
local: starting.
local: initializing cluster settings
SET CLUSTER SETTING
22:31:51 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor local --oneshot --ignore-empty-nodes
22:31:52 cluster.go:1102: running (fast) consistency checks on node 1
22:31:52 cluster.go:949: fetching logs
22:31:52 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod get local logs artifacts/acceptance/build-info/logs
local: getting logs artifacts/acceptance/build-info/logs
1: done
2: done
3: done
4: done
--- PASS: acceptance/build-info (3.17s)
##teamcity[testFinished name='acceptance/build-info' flowId='acceptance/build-info']
##teamcity[testStarted name='acceptance/build-analyze' flowId='acceptance/build-analyze']
22:31:52 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod wipe local
local: stopping and waiting.
local: wiping
22:31:53 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor local --oneshot --ignore-empty-nodes
22:31:53 cluster.go:1106: no live node found, skipping consistency check
22:31:53 cluster.go:949: fetching logs
22:31:53 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod get local logs artifacts/acceptance/build-analyze/logs
local: getting logs artifacts/acceptance/build-analyze/logs
1: stat /home/roach/local/1/logs: no such file or directory
2: stat /home/roach/local/2/logs: no such file or directory
3: stat /home/roach/local/3/logs: no such file or directory
4: stat /home/roach/local/4/logs: no such file or directory
I190625 22:31:53.500026 1 cluster_synced.go:1252 get logs failed
22:31:56 test.go:1213: failed to download logs: /go/src/github.com/cockroachdb/cockroach/bin/roachprod get local logs artifacts/acceptance/build-analyze/logs returned:
stderr:
I190625 22:31:53.500026 1 cluster_synced.go:1252 get logs failed
stdout:
local: getting logs artifacts/acceptance/build-analyze/logs
1: stat /home/roach/local/1/logs: no such file or directory
2: stat /home/roach/local/2/logs: no such file or directory
3: stat /home/roach/local/3/logs: no such file or directory
4: stat /home/roach/local/4/logs: no such file or directory
: exit status 1
##teamcity[testIgnored name='acceptance/build-analyze' message='local execution not supported']
--- SKIP: acceptance/build-analyze (4.17s)
local execution not supported
##teamcity[testFinished name='acceptance/build-analyze' flowId='acceptance/build-analyze']
##teamcity[testStarted name='acceptance/cli/node-status' flowId='acceptance/cli/node-status']
22:31:56 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod wipe local
local: stopping and waiting
local: wiping
22:31:56 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod put local /go/src/github.com/cockroachdb/cockroach/cockroach ./cockroach
local: putting /go/src/github.com/cockroachdb/cockroach/cockroach ./cockroach
1: done
2: done
3: done
4: done
22:31:56 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:1-3
local: starting.
local: initializing cluster settings
SET CLUSTER SETTING
22:32:18 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:1 -- ./cockroach node status --insecure -p {pgport:1}
22:32:19 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:2
local: stopping and waiting.
22:32:20 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:1 -- ./cockroach node status --insecure -p {pgport:1}
22:32:20 cli.go:72: not done: [is_available is_live true true false false true true] vs [is_available is_live true true true true true true]
22:32:21 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:1 -- ./cockroach node status --insecure -p {pgport:1}
22:32:21 cli.go:72: not done: [is_available is_live true true false false true true] vs [is_available is_live true true true true true true]
22:32:22 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:1 -- ./cockroach node status --insecure -p {pgport:1}
22:32:22 cli.go:72: not done: [is_available is_live true true false false true true] vs [is_available is_live true true true false true true]
22:32:23 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:1 -- ./cockroach node status --insecure -p {pgport:1}
22:32:23 cli.go:72: not done: [is_available is_live true true false false true true] vs [is_available is_live true true true false true true]
22:32:24 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:1 -- ./cockroach node status --insecure -p {pgport:1}
22:32:24 cli.go:72: not done: [is_available is_live true true false false true true] vs [is_available is_live true true true false true true]
22:32:25 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:1 -- ./cockroach node status --insecure -p {pgport:1}
22:32:25 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:3
local: stopping and waiting.
22:32:26 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:1 -- ./cockroach node status --insecure -p {pgport:1}
22:32:26 cli.go:72: not done: [is_available is_live false true false false false false] vs [is_available is_live true true false false true true]
22:32:27 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:1 -- ./cockroach node status --insecure -p {pgport:1}
22:32:27 cli.go:72: not done: [is_available is_live false true false false false false] vs [is_available is_live true true false false true false]
22:32:28 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:1 -- ./cockroach node status --insecure -p {pgport:1}
22:32:28 cli.go:72: not done: [is_available is_live false true false false false false] vs [is_available is_live true true false false true false]
22:32:29 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:1 -- ./cockroach node status --insecure -p {pgport:1}
22:32:30 cli.go:72: not done: [is_available is_live false true false false false false] vs [is_available is_live true true false false true false]
22:32:31 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:1 -- ./cockroach node status --insecure -p {pgport:1}
22:32:31 cli.go:72: not done: [is_available is_live false true false false false false] vs [is_available is_live true true false false true false]
22:32:32 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:1 -- ./cockroach node status --insecure -p {pgport:1}
22:32:32 cli.go:72: not done: [is_available is_live false true false false false false] vs [is_available is_live true true false false true false]
22:32:33 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:1 -- ./cockroach node status --insecure -p {pgport:1}
22:32:33 cli.go:72: not done: [is_available is_live false true false false false false] vs [is_available is_live true true false false true false]
22:32:34 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:1 -- ./cockroach node status --insecure -p {pgport:1}
22:32:34 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:1-3
local: stopping and waiting.
22:32:35 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:1-2
local: starting
local: initializing cluster settings
[ 3] acceptance/cli/node-status: ??? (0s)
22:32:41 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:1 -- ./cockroach node status --insecure -p {pgport:1}
22:32:41 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:3
local: starting
22:32:41 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor local --oneshot --ignore-empty-nodes
22:32:41 cluster.go:1102: running (fast) consistency checks on node 1
22:32:42 cluster.go:949: fetching logs
22:32:42 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod get local logs artifacts/acceptance/cli/node-status/logs
local: getting logs artifacts/acceptance/cli/node-status/logs
I190625 22:32:42.037694 1 cluster_synced.go:1252 get logs failed
1: done
2: done
3: done
4: stat /home/roach/local/4/logs: no such file or directory
22:32:45 test.go:1213: failed to download logs: /go/src/github.com/cockroachdb/cockroach/bin/roachprod get local logs artifacts/acceptance/cli/node-status/logs returned:
stderr:
I190625 22:32:42.037694 1 cluster_synced.go:1252 get logs failed
stdout:
local: getting logs artifacts/acceptance/cli/node-status/logs
1: done
2: done
3: done
4: stat /home/roach/local/4/logs: no such file or directory
: exit status 1
--- PASS: acceptance/cli/node-status (48.54s)
##teamcity[testFinished name='acceptance/cli/node-status' flowId='acceptance/cli/node-status']
##teamcity[testStarted name='acceptance/decommission' flowId='acceptance/decommission']
22:32:45 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod wipe local
local: stopping and waiting.
local: wiping
22:32:46 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod put local /go/src/github.com/cockroachdb/cockroach/cockroach ./cockroach
local: putting /go/src/github.com/cockroachdb/cockroach/cockroach ./cockroach
1: done
2: done
3: done
4: done
22:32:46 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start --env=COCKROACH_SCAN_MAX_IDLE_TIME=5ms local
local: starting.
local: initializing cluster settings
SET CLUSTER SETTING
22:32:47 decommission.go:338: decommissioning first node from the second, polling the status manually
22:32:47 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:2 -- ./cockroach node decommission --wait none --format csv 1 --insecure --port={pgport:2}
22:32:47 decommission.go:262:
id,is_live,replicas,is_decommissioning,is_draining
1,true,21,true,false
22:32:48 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:2 -- ./cockroach node decommission --wait none --format csv 1 --insecure --port={pgport:2}
22:32:52 decommission.go:262:
id,is_live,replicas,is_decommissioning,is_draining
1,true,9,true,false
22:32:52 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:2 -- ./cockroach node decommission --wait none --format csv 1 --insecure --port={pgport:2}
22:32:53 decommission.go:262:
id,is_live,replicas,is_decommissioning,is_draining
1,true,0,true,false
No more data reported on target nodes. Please verify cluster health before removing the nodes.
22:32:53 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:2 -- ./cockroach node ls --format csv --insecure --port={pgport:2}
22:32:53 decommission.go:262: id
1
2
3
4
22:32:53 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:2 -- ./cockroach node status --format csv --insecure --port={pgport:2}
22:32:53 decommission.go:262: id,address,build,started_at,updated_at,is_available,is_live
1,127.0.0.1:26257,v19.2.0-alpha.20190606-356-gf488755,2019-06-25 22:32:46.528462+00:00,2019-06-25 22:32:52.193619+00:00,true,true
2,127.0.0.1:26259,v19.2.0-alpha.20190606-356-gf488755,2019-06-25 22:32:46.891016+00:00,2019-06-25 22:32:52.19364+00:00,true,true
3,127.0.0.1:26261,v19.2.0-alpha.20190606-356-gf488755,2019-06-25 22:32:47.273714+00:00,2019-06-25 22:32:52.193645+00:00,true,true
4,127.0.0.1:26263,v19.2.0-alpha.20190606-356-gf488755,2019-06-25 22:32:47.675309+00:00,2019-06-25 22:32:52.277838+00:00,true,true
22:32:53 decommission.go:398: recommissioning first node (from third node)
22:32:53 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:3 -- ./cockroach node recommission 1 --insecure --port={pgport:3}
22:32:53 decommission.go:262: id is_live replicas is_decommissioning is_draining
1 true 0 false false
22:32:53 decommission.go:403: decommissioning second node from third, using --wait=all
22:32:53 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:3 -- ./cockroach node decommission --wait all --format csv 2 --insecure --port={pgport:3}
22:32:58 decommission.go:262:
id,is_live,replicas,is_decommissioning,is_draining
2,true,20,true,false
id,is_live,replicas,is_decommissioning,is_draining
2,true,19,true,false
id,is_live,replicas,is_decommissioning,is_draining
2,true,17,true,false
id,is_live,replicas,is_decommissioning,is_draining
2,true,13,true,false
id,is_live,replicas,is_decommissioning,is_draining
2,true,10,true,false
id,is_live,replicas,is_decommissioning,is_draining
2,true,7,true,false
id,is_live,replicas,is_decommissioning,is_draining
2,true,6,true,false
.
id,is_live,replicas,is_decommissioning,is_draining
2,true,5,true,false
id,is_live,replicas,is_decommissioning,is_draining
2,true,4,true,false
.
id,is_live,replicas,is_decommissioning,is_draining
2,true,2,true,false
id,is_live,replicas,is_decommissioning,is_draining
2,true,1,true,false
..........
id,is_live,replicas,is_decommissioning,is_draining
2,true,0,true,false
No more data reported on target nodes. Please verify cluster health before removing the nodes.
22:32:58 decommission.go:421: recommissioning second node from itself
22:32:58 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:2 -- ./cockroach node recommission 2 --insecure --port={pgport:2}
22:32:59 decommission.go:262: id is_live replicas is_decommissioning is_draining
2 true 0 false false
22:32:59 decommission.go:426: decommissioning third node via `quit --decommission`
22:32:59 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:3 -- ./cockroach quit --decommission --insecure --port={pgport:3}
22:33:05 decommission.go:262:
id is_live replicas is_decommissioning is_draining
3 true 18 true false
.....
id is_live replicas is_decommissioning is_draining
3 true 14 true false
id is_live replicas is_decommissioning is_draining
3 true 11 true false
id is_live replicas is_decommissioning is_draining
3 true 9 true false
..........
id is_live replicas is_decommissioning is_draining
3 true 0 true false
No more data reported on target nodes. Please verify cluster health before removing the nodes.
ok
22:33:05 decommission.go:444: checking that other nodes see node three as successfully decommissioned
22:33:05 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:2 -- ./cockroach node decommission --format csv 3 --insecure --port={pgport:2}
22:33:05 decommission.go:262:
id,is_live,replicas,is_decommissioning,is_draining
3,true,0,true,true
No more data reported on target nodes. Please verify cluster health before removing the nodes.
22:33:05 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:3
local: stopping and waiting
22:33:05 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start --env=COCKROACH_SCAN_MAX_IDLE_TIME=5ms local:3
local: starting
22:33:06 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:2 -- ./cockroach node recommission 3 --insecure --port={pgport:2}
22:33:06 decommission.go:262: id is_live replicas is_decommissioning is_draining
3 true 0 false true
22:33:06 decommission.go:475: intentionally killing first node
22:33:06 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:1
local: stopping and waiting.
22:33:07 decommission.go:477: decommission first node, starting with it down but restarting it for verification
22:33:07 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:2 -- ./cockroach node decommission --wait live 1 --insecure --port={pgport:2}
22:33:29 decommission.go:262:
--wait=live is deprecated and is treated as --wait=all
id is_live replicas is_decommissioning is_draining
1 false 21 true false
......
id is_live replicas is_decommissioning is_draining
1 false 14 true false
id is_live replicas is_decommissioning is_draining
1 false 12 true false
id is_live replicas is_decommissioning is_draining
1 false 11 true false
id is_live replicas is_decommissioning is_draining
1 false 10 true false
.
id is_live replicas is_decommissioning is_draining
1 false 9 true false
id is_live replicas is_decommissioning is_draining
1 false 8 true false
id is_live replicas is_decommissioning is_draining
1 false 6 true false
.
id is_live replicas is_decommissioning is_draining
1 false 5 true false
id is_live replicas is_decommissioning is_draining
1 false 4 true false
.
id is_live replicas is_decommissioning is_draining
1 false 3 true false
.........
id is_live replicas is_decommissioning is_draining
1 false 2 true false
..........
id is_live replicas is_decommissioning is_draining
1 false 0 true false
No more data reported on target nodes. Please verify cluster health before removing the nodes.
22:33:29 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start --env=COCKROACH_SCAN_MAX_IDLE_TIME=5ms local:1
local: starting
local: initializing cluster settings
22:33:29 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:2 -- ./cockroach node decommission --wait all --format csv 1 --insecure --port={pgport:2}
22:33:30 decommission.go:262:
id,is_live,replicas,is_decommissioning,is_draining
1,false,0,true,false
No more data reported on target nodes. Please verify cluster health before removing the nodes.
22:33:30 decommission.go:520: intentionally killing first node
22:33:30 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:1
local: stopping and waiting.
22:33:31 decommission.go:526: decommission first node in absentia using --wait=live
22:33:31 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:3 -- ./cockroach node decommission --wait live --format csv 1 --insecure --port={pgport:3}
22:33:31 decommission.go:262:
--wait=live is deprecated and is treated as --wait=all
id,is_live,replicas,is_decommissioning,is_draining
1,false,0,true,false
No more data reported on target nodes. Please verify cluster health before removing the nodes.
22:33:31 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:2 -- ./cockroach node ls --format csv --insecure --port={pgport:2}
22:33:31 decommission.go:262: id
2
3
4
22:33:31 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:2 -- ./cockroach node status --format csv --insecure --port={pgport:2}
22:33:31 decommission.go:262: id,address,build,started_at,updated_at,is_available,is_live
2,127.0.0.1:26259,v19.2.0-alpha.20190606-356-gf488755,2019-06-25 22:32:46.891016+00:00,2019-06-25 22:33:27.43496+00:00,true,true
3,127.0.0.1:26261,v19.2.0-alpha.20190606-356-gf488755,2019-06-25 22:33:06.82246+00:00,2019-06-25 22:33:29.337222+00:00,true,true
4,127.0.0.1:26263,v19.2.0-alpha.20190606-356-gf488755,2019-06-25 22:32:47.675309+00:00,2019-06-25 22:33:28.230536+00:00,true,true
22:33:31 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod wipe local:1
local: stopping and waiting
local: wiping
22:33:31 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start -a=--join 127.0.0.1:26259 local:1
local: starting
local: initializing cluster settings
SET CLUSTER SETTING
22:33:32 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:2 -- ./cockroach node status --format csv --insecure --port={pgport:2}
22:33:32 decommission.go:262: id,address,build,started_at,updated_at,is_available,is_live
2,127.0.0.1:26259,v19.2.0-alpha.20190606-356-gf488755,2019-06-25 22:32:46.891016+00:00,2019-06-25 22:33:31.938052+00:00,true,true
3,127.0.0.1:26261,v19.2.0-alpha.20190606-356-gf488755,2019-06-25 22:33:06.82246+00:00,2019-06-25 22:33:29.337222+00:00,true,true
4,127.0.0.1:26263,v19.2.0-alpha.20190606-356-gf488755,2019-06-25 22:32:47.675309+00:00,2019-06-25 22:33:28.230536+00:00,true,true
5,127.0.0.1:26257,v19.2.0-alpha.20190606-356-gf488755,2019-06-25 22:33:31.902648+00:00,2019-06-25 22:33:31.926999+00:00,true,true
22:33:32 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:2 -- ./cockroach node decommission --wait none 1 2 3 4 --insecure --port={pgport:2}
22:33:32 decommission.go:262:
id is_live replicas is_decommissioning is_draining
1 false 0 true false
2 true 16 true false
3 true 18 true false
4 true 18 true false
22:33:32 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:2 -- ./cockroach node recommission 1 2 3 4 --insecure --port={pgport:2}
22:33:32 decommission.go:262: id is_live replicas is_decommissioning is_draining
1 false 0 false false
2 true 16 false false
3 true 18 false false
4 true 18 false false
22:33:32 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:2 -- ./cockroach node decommission 1 --insecure --port={pgport:2}
22:33:32 decommission.go:262:
id is_live replicas is_decommissioning is_draining
1 false 0 true false
No more data reported on target nodes. Please verify cluster health before removing the nodes.
22:33:32 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor local --oneshot --ignore-empty-nodes
22:33:33 cluster.go:1102: running (fast) consistency checks on node 1
22:33:33 cluster.go:949: fetching logs
22:33:33 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod get local logs artifacts/acceptance/decommission/logs
local: getting logs artifacts/acceptance/decommission/logs
1: done
2: done
3: done
4: done
--- PASS: acceptance/decommission (48.04s)
##teamcity[testFinished name='acceptance/decommission' flowId='acceptance/decommission']
##teamcity[testStarted name='acceptance/cluster-init' flowId='acceptance/cluster-init']
22:33:33 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod wipe local
local: stopping and waiting.
local: wiping
22:33:34 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod put local /go/src/github.com/cockroachdb/cockroach/cockroach ./cockroach
local: putting /go/src/github.com/cockroachdb/cockroach/cockroach ./cockroach
1: done
2: done
3: done
4: done
22:33:34 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:4 -- mkdir -p {log-dir} && ./cockroach start --insecure --background --store={store-dir} --log-dir={log-dir} --cache=10% --max-sql-memory=10% --listen-addr=:{pgport:4} --http-port=$[{pgport:4}+1] --join=127.0.0.1:26257> {log-dir}/cockroach.stdout 2> {log-dir}/cockroach.stderr
22:33:34 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:3 -- mkdir -p {log-dir} && ./cockroach start --insecure --background --store={store-dir} --log-dir={log-dir} --cache=10% --max-sql-memory=10% --listen-addr=:{pgport:3} --http-port=$[{pgport:3}+1] --join=127.0.0.1:26257> {log-dir}/cockroach.stdout 2> {log-dir}/cockroach.stderr
22:33:34 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:1 -- mkdir -p {log-dir} && ./cockroach start --insecure --background --store={store-dir} --log-dir={log-dir} --cache=10% --max-sql-memory=10% --listen-addr=:{pgport:1} --http-port=$[{pgport:1}+1] > {log-dir}/cockroach.stdout 2> {log-dir}/cockroach.stderr
22:33:34 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:2 -- mkdir -p {log-dir} && ./cockroach start --insecure --background --store={store-dir} --log-dir={log-dir} --cache=10% --max-sql-memory=10% --listen-addr=:{pgport:2} --http-port=$[{pgport:2}+1] --join=127.0.0.1:26257> {log-dir}/cockroach.stdout 2> {log-dir}/cockroach.stderr
[ 4] acceptance/cluster-init: uploading binary (4s)
22:33:52 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod wipe local
local: stopping and waiting.
local: wiping
22:33:53 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:2 -- mkdir -p {log-dir} && ./cockroach start --insecure --background --store={store-dir} --log-dir={log-dir} --cache=10% --max-sql-memory=10% --listen-addr=:{pgport:2} --http-port=$[{pgport:2}+1] --join=127.0.0.1:26257,127.0.0.1:26259,127.0.0.1:26261,127.0.0.1:26263> {log-dir}/cockroach.stdout 2> {log-dir}/cockroach.stderr
22:33:53 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:3 -- mkdir -p {log-dir} && ./cockroach start --insecure --background --store={store-dir} --log-dir={log-dir} --cache=10% --max-sql-memory=10% --listen-addr=:{pgport:3} --http-port=$[{pgport:3}+1] --join=127.0.0.1:26257,127.0.0.1:26259,127.0.0.1:26261,127.0.0.1:26263> {log-dir}/cockroach.stdout 2> {log-dir}/cockroach.stderr
22:33:53 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:4 -- mkdir -p {log-dir} && ./cockroach start --insecure --background --store={store-dir} --log-dir={log-dir} --cache=10% --max-sql-memory=10% --listen-addr=:{pgport:4} --http-port=$[{pgport:4}+1] --join=127.0.0.1:26257,127.0.0.1:26259,127.0.0.1:26261,127.0.0.1:26263> {log-dir}/cockroach.stdout 2> {log-dir}/cockroach.stderr
22:33:53 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:1 -- mkdir -p {log-dir} && ./cockroach start --insecure --background --store={store-dir} --log-dir={log-dir} --cache=10% --max-sql-memory=10% --listen-addr=:{pgport:1} --http-port=$[{pgport:1}+1] --join=127.0.0.1:26257,127.0.0.1:26259,127.0.0.1:26261,127.0.0.1:26263> {log-dir}/cockroach.stdout 2> {log-dir}/cockroach.stderr
22:33:55 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:1 -- ./cockroach init --insecure --port={pgport:1}
Cluster successfully initialized
22:34:14 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:1 -- ./cockroach init --insecure --port={pgport:1}
22:34:14 cluster_init.go:199: E190625 22:34:14.107955 1 cli/error.go:232 rpc error: code = Unknown desc = cluster has already been initialized
Error: rpc error: code = Unknown desc = cluster has already been initialized
Failed running "init"
Error: exit status 1
22:34:14 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod wipe local
local: stopping and waiting.
local: wiping
22:34:15 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:2 -- mkdir -p {log-dir} && ./cockroach start --insecure --background --store={store-dir} --log-dir={log-dir} --cache=10% --max-sql-memory=10% --listen-addr=:{pgport:2} --http-port=$[{pgport:2}+1] --join=127.0.0.1:26257,127.0.0.1:26259,127.0.0.1:26261,127.0.0.1:26263> {log-dir}/cockroach.stdout 2> {log-dir}/cockroach.stderr
22:34:15 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:3 -- mkdir -p {log-dir} && ./cockroach start --insecure --background --store={store-dir} --log-dir={log-dir} --cache=10% --max-sql-memory=10% --listen-addr=:{pgport:3} --http-port=$[{pgport:3}+1] --join=127.0.0.1:26257,127.0.0.1:26259,127.0.0.1:26261,127.0.0.1:26263> {log-dir}/cockroach.stdout 2> {log-dir}/cockroach.stderr
22:34:15 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:1 -- mkdir -p {log-dir} && ./cockroach start --insecure --background --store={store-dir} --log-dir={log-dir} --cache=10% --max-sql-memory=10% --listen-addr=:{pgport:1} --http-port=$[{pgport:1}+1] --join=127.0.0.1:26257,127.0.0.1:26259,127.0.0.1:26261,127.0.0.1:26263> {log-dir}/cockroach.stdout 2> {log-dir}/cockroach.stderr
22:34:15 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:4 -- mkdir -p {log-dir} && ./cockroach start --insecure --background --store={store-dir} --log-dir={log-dir} --cache=10% --max-sql-memory=10% --listen-addr=:{pgport:4} --http-port=$[{pgport:4}+1] --join=127.0.0.1:26257,127.0.0.1:26259,127.0.0.1:26261,127.0.0.1:26263> {log-dir}/cockroach.stdout 2> {log-dir}/cockroach.stderr
22:34:16 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:2 -- ./cockroach init --insecure --port={pgport:2}
Cluster successfully initialized
22:34:35 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:2 -- ./cockroach init --insecure --port={pgport:2}
22:34:35 cluster_init.go:199: E190625 22:34:35.757624 1 cli/error.go:232 rpc error: code = Unknown desc = cluster has already been initialized
Error: rpc error: code = Unknown desc = cluster has already been initialized
Failed running "init"
Error: exit status 1
22:34:35 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor local --oneshot --ignore-empty-nodes
22:34:36 cluster.go:1102: running (fast) consistency checks on node 1
22:34:36 cluster.go:949: fetching logs
22:34:36 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod get local logs artifacts/acceptance/cluster-init/logs
local: getting logs artifacts/acceptance/cluster-init/logs
1: done
2: done
3: done
4: done
--- PASS: acceptance/cluster-init (63.12s)
##teamcity[testFinished name='acceptance/cluster-init' flowId='acceptance/cluster-init']
##teamcity[testStarted name='acceptance/event-log' flowId='acceptance/event-log']
22:34:36 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod wipe local
local: stopping and waiting.
local: wiping
22:34:37 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod put local /go/src/github.com/cockroachdb/cockroach/cockroach ./cockroach
local: putting /go/src/github.com/cockroachdb/cockroach/cockroach ./cockroach
1: done
2: done
3: done
4: done
22:34:37 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local
local: starting[ 5] acceptance/event-log: starting cluster (1s)
.
local: initializing cluster settings
SET CLUSTER SETTING
22:34:58 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:3
local: stopping and waiting.
22:35:00 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:3
local: starting
22:35:01 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor local --oneshot --ignore-empty-nodes
22:35:01 cluster.go:1102: running (fast) consistency checks on node 1
22:35:01 cluster.go:949: fetching logs
22:35:01 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod get local logs artifacts/acceptance/event-log/logs
local: getting logs artifacts/acceptance/event-log/logs
1: done
2: done
3: done
4: done
--- PASS: acceptance/event-log (25.49s)
##teamcity[testFinished name='acceptance/event-log' flowId='acceptance/event-log']
##teamcity[testStarted name='acceptance/gossip/peerings' flowId='acceptance/gossip/peerings']
22:35:01 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod wipe local
local: stopping and waiting.
local: wiping
22:35:02 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod put local /go/src/github.com/cockroachdb/cockroach/cockroach ./cockroach
local: putting /go/src/github.com/cockroachdb/cockroach/cockroach ./cockroach
1: done
2: done
3: done
4: done
22:35:02 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local
local: starting.
local: initializing cluster settings
SET CLUSTER SETTING
22:35:04 gossip.go:263: 1: OK
22:35:04 gossip.go:267: 1: restarting node 4
22:35:04 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:4
local: stopping and waiting.
22:35:05 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:4
local: starting
22:35:06 gossip.go:263: 2: OK
22:35:06 gossip.go:267: 2: restarting node 4
22:35:06 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:4
local: stopping and waiting.
22:35:07 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:4
local: starting
22:35:08 gossip.go:263: 3: OK
22:35:08 gossip.go:267: 3: restarting node 4
22:35:08 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:4
local: stopping and waiting.
22:35:09 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:4
local: starting
22:35:10 gossip.go:263: 4: OK
22:35:10 gossip.go:267: 4: restarting node 1
22:35:10 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:1
local: stopping and waiting.
22:35:11 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:1
local: starting
local: initializing cluster settings
22:35:18 gossip.go:263: 5: OK
22:35:18 gossip.go:267: 5: restarting node 4
22:35:18 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:4
local: stopping and waiting.
22:35:19 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:4
local: starting
22:35:20 gossip.go:263: 6: OK
22:35:20 gossip.go:267: 6: restarting node 4
22:35:20 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:4
local: stopping and waiting.
22:35:21 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:4
local: starting
22:35:22 gossip.go:263: 7: OK
22:35:22 gossip.go:267: 7: restarting node 1
22:35:22 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:1
local: stopping and waiting.
22:35:23 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:1
local: starting
local: initializing cluster settings
22:35:26 gossip.go:263: 8: OK
22:35:26 gossip.go:267: 8: restarting node 4
22:35:26 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:4
local: stopping and waiting.
22:35:27 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:4
local: starting
22:35:28 gossip.go:263: 9: OK
22:35:28 gossip.go:267: 9: restarting node 1
22:35:28 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:1
local: stopping and waiting.
22:35:29 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:1
local: starting
local: initializing cluster settings
22:35:33 gossip.go:263: 10: OK
22:35:33 gossip.go:267: 10: restarting node 3
22:35:33 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:3
local: stopping and waiting.
22:35:34 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:3
local: starting
22:35:35 gossip.go:263: 11: OK
22:35:35 gossip.go:267: 11: restarting node 3
22:35:35 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:3
local: stopping and waiting.
22:35:36 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:3
local: starting
22:35:37 gossip.go:263: 12: OK
22:35:37 gossip.go:267: 12: restarting node 1
22:35:37 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:1
local: stopping and waiting[ 6] acceptance/gossip/peerings: stopping cluster (0s)
.
22:35:38 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:1
local: starting
local: initializing cluster settings
22:35:45 gossip.go:263: 13: OK
22:35:45 gossip.go:267: 13: restarting node 1
22:35:45 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:1
local: stopping and waiting.
22:35:46 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:1
local: starting
local: initializing cluster settings
22:35:52 gossip.go:263: 14: OK
22:35:52 gossip.go:267: 14: restarting node 3
22:35:52 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:3
local: stopping and waiting.
22:35:53 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:3
local: starting
22:35:54 gossip.go:263: 15: OK
22:35:54 gossip.go:267: 15: restarting node 4
22:35:54 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:4
local: stopping and waiting.
22:35:55 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:4
local: starting
22:35:56 gossip.go:263: 16: OK
22:35:56 gossip.go:267: 16: restarting node 3
22:35:56 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:3
local: stopping and waiting.
22:35:57 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:3
local: starting
22:35:58 gossip.go:263: 17: OK
22:35:58 gossip.go:267: 17: restarting node 4
22:35:58 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:4
local: stopping and waiting.
22:35:59 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:4
local: starting
22:36:00 gossip.go:263: 18: OK
22:36:00 gossip.go:267: 18: restarting node 1
22:36:00 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:1
local: stopping and waiting.
22:36:01 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:1
local: starting
local: initializing cluster settings
22:36:05 gossip.go:263: 19: OK
22:36:05 gossip.go:267: 19: restarting node 1
22:36:05 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:1
local: stopping and waiting.
22:36:06 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:1
local: starting
local: initializing cluster settings
22:36:06 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor local --oneshot --ignore-empty-nodes
22:36:10 cluster.go:1102: running (fast) consistency checks on node 1
22:36:10 cluster.go:949: fetching logs
22:36:10 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod get local logs artifacts/acceptance/gossip/peerings/logs
local: getting logs artifacts/acceptance/gossip/peerings/logs
1: done
2: done
3: done
4: done
--- PASS: acceptance/gossip/peerings (69.30s)
##teamcity[testFinished name='acceptance/gossip/peerings' flowId='acceptance/gossip/peerings']
##teamcity[testStarted name='acceptance/gossip/restart' flowId='acceptance/gossip/restart']
22:36:10 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod wipe local
local: stopping and waiting.
local: wiping
22:36:12 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod put local /go/src/github.com/cockroachdb/cockroach/cockroach ./cockroach
local: putting /go/src/github.com/cockroachdb/cockroach/cockroach ./cockroach
1: done
2: done
3: done
4: done
22:36:12 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local
local: starting.
local: initializing cluster settings
SET CLUSTER SETTING
22:36:13 gossip.go:199: waiting for gossip to be connected
22:36:14 gossip.go:287: 1: OK
22:36:14 gossip.go:289: 1: killing all nodes
22:36:14 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local
local: stopping and waiting.
22:36:15 gossip.go:292: 1: restarting all nodes
22:36:15 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local
local: starting.
local: initializing cluster settings
22:36:16 gossip.go:199: waiting for gossip to be connected
22:36:21 gossip.go:287: 2: OK
22:36:21 gossip.go:289: 2: killing all nodes
22:36:21 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local
local: stopping and waiting.
22:36:22 gossip.go:292: 2: restarting all nodes
22:36:22 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local
local: starting.
local: initializing cluster settings
22:36:23 gossip.go:199: waiting for gossip to be connected
22:36:27 gossip.go:287: 3: OK
22:36:27 gossip.go:289: 3: killing all nodes
22:36:27 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local
local: stopping and waiting.
22:36:28 gossip.go:292: 3: restarting all nodes
22:36:28 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local
local: starting.
local: initializing cluster settings
22:36:30 gossip.go:199: waiting for gossip to be connected
22:36:34 gossip.go:287: 4: OK
22:36:34 gossip.go:289: 4: killing all nodes
22:36:34 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local
local: stopping and waiting.
22:36:35 gossip.go:292: 4: restarting all nodes
22:36:35 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local
local: starting.
local: initializing cluster settings
22:36:36 gossip.go:199: waiting for gossip to be connected
[ 7] acceptance/gossip/restart: ??? (0s)
22:36:40 gossip.go:287: 5: OK
22:36:40 gossip.go:289: 5: killing all nodes
22:36:40 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local
local: stopping and waiting.
22:36:41 gossip.go:292: 5: restarting all nodes
22:36:41 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local
local: starting.
local: initializing cluster settings
22:36:43 gossip.go:199: waiting for gossip to be connected
22:36:46 gossip.go:287: 6: OK
22:36:46 gossip.go:289: 6: killing all nodes
22:36:46 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local
local: stopping and waiting.
22:36:47 gossip.go:292: 6: restarting all nodes
22:36:47 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local
local: starting.
local: initializing cluster settings
22:36:49 gossip.go:199: waiting for gossip to be connected
22:36:53 gossip.go:287: 7: OK
22:36:53 gossip.go:289: 7: killing all nodes
22:36:53 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local
local: stopping and waiting.
22:36:54 gossip.go:292: 7: restarting all nodes
22:36:54 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local
local: starting.
local: initializing cluster settings
22:36:56 gossip.go:199: waiting for gossip to be connected
22:37:00 gossip.go:287: 8: OK
22:37:00 gossip.go:289: 8: killing all nodes
22:37:00 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local
local: stopping and waiting.
22:37:01 gossip.go:292: 8: restarting all nodes
22:37:01 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local
local: starting.
local: initializing cluster settings
22:37:03 gossip.go:199: waiting for gossip to be connected
22:37:08 gossip.go:287: 9: OK
22:37:08 gossip.go:289: 9: killing all nodes
22:37:08 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local
local: stopping and waiting.
22:37:09 gossip.go:292: 9: restarting all nodes
22:37:09 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local
local: starting.
local: initializing cluster settings
22:37:11 gossip.go:199: waiting for gossip to be connected
22:37:15 gossip.go:287: 10: OK
22:37:15 gossip.go:289: 10: killing all nodes
22:37:15 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local
local: stopping and waiting.
22:37:16 gossip.go:292: 10: restarting all nodes
22:37:16 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local
local: starting.
local: initializing cluster settings
22:37:18 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor local --oneshot --ignore-empty-nodes
22:37:23 cluster.go:1102: running (fast) consistency checks on node 1
22:37:23 cluster.go:949: fetching logs
22:37:23 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod get local logs artifacts/acceptance/gossip/restart/logs
local: getting logs artifacts/acceptance/gossip/restart/logs
1: done
2: done
3: done
4: done
--- PASS: acceptance/gossip/restart (72.28s)
##teamcity[testFinished name='acceptance/gossip/restart' flowId='acceptance/gossip/restart']
##teamcity[testStarted name='acceptance/gossip/restart-node-one' flowId='acceptance/gossip/restart-node-one']
22:37:23 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod wipe local
local: stopping and waiting.
local: wiping
22:37:24 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod put local /go/src/github.com/cockroachdb/cockroach/cockroach ./cockroach
local: putting /go/src/github.com/cockroachdb/cockroach/cockroach ./cockroach
1: done
2: done
3: done
4: done
22:37:24 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start --racks=4 --env=COCKROACH_SCAN_MAX_IDLE_TIME=5ms --encrypt=false local
local: starting.
local: initializing cluster settings
SET CLUSTER SETTING
22:37:26 gossip.go:308: ALTER RANGE default CONFIGURE ZONE = 'constraints: {"-rack=0"}'
22:37:26 gossip.go:308: ALTER RANGE system CONFIGURE ZONE = 'constraints: {"-rack=0"}'
22:37:26 gossip.go:308: ALTER DATABASE system CONFIGURE ZONE = 'constraints: {"-rack=0"}'
22:37:26 gossip.go:308: ALTER RANGE meta CONFIGURE ZONE = 'constraints: {"-rack=0"}'
22:37:26 gossip.go:308: ALTER RANGE liveness CONFIGURE ZONE = 'constraints: {"-rack=0"}'
22:37:26 gossip.go:369: node 1 still has 18 replicas
22:37:26 gossip.go:369: node 1 still has 17 replicas
22:37:26 gossip.go:369: node 1 still has 16 replicas
22:37:26 gossip.go:369: node 1 still has 15 replicas
22:37:26 gossip.go:369: node 1 still has 14 replicas
22:37:26 gossip.go:369: node 1 still has 13 replicas
22:37:26 gossip.go:369: node 1 still has 11 replicas
22:37:26 gossip.go:369: node 1 still has 9 replicas
22:37:26 gossip.go:369: node 1 still has 7 replicas
22:37:26 gossip.go:369: node 1 still has 5 replicas
22:37:26 gossip.go:369: node 1 still has 3 replicas
22:37:26 gossip.go:378: killing all nodes
22:37:26 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local
local: stopping and waiting.
22:37:27 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:1 -- ./cockroach start --insecure --background --store={store-dir} --log-dir={log-dir} --cache=10% --max-sql-memory=10% --listen-addr=:$[{pgport:1}+10000] --http-port=$[{pgport:1}+1] > {log-dir}/cockroach.stdout 2> {log-dir}/cockroach.stderr
22:37:28 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start --env=COCKROACH_SCAN_MAX_IDLE_TIME=5ms --encrypt=false local:2-4
local: starting.
22:37:29 gossip.go:199: waiting for gossip to be connected
22:37:34 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:1
local: stopping and waiting.
22:37:35 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local:1
local: starting
local: initializing cluster settings
22:37:36 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor local --oneshot --ignore-empty-nodes
22:37:37 cluster.go:1102: running (fast) consistency checks on node 1
[ 8] acceptance/gossip/restart-node-one: ??? (0s)
22:37:38 cluster.go:949: fetching logs
22:37:38 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod get local logs artifacts/acceptance/gossip/restart-node-one/logs
local: getting logs artifacts/acceptance/gossip/restart-node-one/logs
1: done
2: done
3: done
4: done
--- PASS: acceptance/gossip/restart-node-one (15.04s)
##teamcity[testFinished name='acceptance/gossip/restart-node-one' flowId='acceptance/gossip/restart-node-one']
##teamcity[testStarted name='acceptance/gossip/locality-address' flowId='acceptance/gossip/locality-address']
22:37:38 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod wipe local
local: stopping and waiting.
local: wiping
22:37:39 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod put local /go/src/github.com/cockroachdb/cockroach/cockroach ./cockroach
local: putting /go/src/github.com/cockroachdb/cockroach/cockroach ./cockroach
1: done
2: done
3: done
4: done
22:37:39 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start --racks=1 --args=--locality-advertise-addr=rack=0@localhost local:1
local: starting
local: initializing cluster settings
SET CLUSTER SETTING
22:37:40 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start --racks=1 --args=--locality-advertise-addr=rack=0@localhost local:2
local: starting
22:37:40 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start --racks=1 --args=--locality-advertise-addr=rack=0@localhost local:3
local: starting
22:37:41 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start --racks=1 --args=--locality-advertise-addr=rack=0@localhost local:4
local: starting
22:37:41 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor local --oneshot --ignore-empty-nodes
22:37:41 cluster.go:1102: running (fast) consistency checks on node 1
22:37:41 cluster.go:949: fetching logs
22:37:41 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod get local logs artifacts/acceptance/gossip/locality-address/logs
local: getting logs artifacts/acceptance/gossip/locality-address/logs
1: done
2: done
3: done
4: done
--- PASS: acceptance/gossip/locality-address (3.68s)
##teamcity[testFinished name='acceptance/gossip/locality-address' flowId='acceptance/gossip/locality-address']
##teamcity[testStarted name='acceptance/rapid-restart' flowId='acceptance/rapid-restart']
22:37:41 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod wipe local
local: stopping and waiting.
local: wiping
22:37:43 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod put local:1 /go/src/github.com/cockroachdb/cockroach/cockroach ./cockroach
local: putting /go/src/github.com/cockroachdb/cockroach/cockroach ./cockroach
1: done
22:37:43 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod wipe local:1
local: stopping and waiting
local: wiping
22:37:43 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:1 -- mkdir -p {log-dir} && ./cockroach start --insecure --store={store-dir} --log-dir={log-dir} --cache=10% --max-sql-memory=10% --listen-addr=:{pgport:1} --http-port=$[{pgport:1}+1] > {log-dir}/cockroach.stdout 2> {log-dir}/cockroach.stderr
22:37:44 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop --sig=2 local:1
local: stopping
Error: exit status 1
22:37:47 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:1 -- mkdir -p {log-dir} && ./cockroach start --insecure --store={store-dir} --log-dir={log-dir} --cache=10% --max-sql-memory=10% --listen-addr=:{pgport:1} --http-port=$[{pgport:1}+1] > {log-dir}/cockroach.stdout 2> {log-dir}/cockroach.stderr
22:37:47 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop --sig=9 local:1
local: stopping and waiting/bin/bash: line 1: 18324 Killed bash -c "mkdir -p /home/roach/local/1/logs && ./cockroach start --insecure --store=/home/roach/local/1/data --log-dir=/home/roach/local/1/logs --cache=10% --max-sql-memory=10% --listen-addr=:26257 --http-port=\$[26257+1] > /home/roach/local/1/logs/cockroach.stdout 2> /home/roach/local/1/logs/cockroach.stderr"
Error: exit status 137
.
22:37:50 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:1 -- mkdir -p {log-dir} && ./cockroach start --insecure --store={store-dir} --log-dir={log-dir} --cache=10% --max-sql-memory=10% --listen-addr=:{pgport:1} --http-port=$[{pgport:1}+1] > {log-dir}/cockroach.stdout 2> {log-dir}/cockroach.stderr
22:37:50 rapid_restart.go:108: 1 OK
22:37:50 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod wipe local:1
local: stopping and waiting/bin/bash: line 1: 18392 Killed bash -c "mkdir -p /home/roach/local/1/logs && ./cockroach start --insecure --store=/home/roach/local/1/data --log-dir=/home/roach/local/1/logs --cache=10% --max-sql-memory=10% --listen-addr=:26257 --http-port=\$[26257+1] > /home/roach/local/1/logs/cockroach.stdout 2> /home/roach/local/1/logs/cockroach.stderr"
Error: exit status 137
.
local: wiping
22:37:51 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:1 -- mkdir -p {log-dir} && ./cockroach start --insecure --store={store-dir} --log-dir={log-dir} --cache=10% --max-sql-memory=10% --listen-addr=:{pgport:1} --http-port=$[{pgport:1}+1] > {log-dir}/cockroach.stdout 2> {log-dir}/cockroach.stderr
22:37:52 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop --sig=2 local:1
local: stopping
Error: exit status 1
22:37:55 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:1 -- mkdir -p {log-dir} && ./cockroach start --insecure --store={store-dir} --log-dir={log-dir} --cache=10% --max-sql-memory=10% --listen-addr=:{pgport:1} --http-port=$[{pgport:1}+1] > {log-dir}/cockroach.stdout 2> {log-dir}/cockroach.stderr
22:37:55 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop --sig=9 local:1
local: stopping and waiting/bin/bash: line 1: 18625 Killed bash -c "mkdir -p /home/roach/local/1/logs && ./cockroach start --insecure --store=/home/roach/local/1/data --log-dir=/home/roach/local/1/logs --cache=10% --max-sql-memory=10% --listen-addr=:26257 --http-port=\$[26257+1] > /home/roach/local/1/logs/cockroach.stdout 2> /home/roach/local/1/logs/cockroach.stderr"
Error: exit status 137
.
22:37:58 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:1 -- mkdir -p {log-dir} && ./cockroach start --insecure --store={store-dir} --log-dir={log-dir} --cache=10% --max-sql-memory=10% --listen-addr=:{pgport:1} --http-port=$[{pgport:1}+1] > {log-dir}/cockroach.stdout 2> {log-dir}/cockroach.stderr
22:37:59 rapid_restart.go:108: 2 OK
22:37:59 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod wipe local:1
local: stopping and waiting/bin/bash: line 1: 18700 Killed bash -c "mkdir -p /home/roach/local/1/logs && ./cockroach start --insecure --store=/home/roach/local/1/data --log-dir=/home/roach/local/1/logs --cache=10% --max-sql-memory=10% --listen-addr=:26257 --http-port=\$[26257+1] > /home/roach/local/1/logs/cockroach.stdout 2> /home/roach/local/1/logs/cockroach.stderr"
Error: exit status 137
.
local: wiping
22:38:00 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:1 -- mkdir -p {log-dir} && ./cockroach start --insecure --store={store-dir} --log-dir={log-dir} --cache=10% --max-sql-memory=10% --listen-addr=:{pgport:1} --http-port=$[{pgport:1}+1] > {log-dir}/cockroach.stdout 2> {log-dir}/cockroach.stderr
22:38:00 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop --sig=2 local:1
local: stopping
Error: exit status 1
22:38:03 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:1 -- mkdir -p {log-dir} && ./cockroach start --insecure --store={store-dir} --log-dir={log-dir} --cache=10% --max-sql-memory=10% --listen-addr=:{pgport:1} --http-port=$[{pgport:1}+1] > {log-dir}/cockroach.stdout 2> {log-dir}/cockroach.stderr
22:38:04 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop --sig=2 local:1
local: stopping
Error: exit status 1
22:38:07 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:1 -- mkdir -p {log-dir} && ./cockroach start --insecure --store={store-dir} --log-dir={log-dir} --cache=10% --max-sql-memory=10% --listen-addr=:{pgport:1} --http-port=$[{pgport:1}+1] > {log-dir}/cockroach.stdout 2> {log-dir}/cockroach.stderr
22:38:07 rapid_restart.go:108: 3 OK
22:38:07 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod wipe local:1
local: stopping and waiting/bin/bash: line 1: 19001 Killed bash -c "mkdir -p /home/roach/local/1/logs && ./cockroach start --insecure --store=/home/roach/local/1/data --log-dir=/home/roach/local/1/logs --cache=10% --max-sql-memory=10% --listen-addr=:26257 --http-port=\$[26257+1] > /home/roach/local/1/logs/cockroach.stdout 2> /home/roach/local/1/logs/cockroach.stderr"
Error: exit status 137
.
local: wiping
22:38:08 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:1 -- mkdir -p {log-dir} && ./cockroach start --insecure --store={store-dir} --log-dir={log-dir} --cache=10% --max-sql-memory=10% --listen-addr=:{pgport:1} --http-port=$[{pgport:1}+1] > {log-dir}/cockroach.stdout 2> {log-dir}/cockroach.stderr
22:38:09 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop --sig=2 local:1
local: stopping
Error: exit status 1
22:38:12 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:1 -- mkdir -p {log-dir} && ./cockroach start --insecure --store={store-dir} --log-dir={log-dir} --cache=10% --max-sql-memory=10% --listen-addr=:{pgport:1} --http-port=$[{pgport:1}+1] > {log-dir}/cockroach.stdout 2> {log-dir}/cockroach.stderr
22:38:12 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop --sig=2 local:1
local: stopping
Error: exit status 1
22:38:15 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:1 -- mkdir -p {log-dir} && ./cockroach start --insecure --store={store-dir} --log-dir={log-dir} --cache=10% --max-sql-memory=10% --listen-addr=:{pgport:1} --http-port=$[{pgport:1}+1] > {log-dir}/cockroach.stdout 2> {log-dir}/cockroach.stderr
22:38:15 rapid_restart.go:108: 4 OK
22:38:15 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod wipe local:1
local: stopping and waiting/bin/bash: line 1: 19299 Killed bash -c "mkdir -p /home/roach/local/1/logs && ./cockroach start --insecure --store=/home/roach/local/1/data --log-dir=/home/roach/local/1/logs --cache=10% --max-sql-memory=10% --listen-addr=:26257 --http-port=\$[26257+1] > /home/roach/local/1/logs/cockroach.stdout 2> /home/roach/local/1/logs/cockroach.stderr"
Error: exit status 137
.
local: wiping
22:38:17 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:1 -- mkdir -p {log-dir} && ./cockroach start --insecure --store={store-dir} --log-dir={log-dir} --cache=10% --max-sql-memory=10% --listen-addr=:{pgport:1} --http-port=$[{pgport:1}+1] > {log-dir}/cockroach.stdout 2> {log-dir}/cockroach.stderr
22:38:17 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop --sig=9 local:1
local: stopping and waiting/bin/bash: line 1: 19473 Killed bash -c "mkdir -p /home/roach/local/1/logs && ./cockroach start --insecure --store=/home/roach/local/1/data --log-dir=/home/roach/local/1/logs --cache=10% --max-sql-memory=10% --listen-addr=:26257 --http-port=\$[26257+1] > /home/roach/local/1/logs/cockroach.stdout 2> /home/roach/local/1/logs/cockroach.stderr"
Error: exit status 137
.
22:38:20 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:1 -- mkdir -p {log-dir} && ./cockroach start --insecure --store={store-dir} --log-dir={log-dir} --cache=10% --max-sql-memory=10% --listen-addr=:{pgport:1} --http-port=$[{pgport:1}+1] > {log-dir}/cockroach.stdout 2> {log-dir}/cockroach.stderr
22:38:21 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop --sig=9 local:1
local: stopping and waiting/bin/bash: line 1: 19537 Killed bash -c "mkdir -p /home/roach/local/1/logs && ./cockroach start --insecure --store=/home/roach/local/1/data --log-dir=/home/roach/local/1/logs --cache=10% --max-sql-memory=10% --listen-addr=:26257 --http-port=\$[26257+1] > /home/roach/local/1/logs/cockroach.stdout 2> /home/roach/local/1/logs/cockroach.stderr"
Error: exit status 137
.
22:38:24 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:1 -- mkdir -p {log-dir} && ./cockroach start --insecure --store={store-dir} --log-dir={log-dir} --cache=10% --max-sql-memory=10% --listen-addr=:{pgport:1} --http-port=$[{pgport:1}+1] > {log-dir}/cockroach.stdout 2> {log-dir}/cockroach.stderr
22:38:25 rapid_restart.go:108: 5 OK
22:38:25 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod wipe local:1
local: stopping and waiting/bin/bash: line 1: 19613 Killed bash -c "mkdir -p /home/roach/local/1/logs && ./cockroach start --insecure --store=/home/roach/local/1/data --log-dir=/home/roach/local/1/logs --cache=10% --max-sql-memory=10% --listen-addr=:26257 --http-port=\$[26257+1] > /home/roach/local/1/logs/cockroach.stdout 2> /home/roach/local/1/logs/cockroach.stderr"
Error: exit status 137
.
local: wiping
22:38:26 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:1 -- mkdir -p {log-dir} && ./cockroach start --insecure --store={store-dir} --log-dir={log-dir} --cache=10% --max-sql-memory=10% --listen-addr=:{pgport:1} --http-port=$[{pgport:1}+1] > {log-dir}/cockroach.stdout 2> {log-dir}/cockroach.stderr
22:38:27 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop --sig=9 local:1
local: stopping and waiting/bin/bash: line 1: 19785 Killed bash -c "mkdir -p /home/roach/local/1/logs && ./cockroach start --insecure --store=/home/roach/local/1/data --log-dir=/home/roach/local/1/logs --cache=10% --max-sql-memory=10% --listen-addr=:26257 --http-port=\$[26257+1] > /home/roach/local/1/logs/cockroach.stdout 2> /home/roach/local/1/logs/cockroach.stderr"
Error: exit status 137
.
22:38:30 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:1 -- mkdir -p {log-dir} && ./cockroach start --insecure --store={store-dir} --log-dir={log-dir} --cache=10% --max-sql-memory=10% --listen-addr=:{pgport:1} --http-port=$[{pgport:1}+1] > {log-dir}/cockroach.stdout 2> {log-dir}/cockroach.stderr
22:38:30 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop --sig=2 local:1
local: stopping
Error: exit status 1
22:38:33 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:1 -- mkdir -p {log-dir} && ./cockroach start --insecure --store={store-dir} --log-dir={log-dir} --cache=10% --max-sql-memory=10% --listen-addr=:{pgport:1} --http-port=$[{pgport:1}+1] > {log-dir}/cockroach.stdout 2> {log-dir}/cockroach.stderr
22:38:33 rapid_restart.go:108: 6 OK
22:38:33 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod wipe local:1
local: stopping and waiting/bin/bash: line 1: 19916 Killed bash -c "mkdir -p /home/roach/local/1/logs && ./cockroach start --insecure --store=/home/roach/local/1/data --log-dir=/home/roach/local/1/logs --cache=10% --max-sql-memory=10% --listen-addr=:26257 --http-port=\$[26257+1] > /home/roach/local/1/logs/cockroach.stdout 2> /home/roach/local/1/logs/cockroach.stderr"
Error: exit status 137
.
local: wiping
22:38:34 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:1 -- mkdir -p {log-dir} && ./cockroach start --insecure --store={store-dir} --log-dir={log-dir} --cache=10% --max-sql-memory=10% --listen-addr=:{pgport:1} --http-port=$[{pgport:1}+1] > {log-dir}/cockroach.stdout 2> {log-dir}/cockroach.stderr
22:38:34 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop --sig=9 local:1
local: stopping and waiting/bin/bash: line 1: 20100 Killed bash -c "mkdir -p /home/roach/local/1/logs && ./cockroach start --insecure --store=/home/roach/local/1/data --log-dir=/home/roach/local/1/logs --cache=10% --max-sql-memory=10% --listen-addr=:26257 --http-port=\$[26257+1] > /home/roach/local/1/logs/cockroach.stdout 2> /home/roach/local/1/logs/cockroach.stderr"
Error: exit status 137
.
[ 9] acceptance/rapid-restart: ??? (0s)
22:38:37 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:1 -- mkdir -p {log-dir} && ./cockroach start --insecure --store={store-dir} --log-dir={log-dir} --cache=10% --max-sql-memory=10% --listen-addr=:{pgport:1} --http-port=$[{pgport:1}+1] > {log-dir}/cockroach.stdout 2> {log-dir}/cockroach.stderr
22:38:38 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop --sig=9 local:1
local: stopping and waiting/bin/bash: line 1: 20152 Killed bash -c "mkdir -p /home/roach/local/1/logs && ./cockroach start --insecure --store=/home/roach/local/1/data --log-dir=/home/roach/local/1/logs --cache=10% --max-sql-memory=10% --listen-addr=:26257 --http-port=\$[26257+1] > /home/roach/local/1/logs/cockroach.stdout 2> /home/roach/local/1/logs/cockroach.stderr"
Error: exit status 137
.
22:38:41 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:1 -- mkdir -p {log-dir} && ./cockroach start --insecure --store={store-dir} --log-dir={log-dir} --cache=10% --max-sql-memory=10% --listen-addr=:{pgport:1} --http-port=$[{pgport:1}+1] > {log-dir}/cockroach.stdout 2> {log-dir}/cockroach.stderr
22:38:41 rapid_restart.go:108: 7 OK
22:38:41 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod wipe local:1
local: stopping and waiting/bin/bash: line 1: 20226 Killed bash -c "mkdir -p /home/roach/local/1/logs && ./cockroach start --insecure --store=/home/roach/local/1/data --log-dir=/home/roach/local/1/logs --cache=10% --max-sql-memory=10% --listen-addr=:26257 --http-port=\$[26257+1] > /home/roach/local/1/logs/cockroach.stdout 2> /home/roach/local/1/logs/cockroach.stderr"
Error: exit status 137
.
local: wiping
22:38:42 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:1 -- mkdir -p {log-dir} && ./cockroach start --insecure --store={store-dir} --log-dir={log-dir} --cache=10% --max-sql-memory=10% --listen-addr=:{pgport:1} --http-port=$[{pgport:1}+1] > {log-dir}/cockroach.stdout 2> {log-dir}/cockroach.stderr
22:38:42 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop --sig=2 local:1
local: stopping
Error: exit status 1
22:38:45 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:1 -- mkdir -p {log-dir} && ./cockroach start --insecure --store={store-dir} --log-dir={log-dir} --cache=10% --max-sql-memory=10% --listen-addr=:{pgport:1} --http-port=$[{pgport:1}+1] > {log-dir}/cockroach.stdout 2> {log-dir}/cockroach.stderr
22:38:46 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop --sig=2 local:1
local: stopping
Error: exit status 1
22:38:49 rapid_restart.go:108: 8 OK
22:38:49 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:1
22:38:49 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:1 -- mkdir -p {log-dir} && ./cockroach start --insecure --store={store-dir} --log-dir={log-dir} --cache=10% --max-sql-memory=10% --listen-addr=:{pgport:1} --http-port=$[{pgport:1}+1] > {log-dir}/cockroach.stdout 2> {log-dir}/cockroach.stderr
local: stopping and waiting/bin/bash: line 1: 20530 Killed bash -c "mkdir -p /home/roach/local/1/logs && ./cockroach start --insecure --store=/home/roach/local/1/data --log-dir=/home/roach/local/1/logs --cache=10% --max-sql-memory=10% --listen-addr=:26257 --http-port=\$[26257+1] > /home/roach/local/1/logs/cockroach.stdout 2> /home/roach/local/1/logs/cockroach.stderr"
Error: exit status 137
.
22:38:50 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod wipe local:1
local: stopping and waiting
local: wiping
22:38:50 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor local --oneshot --ignore-empty-nodes
22:38:50 cluster.go:1106: no live node found, skipping consistency check
22:38:50 cluster.go:949: fetching logs
22:38:50 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod get local logs artifacts/acceptance/rapid-restart/logs
local: getting logs artifacts/acceptance/rapid-restart/logs
I190625 22:38:50.417968 1 cluster_synced.go:1252 get logs failed
1: stat /home/roach/local/1/logs: no such file or directory
2: stat /home/roach/local/2/logs: no such file or directory
3: stat /home/roach/local/3/logs: no such file or directory
4: stat /home/roach/local/4/logs: no such file or directory
22:38:53 test.go:1213: failed to download logs: /go/src/github.com/cockroachdb/cockroach/bin/roachprod get local logs artifacts/acceptance/rapid-restart/logs returned:
stderr:
I190625 22:38:50.417968 1 cluster_synced.go:1252 get logs failed
stdout:
local: getting logs artifacts/acceptance/rapid-restart/logs
1: stat /home/roach/local/1/logs: no such file or directory
2: stat /home/roach/local/2/logs: no such file or directory
3: stat /home/roach/local/3/logs: no such file or directory
4: stat /home/roach/local/4/logs: no such file or directory
: exit status 1
--- PASS: acceptance/rapid-restart (71.44s)
##teamcity[testFinished name='acceptance/rapid-restart' flowId='acceptance/rapid-restart']
##teamcity[testStarted name='acceptance/status-server' flowId='acceptance/status-server']
22:38:53 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod wipe local
local: stopping and waiting
local: wiping
22:38:53 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod put local /go/src/github.com/cockroachdb/cockroach/cockroach ./cockroach
local: putting /go/src/github.com/cockroachdb/cockroach/cockroach ./cockroach
1: done
2: done
3: done
4: done
22:38:53 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start local
local: starting.
local: initializing cluster settings
SET CLUSTER SETTING
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26258/_status/gossip/1
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26258/_status/nodes/1
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26258/_status/logfiles/1
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26258/_status/logs/1
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26258/_status/stacks/1
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26258/_status/gossip/local
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26258/_status/nodes/local
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26258/_status/logfiles/local
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26258/_status/logs/local
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26258/_status/stacks/local
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26258/_status/vars
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26258/_status/nodes
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26260/_status/gossip/2
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26260/_status/nodes/2
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26260/_status/logfiles/2
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26260/_status/logs/2
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26260/_status/stacks/2
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26260/_status/gossip/local
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26260/_status/nodes/local
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26260/_status/logfiles/local
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26260/_status/logs/local
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26260/_status/stacks/local
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26260/_status/vars
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26260/_status/nodes
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26262/_status/gossip/3
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26262/_status/nodes/3
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26262/_status/logfiles/3
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26262/_status/logs/3
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26262/_status/stacks/3
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26262/_status/gossip/local
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26262/_status/nodes/local
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26262/_status/logfiles/local
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26262/_status/logs/local
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26262/_status/stacks/local
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26262/_status/vars
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26262/_status/nodes
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26264/_status/gossip/4
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26264/_status/nodes/4
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26264/_status/logfiles/4
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26264/_status/logs/4
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26264/_status/stacks/4
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26264/_status/gossip/local
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26264/_status/nodes/local
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26264/_status/logfiles/local
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26264/_status/logs/local
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26264/_status/stacks/local
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26264/_status/vars
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26264/_status/nodes
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26258/_status/gossip/4
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26258/_status/nodes/4
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26258/_status/logfiles/4
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26258/_status/logs/4
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26258/_status/stacks/4
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26258/_status/vars
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26264/_status/gossip/1
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26264/_status/nodes/1
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26264/_status/logfiles/1
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26264/_status/logs/1
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26264/_status/stacks/1
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26264/_status/vars
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26264/_status/gossip/4
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26264/_status/nodes/4
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26264/_status/logfiles/4
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26264/_status/logs/4
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26264/_status/stacks/4
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26264/_status/gossip/local
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26264/_status/nodes/local
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26264/_status/logfiles/local
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26264/_status/logs/local
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26264/_status/stacks/local
22:38:55 status_server.go:63: OK response from http://127.0.0.1:26264/_status/vars
22:38:55 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor local --oneshot --ignore-empty-nodes
22:38:55 cluster.go:1102: running (fast) consistency checks on node 1
22:38:55 cluster.go:949: fetching logs
22:38:55 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod get local logs artifacts/acceptance/status-server/logs
local: getting logs artifacts/acceptance/status-server/logs
1: done
2: done
3: done
4: done
--- PASS: acceptance/status-server (2.47s)
##teamcity[testFinished name='acceptance/status-server' flowId='acceptance/status-server']
##teamcity[testStarted name='acceptance/version-upgrade' flowId='acceptance/version-upgrade']
22:38:55 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod wipe local
local: stopping and waiting.
local: wiping
I190625 22:38:56.961936 47516 util/binfetcher/binfetcher.go:163 downloading https://binaries.cockroachdb.com/cockroach-v1.0.6.linux-amd64.tgz?ci=true to /tmp/cockroach-v1.0.6.linux-amd64
22:38:57 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod put local:1-3 /tmp/cockroach-v1.0.6.linux-amd64 ./cockroach-v1.0.6
local: putting /tmp/cockroach-v1.0.6.linux-amd64 ./cockroach-v1.0.6
1: done
2: done
3: done
22:38:57 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod run local:1 -- mkdir -p {store-dir} && touch {store-dir}/settings-initialized
22:38:57 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start --binary=./cockroach-v1.0.6 --encrypt=false local:1-3
local: starting.
local: initializing cluster settings
22:39:02 upgrade.go:342: v1.1.9: binary
I190625 22:39:02.712833 47516 util/binfetcher/binfetcher.go:163 downloading https://binaries.cockroachdb.com/cockroach-v1.1.9.linux-amd64.tgz?ci=true to /tmp/cockroach-v1.1.9.linux-amd64
22:39:03 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod put local:1-3 /tmp/cockroach-v1.1.9.linux-amd64 ./cockroach-v1.1.9
local: putting /tmp/cockroach-v1.1.9.linux-amd64 ./cockroach-v1.1.9
1: done
2: done
3: done
22:39:03 upgrade.go:351: v1.1.9: upgrading node 1
22:39:03 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:1
local: stopping and waiting.
22:39:04 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start --binary=./cockroach-v1.1.9 --encrypt=false local:1
local: starting......
local: initializing cluster settings
22:39:11 upgrade.go:351: v1.1.9: upgrading node 3
22:39:11 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:3
local: stopping and waiting.
22:39:12 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start --binary=./cockroach-v1.1.9 --encrypt=false local:3
local: starting
22:39:14 upgrade.go:351: v1.1.9: upgrading node 2
22:39:14 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:2
local: stopping and waiting.
22:39:15 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start --binary=./cockroach-v1.1.9 --encrypt=false local:2
local: starting
22:39:17 upgrade.go:387: 1.1: cluster
22:39:17 upgrade.go:411: 1.1: upgrading cluster version (node 3)
22:39:17 upgrade.go:419: 1.1: using workaround for upgrade
22:39:17 upgrade.go:450: 1.1: cluster is upgraded
22:39:18 upgrade.go:530: 1.1: JSONB fails expected
22:39:18 upgrade.go:530: 1.1: Sequences fails expected
22:39:18 upgrade.go:530: 1.1: Computed Columns fails expected
22:39:18 upgrade.go:342: v2.0.7: binary
I190625 22:39:18.852955 47516 util/binfetcher/binfetcher.go:163 downloading https://binaries.cockroachdb.com/cockroach-v2.0.7.linux-amd64.tgz?ci=true to /tmp/cockroach-v2.0.7.linux-amd64
22:39:19 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod put local:1-3 /tmp/cockroach-v2.0.7.linux-amd64 ./cockroach-v2.0.7
local: putting /tmp/cockroach-v2.0.7.linux-amd64 ./cockroach-v2.0.7
1: done
2: done
3: done
22:39:19 upgrade.go:351: v2.0.7: upgrading node 3
22:39:19 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:3
local: stopping and waiting.
22:39:20 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start --binary=./cockroach-v2.0.7 --encrypt=false local:3
local: starting
22:39:29 upgrade.go:351: v2.0.7: upgrading node 1
22:39:29 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:1
local: stopping and waiting.
22:39:30 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start --binary=./cockroach-v2.0.7 --encrypt=false local:1
local: starting
local: initializing cluster settings
22:39:32 upgrade.go:351: v2.0.7: upgrading node 2
22:39:32 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:2
local: stopping and waiting.
22:39:33 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start --binary=./cockroach-v2.0.7 --encrypt=false local:2
local: starting
22:39:34 upgrade.go:387: 2.0: cluster
22:39:34 upgrade.go:411: 2.0: upgrading cluster version (node 3)
22:39:36 upgrade.go:450: 2.0: cluster is upgraded
[ 10] acceptance/version-upgrade: ??? (0s)
22:39:37 upgrade.go:535: 2.0: JSONB works as expected
22:39:37 upgrade.go:535: 2.0: Sequences works as expected
22:39:38 upgrade.go:535: 2.0: Computed Columns works as expected
22:39:38 upgrade.go:342: v2.1.2: binary
I190625 22:39:38.079704 47516 util/binfetcher/binfetcher.go:163 downloading https://binaries.cockroachdb.com/cockroach-v2.1.2.linux-amd64.tgz?ci=true to /tmp/cockroach-v2.1.2.linux-amd64
22:39:40 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod put local:1-3 /tmp/cockroach-v2.1.2.linux-amd64 ./cockroach-v2.1.2
local: putting /tmp/cockroach-v2.1.2.linux-amd64 ./cockroach-v2.1.2
1: done
2: done
3: done
22:39:40 upgrade.go:351: v2.1.2: upgrading node 3
22:39:40 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:3
local: stopping and waiting.
22:39:41 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start --binary=./cockroach-v2.1.2 --encrypt=false local:3
local: starting
22:39:43 upgrade.go:351: v2.1.2: upgrading node 2
22:39:43 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:2
local: stopping and waiting.
22:39:44 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start --binary=./cockroach-v2.1.2 --encrypt=false local:2
local: starting
22:39:46 upgrade.go:351: v2.1.2: upgrading node 1
22:39:46 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:1
local: stopping and waiting.
22:39:47 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start --binary=./cockroach-v2.1.2 --encrypt=false local:1
local: starting
local: initializing cluster settings
22:39:50 upgrade.go:387: 2.1: cluster
22:39:50 upgrade.go:411: 2.1: upgrading cluster version (node 1)
22:39:51 upgrade.go:450: 2.1: cluster is upgraded
22:39:52 upgrade.go:535: 2.1: JSONB works as expected
22:39:52 upgrade.go:535: 2.1: Sequences works as expected
22:39:52 upgrade.go:535: 2.1: Computed Columns works as expected
22:39:52 upgrade.go:342: v19.1.0-rc.4: binary
I190625 22:39:52.745568 47516 util/binfetcher/binfetcher.go:163 downloading https://binaries.cockroachdb.com/cockroach-v19.1.0-rc.4.linux-amd64.tgz?ci=true to /tmp/cockroach-v19.1.0-rc.4.linux-amd64
22:39:55 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod put local:1-3 /tmp/cockroach-v19.1.0-rc.4.linux-amd64 ./cockroach-v19.1.0-rc.4
local: putting /tmp/cockroach-v19.1.0-rc.4.linux-amd64 ./cockroach-v19.1.0-rc.4
1: done
2: done
3: done
22:39:55 upgrade.go:351: v19.1.0-rc.4: upgrading node 2
22:39:55 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:2
local: stopping and waiting.
22:39:56 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start --binary=./cockroach-v19.1.0-rc.4 --encrypt=false local:2
local: starting
22:39:58 upgrade.go:351: v19.1.0-rc.4: upgrading node 3
22:39:58 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:3
local: stopping and waiting.
22:39:59 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start --binary=./cockroach-v19.1.0-rc.4 --encrypt=false local:3
local: starting
22:40:01 upgrade.go:351: v19.1.0-rc.4: upgrading node 1
22:40:01 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:1
local: stopping and waiting.
22:40:02 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start --binary=./cockroach-v19.1.0-rc.4 --encrypt=false local:1
local: starting
local: initializing cluster settings
22:40:07 upgrade.go:387: 19.1: cluster
22:40:07 upgrade.go:450: 19.1: cluster is upgraded
22:40:09 upgrade.go:535: 19.1: JSONB works as expected
22:40:09 upgrade.go:535: 19.1: Sequences works as expected
22:40:09 upgrade.go:535: 19.1: Computed Columns works as expected
22:40:09 upgrade.go:342: HEAD: binary
22:40:09 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod put local:1-3 /go/src/github.com/cockroachdb/cockroach/cockroach ./cockroach-HEAD
local: putting /go/src/github.com/cockroachdb/cockroach/cockroach ./cockroach-HEAD
1: done
2: done
3: done
22:40:09 upgrade.go:351: HEAD: upgrading node 1
22:40:09 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:1
local: stopping and waiting.
22:40:10 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start --binary=./cockroach-HEAD --encrypt=false local:1
local: starting
local: initializing cluster settings
22:40:15 upgrade.go:351: HEAD: upgrading node 3
22:40:15 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:3
local: stopping and waiting.
22:40:16 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start --binary=./cockroach-HEAD --encrypt=false local:3
local: starting
22:40:20 upgrade.go:351: HEAD: upgrading node 2
22:40:20 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:2
local: stopping and waiting.
22:40:21 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod start --binary=./cockroach-HEAD --encrypt=false local:2
local: starting
22:40:23 upgrade.go:387: 19.1-4: cluster
[ 11] acceptance/version-upgrade: ??? (0s)
22:40:37 test.go:808: test failure: upgrade.go:429,retry.go:184,upgrade.go:423,upgrade.go:568,acceptance.go:90,test.go:1249: 2: dial tcp 127.0.0.1:26259: connect: connection refused
22:40:37 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor local --oneshot --ignore-empty-nodes
22:40:38 test.go:808: test failure: cluster.go:1033,context.go:122,cluster.go:1022,panic.go:406,test.go:783,test.go:773,upgrade.go:429,retry.go:184,upgrade.go:423,upgrade.go:568,acceptance.go:90,test.go:1249: dead node detection: /go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor local --oneshot --ignore-empty-nodes: exit status 1 4: skipped
2: dead
1: 23187
3: 23306
Error: 2: dead
22:40:38 cluster.go:1102: running (fast) consistency checks on node 1
[ 12] acceptance/version-upgrade: ??? (0s)
[ 13] acceptance/version-upgrade: ??? (0s)
[ 14] acceptance/version-upgrade: ??? (0s)
[ 15] acceptance/version-upgrade: ??? (0s)
[ 16] acceptance/version-upgrade: ??? (0s)
[ 17] acceptance/version-upgrade: ??? (0s)
[ 18] acceptance/version-upgrade: ??? (0s)
[ 19] acceptance/version-upgrade: ??? (0s)
22:48:55 test.go:808: test failure: test.go:1235: test timed out (10m0s)
22:48:55 cluster.go:992: fetching debug zip
22:48:55 cluster.go:344: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod ssh local:1 -- ./cockroach debug zip --url {pgurl:1} debug.zip
22:48:56 cluster.go:270: > /go/src/github.com/cockroachdb/cockroach/bin/roachprod get local:1 debug.zip artifacts/acceptance/version-upgrade/debug.zip
local: getting debug.zip artifacts/acceptance/version-upgrade/debug.zip
1: done
[ 20] acceptance/version-upgrade: fetching debug zip (42s)
[ 21] acceptance/version-upgrade: fetching debug zip (1m42s)
[ 22] acceptance/version-upgrade: fetching debug zip (2m42s)
[ 23] acceptance/version-upgrade: fetching debug zip (3m42s)
[ 24] acceptance/version-upgrade: fetching debug zip (4m42s)
[ 25] acceptance/version-upgrade: fetching debug zip (5m42s)
[ 26] acceptance/version-upgrade: fetching debug zip (6m42s)
[ 27] acceptance/version-upgrade: fetching debug zip (7m42s)
[ 28] acceptance/version-upgrade: fetching debug zip (8m42s)
[ 29] acceptance/version-upgrade: fetching debug zip (9m42s)
[ 30] acceptance/version-upgrade: fetching debug zip (10m42s)
[ 31] acceptance/version-upgrade: fetching debug zip (11m42s)
[ 32] acceptance/version-upgrade: fetching debug zip (12m42s)
[ 33] acceptance/version-upgrade: fetching debug zip (13m42s)
[ 34] acceptance/version-upgrade: fetching debug zip (14m42s)
[ 35] acceptance/version-upgrade: fetching debug zip (15m42s)
[ 36] acceptance/version-upgrade: fetching debug zip (16m42s)
[ 37] acceptance/version-upgrade: fetching debug zip (17m42s)
[ 38] acceptance/version-upgrade: fetching debug zip (18m42s)
[ 39] acceptance/version-upgrade: fetching debug zip (19m42s)
[ 40] acceptance/version-upgrade: fetching debug zip (20m42s)
[ 41] acceptance/version-upgrade: fetching debug zip (21m42s)
[ 42] acceptance/version-upgrade: fetching debug zip (22m42s)
[ 43] acceptance/version-upgrade: fetching debug zip (23m42s)
[ 44] acceptance/version-upgrade: fetching debug zip (24m42s)
[ 45] acceptance/version-upgrade: fetching debug zip (25m42s)
[ 46] acceptance/version-upgrade: fetching debug zip (26m42s)
[ 47] acceptance/version-upgrade: fetching debug zip (27m42s)
[ 48] acceptance/version-upgrade: fetching debug zip (28m42s)
[ 49] acceptance/version-upgrade: fetching debug zip (29m42s)
[ 50] acceptance/version-upgrade: fetching debug zip (30m42s)