roachtest: kv/splits/nodes=3/quiesce=false failed #31125
Description
SHA: https://github.com/cockroachdb/cockroach/commits/dc0e73c728e533fdb3bec63e53eec174e920ff22
Parameters:
To repro, try:
# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stressrace instead of stress and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
make stress TESTS=kv/splits/nodes=3/quiesce=false PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-stderr=false -maxtime 20m -timeout 10m'
Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=952220&tab=buildLog
The test failed on release-2.1:
test.go:570,cluster.go:975,kv.go:192,cluster.go:1297,errgroup.go:58: /home/agent/work/.go/bin/roachprod run teamcity-952220-kv-splits-nodes-3-quiesce-false:4 -- ./workload run kv --init --max-ops=1 --concurrency=192 --splits=500000 {pgurl:1-3} returned:
stderr:
stdout:
) TO (-5752107573081187328): dial tcp 10.128.0.29:26257: connect: connection refused
W181009 08:09:06.229009 171 workload/workload.go:434 ALTER TABLE kv SCATTER FROM (-5671569249532018688) TO (-5671569249532018688): dial tcp 10.128.0.29:26257: connect: connection refused
W181009 08:09:06.230111 30 workload/workload.go:434 ALTER TABLE kv SCATTER FROM (-5625710735481804800) TO (-5625710735481804800): dial tcp 10.128.0.29:26257: connect: connection refused
W181009 08:09:06.230345 322 workload/workload.go:434 ALTER TABLE kv SCATTER FROM (-5759412469124583424) TO (-5759412469124583424): dial tcp 10.128.0.29:26257: connect: connection refused
W181009 08:09:06.230405 250 workload/workload.go:434 ALTER TABLE kv SCATTER FROM (-5689204301596381184) TO (-5689204301596381184): dial tcp 10.128.0.29:26257: connect: connection refused
W181009 08:09:06.230571 120 workload/workload.go:434 ALTER TABLE kv SCATTER FROM (-5763802785433494528) TO (-5763802785433494528): dial tcp 10.128.0.29:26257: connect: connection refused
: signal: killed
test.go:570,cluster.go:1318,kv.go:195: unexpected node event: 1: dead