roachtest: scrub/all-checks/tpcc/w=1000 failed #35986
Description
SHA: https://github.com/cockroachdb/cockroach/commits/3a7ea2d8c9d4a3e0d97f8f106fcf95b3f03765ec
Parameters:
To repro, try:
# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=scrub/all-checks/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log
Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1187480&tab=buildLog
The test failed on master:
cluster.go:1267,tpcc.go:130,cluster.go:1605,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1187480-scrub-all-checks-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=2h0m0s {pgurl:1-4} returned:
stderr:
stdout:
9.2 103079.2 newOrder
59m6s 26598 0.0 2.1 0.0 0.0 0.0 0.0 orderStatus
59m6s 26598 0.0 20.0 0.0 0.0 0.0 0.0 payment
59m6s 26598 0.0 2.1 0.0 0.0 0.0 0.0 stockLevel
E190320 15:17:10.469844 1 workload/cli/run.go:420 error in payment: dial tcp 10.142.0.62:26257: connect: connection refused
59m7s 71358 0.0 2.2 0.0 0.0 0.0 0.0 delivery
59m7s 71358 1.0 21.1 103079.2 103079.2 103079.2 103079.2 newOrder
59m7s 71358 0.0 2.1 0.0 0.0 0.0 0.0 orderStatus
59m7s 71358 0.0 20.0 0.0 0.0 0.0 0.0 payment
59m7s 71358 0.0 2.1 0.0 0.0 0.0 0.0 stockLevel
E190320 15:17:11.469848 1 workload/cli/run.go:420 error in payment: dial tcp 10.142.0.62:26257: connect: connection refused
: signal: killed
cluster.go:1626,tpcc.go:140,scrub.go:58,test.go:1214: unexpected node event: 3: dead