roachtest: clearrange/checks=true failed #38131
Description
SHA: https://github.com/cockroachdb/cockroach/commits/a581c20147539c74a8c72f8e6b4a7a4206a50853
Parameters:
To repro, try:
# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=clearrange/checks=true PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log
Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1334794&tab=buildLog
The test failed on branch=master, cloud=gce:
cluster.go:1872,clearrange.go:190,clearrange.go:34,test.go:1248: unexpected node event: 9: dead
cluster.go:1035,context.go:87,cluster.go:1024,asm_amd64.s:522,panic.go:397,test.go:785,test.go:771,cluster.go:1872,clearrange.go:190,clearrange.go:34,test.go:1248: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1334794-clearrange-checks-true --oneshot --ignore-empty-nodes: exit status 1 9: dead
8: 28075
3: 2198
2: 24065
7: 26724
1: 18844
4: 15068
5: 1798
6: 26271
10: 18627
Error: 9: dead