-
Notifications
You must be signed in to change notification settings - Fork 4.1k
roachtest: scrub/all-checks/tpcc/w=100 failed #37108
Copy link
Copy link
Closed
Labels
C-test-failureBroken test (automatically or manually discovered).Broken test (automatically or manually discovered).O-roachtestO-robotOriginated from a bot.Originated from a bot.
Milestone
Description
SHA: https://github.com/cockroachdb/cockroach/commits/7109d291e3b9edfa38264361f832cec14fff66ee
Parameters:
To repro, try:
# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=scrub/all-checks/tpcc/w=100 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log
Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1259219&tab=buildLog
The test failed on release-19.1:
scrub.go:83,cluster.go:1761,errgroup.go:57: dial tcp 35.188.39.106:26257: connect: connection refused
cluster.go:1423,tpcc.go:168,cluster.go:1761,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1259219-scrub-all-checks-tpcc-w-100:5 -- ./workload run tpcc --warehouses=100 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=30m0s {pgurl:1-4} returned:
stderr:
stdout:
l
_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
10m49s 26544 0.0 0.1 0.0 0.0 0.0 0.0 delivery
10m49s 26544 0.0 0.5 0.0 0.0 0.0 0.0 newOrder
10m49s 26544 0.0 0.1 0.0 0.0 0.0 0.0 orderStatus
10m49s 26544 0.0 0.3 0.0 0.0 0.0 0.0 payment
10m49s 26544 0.0 0.0 0.0 0.0 0.0 0.0 stockLevel
10m50s 26544 0.0 0.1 0.0 0.0 0.0 0.0 delivery
10m50s 26544 0.0 0.5 0.0 0.0 0.0 0.0 newOrder
10m50s 26544 0.0 0.1 0.0 0.0 0.0 0.0 orderStatus
10m50s 26544 0.0 0.3 0.0 0.0 0.0 0.0 payment
10m50s 26544 0.0 0.0 0.0 0.0 0.0 0.0 stockLevel
: signal: killed
cluster.go:1782,tpcc.go:178,scrub.go:58,test.go:1245: Goexit() was called
cluster.go:1016,context.go:89,cluster.go:1005,asm_amd64.s:522,panic.go:397,test.go:790,test.go:776,cluster.go:1782,tpcc.go:178,scrub.go:58,test.go:1245: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1259219-scrub-all-checks-tpcc-w-100 --oneshot --ignore-empty-nodes: exit status 1 5: skipped
1: dead
3: 3213
2: 3189
4: 3156
Error: 1: dead
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
C-test-failureBroken test (automatically or manually discovered).Broken test (automatically or manually discovered).O-roachtestO-robotOriginated from a bot.Originated from a bot.