-
Notifications
You must be signed in to change notification settings - Fork 4.1k
roachtest: disk-full failed [skipped] #35328
Copy link
Copy link
Closed
Labels
A-disaster-recoveryC-test-failureBroken test (automatically or manually discovered).Broken test (automatically or manually discovered).E-starterMight be suitable for a starter project for new employees or team members.Might be suitable for a starter project for new employees or team members.O-roachtestO-robotOriginated from a bot.Originated from a bot.branch-masterFailures and bugs on the master branch.Failures and bugs on the master branch.
Description
SHA: https://github.com/cockroachdb/cockroach/commits/032c4980720abc1bdd71e4428e4111e6e6383297
Parameters:
To repro, try:
# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=disk-full PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log
Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1158877&tab=buildLog
The test failed on master:
cluster.go:1152,disk_full.go:84,cluster.go:1564,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start teamcity-1158877-disk-full:1 --encrypt returned:
stderr:
stdout:
t how to secure your cluster: https://www.cockroachlabs.com/docs/v19.1/secure-a-cluster.html
*
*
* ERROR: could not cleanup temporary directories from record file: could not lock temporary directory /mnt/data1/cockroach/cockroach-temp077746866, may still be in use: IO error: While lock file: /mnt/data1/cockroach/cockroach-temp077746866/TEMP_DIR.LOCK: Resource temporarily unavailable
*
Failed running "start"
E190302 08:18:45.376333 1 cli/error.go:229 exit status 1
Error: exit status 1
Failed running "start"
github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.Cockroach.Start.func7
/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:396
github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*SyncedCluster).Parallel.func1.1
/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:1301
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1333:
I190302 08:18:45.381201 1 cluster_synced.go:1383 command failed
: exit status 1
cluster.go:1226,disk_full.go:50,cluster.go:1564,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1158877-disk-full:5 -- ./workload run kv --tolerate-errors --init --read-percent=0 --concurrency=10 --duration=2m {pgurl:1-4} returned:
stderr:
stdout:
3.3 12.1 21.0 41.9 write
_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
1m1s 0 2182.0 1961.8 3.1 12.1 18.9 30.4 write
1m2s 0 2181.0 1965.4 3.0 12.1 22.0 33.6 write
1m3s 0 2145.1 1968.2 3.4 11.5 16.8 33.6 write
1m4s 0 1222.9 1956.6 3.0 12.1 21.0 31.5 write
1m5s 0 0.0 1926.5 0.0 0.0 0.0 0.0 write
1m6s 0 0.0 1897.3 0.0 0.0 0.0 0.0 write
1m7s 0 595.0 1877.9 3.5 15.2 3221.2 3221.2 write
1m8s 0 1971.1 1879.2 3.5 13.1 18.9 32.5 write
1m9s 0 2065.9 1881.9 3.4 12.1 19.9 28.3 write
1m10s 0 2119.1 1885.3 3.4 11.5 18.9 33.6 write
: signal: killed
cluster.go:1585,disk_full.go:87,test.go:1211: Goexit() was called
Reactions are currently unavailable
Metadata
Metadata
Labels
A-disaster-recoveryC-test-failureBroken test (automatically or manually discovered).Broken test (automatically or manually discovered).E-starterMight be suitable for a starter project for new employees or team members.Might be suitable for a starter project for new employees or team members.O-roachtestO-robotOriginated from a bot.Originated from a bot.branch-masterFailures and bugs on the master branch.Failures and bugs on the master branch.