Skip to content

roachtest: disk-full failed [skipped] #35328

@cockroach-teamcity

Description

@cockroach-teamcity

SHA: https://github.com/cockroachdb/cockroach/commits/032c4980720abc1bdd71e4428e4111e6e6383297

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=disk-full PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1158877&tab=buildLog

The test failed on master:
	cluster.go:1152,disk_full.go:84,cluster.go:1564,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start teamcity-1158877-disk-full:1 --encrypt returned:
		stderr:
		
		stdout:
		t how to secure your cluster: https://www.cockroachlabs.com/docs/v19.1/secure-a-cluster.html
		*
		*
		* ERROR: could not cleanup temporary directories from record file: could not lock temporary directory /mnt/data1/cockroach/cockroach-temp077746866, may still be in use: IO error: While lock file: /mnt/data1/cockroach/cockroach-temp077746866/TEMP_DIR.LOCK: Resource temporarily unavailable
		*
		Failed running "start"
		E190302 08:18:45.376333 1 cli/error.go:229  exit status 1
		Error: exit status 1
		Failed running "start"
		
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.Cockroach.Start.func7
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:396
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*SyncedCluster).Parallel.func1.1
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:1301
		runtime.goexit
			/usr/local/go/src/runtime/asm_amd64.s:1333: 
		I190302 08:18:45.381201 1 cluster_synced.go:1383  command failed
		: exit status 1
	cluster.go:1226,disk_full.go:50,cluster.go:1564,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1158877-disk-full:5 -- ./workload run kv --tolerate-errors --init --read-percent=0 --concurrency=10 --duration=2m {pgurl:1-4} returned:
		stderr:
		
		stdout:
		   3.3     12.1     21.0     41.9 write
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		    1m1s        0         2182.0         1961.8      3.1     12.1     18.9     30.4 write
		    1m2s        0         2181.0         1965.4      3.0     12.1     22.0     33.6 write
		    1m3s        0         2145.1         1968.2      3.4     11.5     16.8     33.6 write
		    1m4s        0         1222.9         1956.6      3.0     12.1     21.0     31.5 write
		    1m5s        0            0.0         1926.5      0.0      0.0      0.0      0.0 write
		    1m6s        0            0.0         1897.3      0.0      0.0      0.0      0.0 write
		    1m7s        0          595.0         1877.9      3.5     15.2   3221.2   3221.2 write
		    1m8s        0         1971.1         1879.2      3.5     13.1     18.9     32.5 write
		    1m9s        0         2065.9         1881.9      3.4     12.1     19.9     28.3 write
		   1m10s        0         2119.1         1885.3      3.4     11.5     18.9     33.6 write
		: signal: killed
	cluster.go:1585,disk_full.go:87,test.go:1211: Goexit() was called

Metadata

Metadata

Labels

A-disaster-recoveryC-test-failureBroken test (automatically or manually discovered).E-starterMight be suitable for a starter project for new employees or team members.O-roachtestO-robotOriginated from a bot.branch-masterFailures and bugs on the master branch.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions