# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=scaledata/distributed_semaphore/nodes=6 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191019-1546763/scaledata/distributed_semaphore/nodes=6/run_1
	cluster.go:2159,scaledata.go:121,scaledata.go:48,test_runner.go:689: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1546763-1571461908-63-n7cpu4:7 -- ./distributed_semaphore  --duration_secs=600 --num_workers=16 --cockroach_ip_addresses_csv='10.128.0.169:26257,10.128.0.157:26257,10.128.0.170:26257,10.128.0.156:26257,10.128.0.155:26257,10.128.0.171:26257'  returned:
		stderr:
		54396833 +0000 UTC m=+88.455372790, took 4.053836998s
		2019/10/19 09:26:45 [/Users/nathan/Go/src/github.com/scaledata/rksql/src/go/src/rubrik/sqlapp/distributed_semaphore/cockroach.go:54 pq: communication error: rpc error: code = Canceled desc = context canceled]
		2019/10/19 09:26:45 pq error - Error code : 58C01, Error class : 58
		2019/10/19 09:26:45 ExecuteTx retry attempt 1 failed, started at 2019-10-19 09:26:43.683156249 +0000 UTC m=+86.784132196, now = 2019-10-19 09:26:45.354440368 +0000 UTC m=+88.455416341, took 1.671284145s
		2019/10/19 09:26:45 pq error - Error code : 58C01, Error class : 58
		2019/10/19 09:26:45 pq error - Error code : 58C01, Error class : 58
		2019/10/19 09:26:45 Aborting Retries because this error of type *pq.Error is not retryable : pq: communication error: rpc error: code = Canceled desc = context canceled
		2019/10/19 09:26:45 postgres error code is 58C01 and class is 58
		2019/10/19 09:26:45 pq: communication error: rpc error: code = Canceled desc = context canceled
		Error:  exit status 255
		
		stdout:
		rt transaction: TransactionRetryWithProtoRefreshError: ReadWithinUncertaintyIntervalError: read at time 1571477116.912140917,0 encountered previous write with future timestamp 1571477116.924516584,0 within uncertainty interval `t <= 1571477116.926851286,0`; observed timestamps: [{2 1571477116.912140917,0} {6 1571477116.926851286,0}]
		pq: restart transaction: TransactionRetryWithProtoRefreshError: ReadWithinUncertaintyIntervalError: read at time 1571477116.910506567,0 encountered previous write with future timestamp 1571477116.914352153,0 within uncertainty interval `t <= 1571477116.922464279,0`; observed timestamps: [{1 1571477116.910506567,0} {6 1571477116.922464279,0}]
		pq: restart transaction: TransactionRetryWithProtoRefreshError: ReadWithinUncertaintyIntervalError: read at time 1571477116.914617385,0 encountered previous write with future timestamp 1571477116.925127015,11 within uncertainty interval `t <= 1571477116.927919347,0`; observed timestamps: [{4 1571477116.914617385,0} {6 1571477116.927919347,0}]
		: exit status 1