ccl/backupccl: TestBackupRestoreWithConcurrentWrites failed under stress #40361
Description
SHA: https://github.com/cockroachdb/cockroach/commits/66bd279c9aa682c2b7adcec87ec0c639b8039a33
Parameters:
TAGS=
GOFLAGS=-parallel=4
To repro, try:
# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=TestBackupRestoreWithConcurrentWrites PKG=github.com/cockroachdb/cockroach/pkg/ccl/backupccl TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log
Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1461687&tab=buildLog
I190830 05:34:37.282071 77288 util/stop/stopper.go:542 quiescing; tasks left:
7 [async] storage.IntentResolver: cleanup txn records
2 [async] closedts-subscription
1 [async] closedts-rangefeed-subscriber
I190830 05:34:37.282133 77288 util/stop/stopper.go:542 quiescing; tasks left:
6 [async] storage.IntentResolver: cleanup txn records
2 [async] closedts-subscription
1 [async] closedts-rangefeed-subscriber
I190830 05:34:37.282141 77289 util/stop/stopper.go:542 quiescing; tasks left:
6 [async] storage.IntentResolver: cleanup txn records
2 [async] closedts-subscription
1 [async] intent_resolver_ir_batcher
1 [async] closedts-rangefeed-subscriber
I190830 05:34:37.282248 77288 util/stop/stopper.go:542 quiescing; tasks left:
5 [async] storage.IntentResolver: cleanup txn records
2 [async] closedts-subscription
1 [async] closedts-rangefeed-subscriber
I190830 05:34:37.282314 77288 util/stop/stopper.go:542 quiescing; tasks left:
4 [async] storage.IntentResolver: cleanup txn records
2 [async] closedts-subscription
1 [async] closedts-rangefeed-subscriber
W190830 05:34:37.282332 70617 gossip/gossip.go:1517 [n2] no incoming or outgoing connections
I190830 05:34:37.282377 77288 util/stop/stopper.go:542 quiescing; tasks left:
3 [async] storage.IntentResolver: cleanup txn records
2 [async] closedts-subscription
1 [async] closedts-rangefeed-subscriber
I190830 05:34:37.282483 77288 util/stop/stopper.go:542 quiescing; tasks left:
2 [async] storage.IntentResolver: cleanup txn records
2 [async] closedts-subscription
1 [async] closedts-rangefeed-subscriber
I190830 05:34:37.282556 77288 util/stop/stopper.go:542 quiescing; tasks left:
2 [async] closedts-subscription
1 [async] storage.IntentResolver: cleanup txn records
1 [async] closedts-rangefeed-subscriber
I190830 05:34:37.282611 77288 util/stop/stopper.go:542 quiescing; tasks left:
2 [async] closedts-subscription
1 [async] closedts-rangefeed-subscriber
I190830 05:34:37.282666 77289 util/stop/stopper.go:542 quiescing; tasks left:
4 [async] storage.IntentResolver: cleanup txn records
2 [async] closedts-subscription
1 [async] closedts-rangefeed-subscriber
I190830 05:34:37.282683 77290 util/stop/stopper.go:542 quiescing; tasks left:
2 [async] closedts-subscription
1 [async] closedts-rangefeed-subscriber
I190830 05:34:37.282745 77289 util/stop/stopper.go:542 quiescing; tasks left:
3 [async] storage.IntentResolver: cleanup txn records
2 [async] closedts-subscription
1 [async] closedts-rangefeed-subscriber
I190830 05:34:37.282826 77289 util/stop/stopper.go:542 quiescing; tasks left:
3 [async] storage.IntentResolver: cleanup txn records
2 [async] closedts-subscription
I190830 05:34:37.283011 77289 util/stop/stopper.go:542 quiescing; tasks left:
2 [async] storage.IntentResolver: cleanup txn records
2 [async] closedts-subscription
I190830 05:34:37.283037 77289 util/stop/stopper.go:542 quiescing; tasks left:
2 [async] closedts-subscription
1 [async] storage.IntentResolver: cleanup txn records
I190830 05:34:37.283141 77288 util/stop/stopper.go:542 quiescing; tasks left:
1 [async] closedts-subscription
1 [async] closedts-rangefeed-subscriber
I190830 05:34:37.283168 77288 util/stop/stopper.go:542 quiescing; tasks left:
1 [async] closedts-subscription
I190830 05:34:37.283214 77289 util/stop/stopper.go:542 quiescing; tasks left:
1 [async] storage.IntentResolver: cleanup txn records
1 [async] closedts-subscription
I190830 05:34:37.283235 77290 util/stop/stopper.go:542 quiescing; tasks left:
1 [async] closedts-subscription
1 [async] closedts-rangefeed-subscriber
I190830 05:34:37.283353 77289 util/stop/stopper.go:542 quiescing; tasks left:
1 [async] storage.IntentResolver: cleanup txn records
--- FAIL: TestBackupRestoreWithConcurrentWrites (3.42s)
backup_test.go:1888: error executing 'SELECT id, balance, payload FROM data.bank WHERE id != balance': pq: table "bank" is offline: restoring