Skip to content

sql: TestDropWhileBackfill failed under stress #35782

@cockroach-teamcity

Description

@cockroach-teamcity

SHA: https://github.com/cockroachdb/cockroach/commits/70e3468e7ed5fb30b10eaaf972acbb0f16099d01

Parameters:

TAGS=
GOFLAGS=-race

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=TestDropWhileBackfill PKG=github.com/cockroachdb/cockroach/pkg/sql TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1179814&tab=buildLog

I190315 08:03:14.291456 96604 util/stop/stopper.go:546  quiescing; tasks left:
3      [async] closedts-subscription
I190315 08:03:14.292327 96693 rpc/nodedialer/nodedialer.go:143  [ct-client] unable to connect to n4: failed to connect to n4 at 127.0.0.1:39967: context canceled
I190315 08:03:14.293814 96694 rpc/nodedialer/nodedialer.go:143  [ct-client] unable to connect to n5: failed to connect to n5 at 127.0.0.1:39319: context canceled
I190315 08:03:14.295997 96604 util/stop/stopper.go:546  quiescing; tasks left:
2      [async] closedts-subscription
I190315 08:03:14.297050 96606 util/stop/stopper.go:546  quiescing; tasks left:
1      [async] closedts-subscription
I190315 08:03:14.297692 96605 util/stop/stopper.go:546  quiescing; tasks left:
3      [async] closedts-subscription
1      [async] intent_resolver_ir_batcher
1      [async] intent_resolver_gc_batcher
1      [async] closedts-rangefeed-subscriber
I190315 08:03:14.298499 96605 util/stop/stopper.go:546  quiescing; tasks left:
3      [async] closedts-subscription
1      [async] intent_resolver_gc_batcher
1      [async] closedts-rangefeed-subscriber
I190315 08:03:14.298877 96605 util/stop/stopper.go:546  quiescing; tasks left:
3      [async] closedts-subscription
1      [async] closedts-rangefeed-subscriber
I190315 08:03:14.302959 96604 util/stop/stopper.go:546  quiescing; tasks left:
1      [async] closedts-subscription
I190315 08:03:14.307454 96607 util/stop/stopper.go:546  quiescing; tasks left:
4      [async] closedts-subscription
1      [async] closedts-rangefeed-subscriber
W190315 08:03:14.312472 96010 storage/raft_transport.go:583  [n1] while processing outgoing Raft queue to node 2: rpc error: code = Canceled desc = context canceled:
I190315 08:03:14.326053 96699 rpc/nodedialer/nodedialer.go:143  [ct-client] unable to connect to n2: failed to connect to n2 at 127.0.0.1:35317: context canceled
W190315 08:03:14.326433 95853 storage/raft_transport.go:583  [n1] while processing outgoing Raft queue to node 3: rpc error: code = Canceled desc = context canceled:
I190315 08:03:14.327046 96608 util/stop/stopper.go:546  quiescing; tasks left:
3      [async] closedts-subscription
1      [async] closedts-rangefeed-subscriber
I190315 08:03:14.327721 96698 rpc/nodedialer/nodedialer.go:143  [ct-client] unable to connect to n3: failed to connect to n3 at 127.0.0.1:36945: context canceled
I190315 08:03:14.327916 96608 util/stop/stopper.go:546  quiescing; tasks left:
1      [async] closedts-subscription
I190315 08:03:14.328901 96700 rpc/nodedialer/nodedialer.go:143  [ct-client] unable to connect to n1: failed to connect to n1 at 127.0.0.1:43721: context canceled
W190315 08:03:14.329799 95878 storage/raft_transport.go:583  [n1] while processing outgoing Raft queue to node 4: rpc error: code = Canceled desc = grpc: the client connection is closing:
I190315 08:03:14.330167 96701 rpc/nodedialer/nodedialer.go:143  [ct-client] unable to connect to n4: failed to connect to n4 at 127.0.0.1:39967: context canceled
I190315 08:03:14.331071 96607 util/stop/stopper.go:546  quiescing; tasks left:
2      [async] closedts-subscription
1      [async] closedts-rangefeed-subscriber
I190315 08:03:14.331553 96607 util/stop/stopper.go:546  quiescing; tasks left:
1      [async] closedts-subscription
I190315 08:03:14.331781 96605 util/stop/stopper.go:546  quiescing; tasks left:
1      [async] closedts-subscription
I190315 08:03:14.388260 96706 rpc/nodedialer/nodedialer.go:143  [ct-client] unable to connect to n5: failed to connect to n5 at 127.0.0.1:39319: context canceled
I190315 08:03:14.388477 96708 rpc/nodedialer/nodedialer.go:143  [ct-client] unable to connect to n2: failed to connect to n2 at 127.0.0.1:35317: context canceled
I190315 08:03:14.388860 96707 rpc/nodedialer/nodedialer.go:143  [ct-client] unable to connect to n1: failed to connect to n1 at 127.0.0.1:43721: context canceled
I190315 08:03:14.389565 96709 rpc/nodedialer/nodedialer.go:143  [ct-client] unable to connect to n3: failed to connect to n3 at 127.0.0.1:36945: context canceled
--- FAIL: TestDropWhileBackfill (15.74s)
    schema_changer_test.go:833: pq: internal error: uncaught error: context deadline exceeded

Metadata

Metadata

Assignees

Labels

C-test-failureBroken test (automatically or manually discovered).O-robotOriginated from a bot.

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions