-
Notifications
You must be signed in to change notification settings - Fork 4.1k
ccl/backupccl: TestBackupRestoreWithConcurrentWrites failed #43251
Copy link
Copy link
Closed
Labels
C-test-failureBroken test (automatically or manually discovered).Broken test (automatically or manually discovered).O-robotOriginated from a bot.Originated from a bot.branch-masterFailures and bugs on the master branch.Failures and bugs on the master branch.
Milestone
Description
(ccl/backupccl).TestBackupRestoreWithConcurrentWrites failed on master@20e2f036ac18eed4ee2d58958bb8e725add9f4fa:
E191217 17:31:04.478309 171977 sql/lease.go:1194 background lease renewal for tableID=53 tableName="bank" failed: table is being dropped
E191217 17:31:04.601871 171982 sql/lease.go:1194 background lease renewal for tableID=53 tableName="bank" failed: table is being dropped
E191217 17:31:04.754402 172040 sql/lease.go:1194 background lease renewal for tableID=53 tableName="bank" failed: table is being dropped
E191217 17:31:04.862610 172057 sql/lease.go:1194 background lease renewal for tableID=53 tableName="bank" failed: table is being dropped
E191217 17:31:04.967536 172076 sql/lease.go:1194 background lease renewal for tableID=53 tableName="bank" failed: table is being dropped
E191217 17:31:05.069155 172060 sql/lease.go:1194 background lease renewal for tableID=53 tableName="bank" failed: table is being dropped
I191217 17:31:05.178939 117698 server/status/runtime.go:498 [n1] runtime stats: 2.2 GiB RSS, 761 goroutines, 111 MiB/62 MiB/204 MiB GO alloc/idle/total, 281 MiB/326 MiB CGO alloc/total, 787.5 CGO/sec, 173.4/13.6 %(u/s)time, 0.9 %gc (6x), 4.8 MiB/4.8 MiB (r/w)net
E191217 17:31:05.300711 172099 sql/lease.go:1194 background lease renewal for tableID=53 tableName="bank" failed: table is being dropped
E191217 17:31:05.573781 172151 sql/lease.go:1194 background lease renewal for tableID=53 tableName="bank" failed: table is being dropped
E191217 17:31:05.835404 172218 sql/lease.go:1194 background lease renewal for tableID=53 tableName="bank" failed: table is being dropped
E191217 17:31:05.961724 172147 sql/lease.go:1194 background lease renewal for tableID=53 tableName="bank" failed: table is being dropped
E191217 17:31:06.100651 172228 sql/lease.go:1194 background lease renewal for tableID=53 tableName="bank" failed: table is being dropped
E191217 17:31:06.212820 172172 sql/lease.go:1194 background lease renewal for tableID=53 tableName="bank" failed: table is being dropped
E191217 17:31:06.431140 172297 sql/lease.go:1194 background lease renewal for tableID=53 tableName="bank" failed: table is being dropped
E191217 17:31:06.502178 172347 sql/lease.go:1194 background lease renewal for tableID=53 tableName="bank" failed: table is being dropped
I191217 17:31:06.892594 118646 server/status/runtime.go:498 [n2] runtime stats: 2.2 GiB RSS, 765 goroutines, 112 MiB/61 MiB/204 MiB GO alloc/idle/total, 281 MiB/326 MiB CGO alloc/total, 787.1 CGO/sec, 173.1/12.5 %(u/s)time, 0.9 %gc (6x), 4.8 MiB/4.8 MiB (r/w)net
I191217 17:31:07.205626 122001 sql/sqlbase/structured.go:1425 [n1,client=127.0.0.1:42394,user=root,scExec] publish: descID=53 (bank) version=3 mtime=1970-01-01 00:00:00 +0000 UTC
W191217 17:31:07.519020 122001 sql/exec_util.go:1112 [n1,client=127.0.0.1:42394,user=root] error executing schema change: job with ID 512927886517698561 does not exist
I191217 17:31:07.544838 117442 util/stop/stopper.go:539 quiescing
I191217 17:31:07.699922 172566 util/stop/stopper.go:539 quiescing
I191217 17:31:07.700097 172568 util/stop/stopper.go:539 quiescing
I191217 17:31:07.702382 172567 util/stop/stopper.go:539 quiescing
W191217 17:31:07.717040 122270 storage/raft_transport.go:637 [n2] while processing outgoing Raft queue to node 3: EOF:
W191217 17:31:07.720062 119034 storage/raft_transport.go:637 [n3] while processing outgoing Raft queue to node 1: rpc error: code = Canceled desc = grpc: the client connection is closing:
W191217 17:31:07.720508 122076 storage/raft_transport.go:637 [n3] while processing outgoing Raft queue to node 2: rpc error: code = Canceled desc = grpc: the client connection is closing:
W191217 17:31:07.721179 122313 storage/raft_transport.go:637 [n3] while processing outgoing Raft queue to node 2: rpc error: code = Canceled desc = grpc: the client connection is closing:
W191217 17:31:07.722889 118671 gossip/gossip.go:1517 [n3] no incoming or outgoing connections
W191217 17:31:07.724432 119120 storage/raft_transport.go:637 [n2] while processing outgoing Raft queue to node 1: EOF:
W191217 17:31:07.734943 119071 storage/raft_transport.go:637 [n1] while processing outgoing Raft queue to node 2: rpc error: code = Canceled desc = grpc: the client connection is closing:
W191217 17:31:07.737886 119079 storage/raft_transport.go:637 [n3] while processing outgoing Raft queue to node 1: EOF:
W191217 17:31:07.740848 118953 storage/raft_transport.go:637 [n1] while processing outgoing Raft queue to node 3: rpc error: code = Canceled desc = grpc: the client connection is closing:
W191217 17:31:07.734478 119043 storage/raft_transport.go:637 [n1] while processing outgoing Raft queue to node 3: rpc error: code = Canceled desc = grpc: the client connection is closing:
W191217 17:31:07.745381 119186 storage/raft_transport.go:637 [n1] while processing outgoing Raft queue to node 2: EOF:
W191217 17:31:07.751420 119148 storage/raft_transport.go:637 [n2] while processing outgoing Raft queue to node 1: EOF:
I191217 17:31:07.757611 172556 rpc/nodedialer/nodedialer.go:160 [ct-client] unable to connect to n1: failed to connect to n1 at 127.0.0.1:44127: context canceled
I191217 17:31:07.758938 172557 rpc/nodedialer/nodedialer.go:160 [ct-client] unable to connect to n3: failed to connect to n3 at 127.0.0.1:40541: context canceled
W191217 17:31:07.766687 118497 gossip/gossip.go:1517 [n2] no incoming or outgoing connections
W191217 17:31:07.769363 172372 storage/intentresolver/intent_resolver.go:818 failed to gc transaction record: could not GC completed transaction anchored at /Table/SystemConfigSpan/Start: node unavailable; try another peer
W191217 17:31:07.774556 122347 storage/raft_transport.go:637 [n2] while processing outgoing Raft queue to node 3: rpc error: code = Canceled desc = grpc: the client connection is closing:
I191217 17:31:07.796195 172505 rpc/nodedialer/nodedialer.go:160 [ct-client] unable to connect to n2: failed to connect to n2 at 127.0.0.1:35659: context canceled
I191217 17:31:07.805069 172506 rpc/nodedialer/nodedialer.go:160 [ct-client] unable to connect to n3: failed to connect to n3 at 127.0.0.1:40541: context canceled
I191217 17:31:07.825011 118934 server/status/runtime.go:498 [n3] runtime stats: 2.1 GiB RSS, 408 goroutines, 85 MiB/84 MiB/204 MiB GO alloc/idle/total(stale), 259 MiB/311 MiB CGO alloc/total, 763.3 CGO/sec, 174.1/12.9 %(u/s)time, 0.7 %gc (6x), 4.8 MiB/4.8 MiB (r/w)net
W191217 17:31:07.829547 118647 ts/db.go:194 [n2,ts-poll] error writing time series data: node unavailable; try another peer
I191217 17:31:07.847632 172569 rpc/nodedialer/nodedialer.go:160 [ct-client] unable to connect to n2: failed to connect to n2 at 127.0.0.1:35659: context canceled
I191217 17:31:07.850185 172570 rpc/nodedialer/nodedialer.go:160 [ct-client] unable to connect to n1: failed to connect to n1 at 127.0.0.1:44127: context canceled
E191217 17:31:07.944140 117613 jobs/registry.go:358 error while adopting jobs: adopt-job: node unavailable; try another peer
I191217 17:31:07.957373 117709 kv/transport_race.go:108 transport race promotion: ran 293 iterations on up to 48119 requests
E191217 17:31:08.024460 172462 vendor/google.golang.org/grpc/pickfirst.go:61 pickfirstBalancer: failed to NewSubConn: rpc error: code = Canceled desc = grpc: the client connection is closing
--- FAIL: TestBackupRestoreWithConcurrentWrites (333.34s)
backup_test.go:2007: error executing 'DROP TABLE data.bank': pq: job with ID 512927886517698561 does not exist
Repro
Parameters:
- GOFLAGS=-json
make stressrace TESTS=TestBackupRestoreWithConcurrentWrites PKG=./pkg/ccl/backupccl TESTTIMEOUT=5m STRESSFLAGS=-timeout 5m' 2>&1
powered by pkg/cmd/internal/issues
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
C-test-failureBroken test (automatically or manually discovered).Broken test (automatically or manually discovered).O-robotOriginated from a bot.Originated from a bot.branch-masterFailures and bugs on the master branch.Failures and bugs on the master branch.