-
Notifications
You must be signed in to change notification settings - Fork 4.1k
ccl/partitionccl: TestInitialPartitioning failed under stress #39651
Copy link
Copy link
Closed
Labels
C-test-failureBroken test (automatically or manually discovered).Broken test (automatically or manually discovered).O-robotOriginated from a bot.Originated from a bot.
Milestone
Description
SHA: https://github.com/cockroachdb/cockroach/commits/bad5e2ea8674f0513f949d3940609b1f4d685581
Parameters:
TAGS=
GOFLAGS=-parallel=4
To repro, try:
# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=TestInitialPartitioning PKG=github.com/cockroachdb/cockroach/pkg/ccl/partitionccl TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log
Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1438014&tab=buildLog
I190814 05:43:30.845972 176419 storage/replica_command.go:1198 [n3,replicate,s3,r242/1:/Table/83/1{-/00:00:…}] change replicas (ADD_REPLICA (n1,s1):3LEARNER): existing descriptor r242:/Table/83/1{-/00:00:00} [(n3,s3):1, next=3, gen=213]
I190814 05:43:30.848659 173954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM interval_table WHERE a < '00:00:00':::INTERVAL
I190814 05:43:30.851700 173954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM interval_table WHERE a < '00:00:00':::INTERVAL
I190814 05:43:30.854114 176379 storage/store.go:2571 [n1,replicaGC,s1,r242/2:/Table/83/1{-/00:00:…}] removing replica r242/2
I190814 05:43:30.854448 176379 storage/replica_destroy.go:146 [n1,replicaGC,s1,r242/2:/Table/83/1{-/00:00:…}] removed 7 (0+7) keys in 0ms [clear=0ms commit=0ms]
I190814 05:43:30.854805 173954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM interval_table WHERE a < '00:00:00':::INTERVAL
I190814 05:43:30.856258 176419 storage/replica_raft.go:291 [n3,s3,r242/1:/Table/83/1{-/00:00:…}] proposing ADD_REPLICA[(n1,s1):3LEARNER]: after=[(n3,s3):1 (n1,s1):3LEARNER] next=4
I190814 05:43:30.857100 176419 storage/store_snapshot.go:786 [n3,replicate,s3,r242/1:/Table/83/1{-/00:00:…}] sending LEARNER snapshot 25383b71 at applied index 24
I190814 05:43:30.857235 176419 storage/store_snapshot.go:829 [n3,replicate,s3,r242/1:/Table/83/1{-/00:00:…}] streamed snapshot to (n1,s1):3: kv pairs: 14, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s
I190814 05:43:30.857513 173954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM interval_table WHERE a IN ('00:00:00':::INTERVAL)
I190814 05:43:30.858089 176534 storage/replica_raftstorage.go:823 [n1,s1,r242/3:{-}] applying LEARNER snapshot at index 24 (id=25383b71, encoded size=2013, 1 rocksdb batches, 0 log entries)
I190814 05:43:30.858378 176534 storage/replica_raftstorage.go:829 [n1,s1,r242/3:/Table/83/1{-/00:00:…}] applied LEARNER snapshot in 0ms [clear=0ms batch=0ms entries=0ms commit=0ms]
I190814 05:43:30.858772 176419 storage/replica_command.go:1198 [n3,replicate,s3,r242/1:/Table/83/1{-/00:00:…}] change replicas (ADD_REPLICA (n1,s1):3): existing descriptor r242:/Table/83/1{-/00:00:00} [(n3,s3):1, (n1,s1):3LEARNER, next=4, gen=214]
I190814 05:43:30.860648 173954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM interval_table WHERE a < '00:00:00':::INTERVAL
I190814 05:43:30.863465 173954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM interval_table WHERE a < '00:00:00':::INTERVAL
I190814 05:43:30.866260 173954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM interval_table WHERE a < '00:00:00':::INTERVAL
I190814 05:43:30.867352 176419 storage/replica_raft.go:291 [n3,s3,r242/1:/Table/83/1{-/00:00:…}] proposing ADD_REPLICA[(n1,s1):3]: after=[(n3,s3):1 (n1,s1):3] next=4
E190814 05:43:30.867779 176419 storage/queue.go:1027 [n3,replicate,s3,r242/1:/Table/83/1{-/00:00:…}] no removable replicas from range that needs a removal: [1*:26, 3:25]
I190814 05:43:30.868235 176590 storage/replica_command.go:1198 [n3,replicate,s3,r245/1:/Table/83/{1/1 day…-2}] change replicas (REMOVE_REPLICA (n1,s1):2LEARNER): existing descriptor r245:/Table/83/{1/1 day -24:00:00-2} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=213]
I190814 05:43:30.869667 173954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM interval_table WHERE a = '00:00:00':::INTERVAL
I190814 05:43:30.872257 173954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM interval_table WHERE a < '00:00:00':::INTERVAL
I190814 05:43:30.872898 176590 storage/replica_raft.go:291 [n3,s3,r245/1:/Table/83/{1/1 day…-2}] proposing REMOVE_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1] next=3
I190814 05:43:30.873574 176590 storage/replica_command.go:1198 [n3,replicate,s3,r245/1:/Table/83/{1/1 day…-2}] change replicas (ADD_REPLICA (n1,s1):3LEARNER): existing descriptor r245:/Table/83/{1/1 day -24:00:00-2} [(n3,s3):1, next=3, gen=214]
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
C-test-failureBroken test (automatically or manually discovered).Broken test (automatically or manually discovered).O-robotOriginated from a bot.Originated from a bot.