Skip to content

storage: TestStoreRangeMergeRaftSnapshot failed under stress #33006

@cockroach-teamcity

Description

@cockroach-teamcity

SHA: https://github.com/cockroachdb/cockroach/commits/1cd7dcfe1bbf487a714c7f00a5dd709360dbe9e4

Parameters:

TAGS=
GOFLAGS=

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=TestStoreRangeMergeRaftSnapshot PKG=github.com/cockroachdb/cockroach/pkg/storage TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1049322&tab=buildLog

I181211 06:36:03.895686 37049 storage/client_merge_test.go:2865  SucceedsSoon: expected store2 to apply at least 1 additional raft snapshot
I181211 06:36:04.895891 37049 storage/client_merge_test.go:2865  SucceedsSoon: expected store2 to apply at least 1 additional raft snapshot
I181211 06:36:05.896115 37049 storage/client_merge_test.go:2865  SucceedsSoon: expected store2 to apply at least 1 additional raft snapshot
I181211 06:36:06.896334 37049 storage/client_merge_test.go:2865  SucceedsSoon: expected store2 to apply at least 1 additional raft snapshot
I181211 06:36:07.896595 37049 storage/client_merge_test.go:2865  SucceedsSoon: expected store2 to apply at least 1 additional raft snapshot
I181211 06:36:08.896840 37049 storage/client_merge_test.go:2865  SucceedsSoon: expected store2 to apply at least 1 additional raft snapshot
I181211 06:36:09.897122 37049 storage/client_merge_test.go:2865  SucceedsSoon: expected store2 to apply at least 1 additional raft snapshot
I181211 06:36:10.897380 37049 storage/client_merge_test.go:2865  SucceedsSoon: expected store2 to apply at least 1 additional raft snapshot
I181211 06:36:11.897567 37049 storage/client_merge_test.go:2865  SucceedsSoon: expected store2 to apply at least 1 additional raft snapshot
I181211 06:36:12.897781 37049 storage/client_merge_test.go:2865  SucceedsSoon: expected store2 to apply at least 1 additional raft snapshot
I181211 06:36:13.897991 37049 storage/client_merge_test.go:2865  SucceedsSoon: expected store2 to apply at least 1 additional raft snapshot
W181211 06:36:14.551227 37380 storage/store_rebalancer.go:226  [s3,store-rebalancer] StorePool missing descriptor for local store
I181211 06:36:14.898281 37049 storage/client_merge_test.go:2865  SucceedsSoon: expected store2 to apply at least 1 additional raft snapshot
I181211 06:36:15.898528 37049 storage/client_merge_test.go:2865  SucceedsSoon: expected store2 to apply at least 1 additional raft snapshot
I181211 06:36:16.898815 37049 storage/client_merge_test.go:2865  SucceedsSoon: expected store2 to apply at least 1 additional raft snapshot
I181211 06:36:17.898934 37049 storage/client_merge_test.go:2865  SucceedsSoon: expected store2 to apply at least 1 additional raft snapshot
I181211 06:36:18.899136 37049 storage/client_merge_test.go:2865  SucceedsSoon: expected store2 to apply at least 1 additional raft snapshot
I181211 06:36:19.899320 37049 storage/client_merge_test.go:2865  SucceedsSoon: expected store2 to apply at least 1 additional raft snapshot
I181211 06:36:20.899530 37049 storage/client_merge_test.go:2865  SucceedsSoon: expected store2 to apply at least 1 additional raft snapshot
I181211 06:36:21.899693 37049 storage/client_merge_test.go:2865  SucceedsSoon: expected store2 to apply at least 1 additional raft snapshot
I181211 06:36:22.899883 37049 storage/client_merge_test.go:2865  SucceedsSoon: expected store2 to apply at least 1 additional raft snapshot
I181211 06:36:23.900094 37049 storage/client_merge_test.go:2865  SucceedsSoon: expected store2 to apply at least 1 additional raft snapshot
W181211 06:36:24.901462 37272 gossip/gossip.go:1499  [n2] no incoming or outgoing connections
W181211 06:36:24.901564 37399 gossip/gossip.go:1499  [n3] no incoming or outgoing connections
--- FAIL: TestStoreRangeMergeRaftSnapshot (57.87s)
	client_merge_test.go:2865: condition failed to evaluate within 45s: expected store2 to apply at least 1 additional raft snapshot
		goroutine 37049 [running]:
		runtime/debug.Stack(0xa7a358200, 0xc420838cc0, 0x2e9afa0)
			/usr/local/go/src/runtime/debug/stack.go:24 +0xa7
		github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon(0x2ef6120, 0xc4215dc0f0, 0xc420838c90)
			/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:49 +0x10d
		github.com/cockroachdb/cockroach/pkg/storage_test.TestStoreRangeMergeRaftSnapshot(0xc4215dc0f0)
			/go/src/github.com/cockroachdb/cockroach/pkg/storage/client_merge_test.go:2865 +0xd1c
		testing.tRunner(0xc4215dc0f0, 0x2a4ab50)
			/usr/local/go/src/testing/testing.go:777 +0xd0
		created by testing.(*T).Run
			/usr/local/go/src/testing/testing.go:824 +0x2e0

Metadata

Metadata

Assignees

Labels

C-test-failureBroken test (automatically or manually discovered).O-robotOriginated from a bot.

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions