Skip to content

test hang at package table/tables #29632

@unconsolable

Description

@unconsolable

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

Run check_dev_2

2. What did you expect to see? (Required)

At least no test hang

3. What did you see instead (Required)

Test hang
https://ci.pingcap.net/blue/organizations/jenkins/tidb_ghpr_check_2/detail/tidb_ghpr_check_2/44389/pipeline/

[2021-11-10T00:09:21.408Z] panic: test timed out after 9m0s

[2021-11-10T00:09:21.408Z] 

[2021-11-10T00:09:21.408Z] goroutine 78408 [running]:

[2021-11-10T00:09:21.408Z] testing.(*M).startAlarm.func1()

[2021-11-10T00:09:21.408Z] 	/usr/local/go/src/testing/testing.go:1700 +0xe5

[2021-11-10T00:09:21.408Z] created by time.goFunc

[2021-11-10T00:09:21.408Z] 	/usr/local/go/src/time/sleep.go:180 +0x45

[2021-11-10T00:09:21.408Z] 

[2021-11-10T00:09:21.408Z] goroutine 1 [chan receive, 8 minutes]:

[2021-11-10T00:09:21.408Z] testing.tRunner.func1(0xc00058a600)

[2021-11-10T00:09:21.408Z] 	/usr/local/go/src/testing/testing.go:1159 +0x2bc

[2021-11-10T00:09:21.408Z] testing.tRunner(0xc00058a600, 0xc000dc7c60)

[2021-11-10T00:09:21.408Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.408Z] testing.runTests(0xc0010aa468, 0x58f2c60, 0x26, 0x26, 0xc05ae14b587facd8, 0x7dbdb64229, 0x5fd80a0, 0x38)

[2021-11-10T00:09:21.408Z] 	/usr/local/go/src/testing/testing.go:1509 +0x2fe

[2021-11-10T00:09:21.408Z] testing.(*M).Run(0xc001039900, 0x0)

[2021-11-10T00:09:21.408Z] 	/usr/local/go/src/testing/testing.go:1417 +0x1eb

[2021-11-10T00:09:21.408Z] go.uber.org/goleak.VerifyTestMain(0x41574c0, 0xc001039900, 0xc0014c7e60, 0x3, 0x3)

[2021-11-10T00:09:21.408Z] 	/nfs/cache/mod/go.uber.org/goleak@v1.1.11-0.20210813005559-691160354723/testmain.go:53 +0x38

[2021-11-10T00:09:21.408Z] github.com/pingcap/tidb/table/tables_test.TestMain(0xc001039900)

[2021-11-10T00:09:21.408Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/table/tables/main_test.go:31 +0x125

[2021-11-10T00:09:21.408Z] main.main()

[2021-11-10T00:09:21.408Z] 	_testmain.go:177 +0x1f6

[2021-11-10T00:09:21.408Z] 

[2021-11-10T00:09:21.408Z] goroutine 71 [select]:

[2021-11-10T00:09:21.408Z] go.opencensus.io/stats/view.(*worker).start(0xc000ae2b00)

[2021-11-10T00:09:21.408Z] 	/nfs/cache/mod/go.opencensus.io@v0.23.0/stats/view/worker.go:276 +0xcd

[2021-11-10T00:09:21.408Z] created by go.opencensus.io/stats/view.init.0

[2021-11-10T00:09:21.408Z] 	/nfs/cache/mod/go.opencensus.io@v0.23.0/stats/view/worker.go:34 +0x68

[2021-11-10T00:09:21.408Z] 

[2021-11-10T00:09:21.408Z] goroutine 149 [chan receive]:

[2021-11-10T00:09:21.408Z] go.etcd.io/etcd/pkg/logutil.(*MergeLogger).outputLoop(0xc000521aa0)

[2021-11-10T00:09:21.408Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/pkg/logutil/merge_logger.go:173 +0x3ac

[2021-11-10T00:09:21.408Z] created by go.etcd.io/etcd/pkg/logutil.NewMergeLogger

[2021-11-10T00:09:21.408Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/pkg/logutil/merge_logger.go:91 +0x85

[2021-11-10T00:09:21.408Z] 

[2021-11-10T00:09:21.408Z] goroutine 171 [chan receive]:

[2021-11-10T00:09:21.408Z] go.etcd.io/etcd/pkg/logutil.(*MergeLogger).outputLoop(0xc0005a3590)

[2021-11-10T00:09:21.408Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/pkg/logutil/merge_logger.go:173 +0x3ac

[2021-11-10T00:09:21.408Z] created by go.etcd.io/etcd/pkg/logutil.NewMergeLogger

[2021-11-10T00:09:21.408Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/pkg/logutil/merge_logger.go:91 +0x85

[2021-11-10T00:09:21.408Z] 

[2021-11-10T00:09:21.408Z] goroutine 49 [chan receive]:

[2021-11-10T00:09:21.408Z] go.etcd.io/etcd/pkg/logutil.(*MergeLogger).outputLoop(0xc00000f398)

[2021-11-10T00:09:21.408Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/pkg/logutil/merge_logger.go:173 +0x3ac

[2021-11-10T00:09:21.408Z] created by go.etcd.io/etcd/pkg/logutil.NewMergeLogger

[2021-11-10T00:09:21.409Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/pkg/logutil/merge_logger.go:91 +0x85

[2021-11-10T00:09:21.409Z] 

[2021-11-10T00:09:21.409Z] goroutine 1110 [chan receive, 8 minutes]:

[2021-11-10T00:09:21.409Z] github.com/pingcap/tidb/table/tables_test.TestCacheTableComplexRead(0xc00592c300)

[2021-11-10T00:09:21.409Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/table/tables/cache_test.go:235 +0x3a9

[2021-11-10T00:09:21.409Z] testing.tRunner(0xc00592c300, 0x3d80c08)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1193 +0xef

[2021-11-10T00:09:21.409Z] created by testing.(*T).Run

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.409Z] 

[2021-11-10T00:09:21.409Z] goroutine 23311 [chan send, 8 minutes]:

[2021-11-10T00:09:21.409Z] testing.tRunner.func1.1(0xc073a67ed5, 0x0, 0x0, 0xc004d72780, 0xc073a67e01)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.409Z] testing.tRunner.func1(0xc004d72780)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.409Z] testing.tRunner(0xc004d72780, 0x3d80ce0)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.409Z] created by testing.(*T).Run

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.409Z] 

[2021-11-10T00:09:21.409Z] goroutine 24014 [select, 4 minutes]:

[2021-11-10T00:09:21.409Z] github.com/pingcap/tidb/store/mockstore/unistore/tikv.(*MVCCStore).StartDeadlockDetection.func1(0xc000ef6380)

[2021-11-10T00:09:21.409Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/store/mockstore/unistore/tikv/mvcc.go:1286 +0x8a

[2021-11-10T00:09:21.409Z] created by github.com/pingcap/tidb/store/mockstore/unistore/tikv.(*MVCCStore).StartDeadlockDetection

[2021-11-10T00:09:21.409Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/store/mockstore/unistore/tikv/mvcc.go:1284 +0x78

[2021-11-10T00:09:21.409Z] 

[2021-11-10T00:09:21.409Z] goroutine 723 [chan receive, 8 minutes]:

[2021-11-10T00:09:21.409Z] github.com/pingcap/tidb/table/tables.mockRemoteService(0xc0052776f8, 0xc0064f4ae0)

[2021-11-10T00:09:21.409Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/table/tables/state_remote.go:109 +0x7a

[2021-11-10T00:09:21.409Z] created by github.com/pingcap/tidb/table/tables.NewCachedTable

[2021-11-10T00:09:21.409Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/table/tables/cache.go:81 +0x1dd

[2021-11-10T00:09:21.409Z] 

[2021-11-10T00:09:21.409Z] goroutine 23377 [chan send, 8 minutes]:

[2021-11-10T00:09:21.409Z] testing.tRunner.func1.1(0xc07a549ed5, 0x0, 0x0, 0xc00074c900, 0xc07a549e01)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.409Z] testing.tRunner.func1(0xc00074c900)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.409Z] testing.tRunner(0xc00074c900, 0x3d80c70)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.409Z] created by testing.(*T).Run

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.409Z] 

[2021-11-10T00:09:21.409Z] goroutine 23384 [chan send, 8 minutes]:

[2021-11-10T00:09:21.409Z] testing.tRunner.func1.1(0xc005b33ed5, 0x0, 0x0, 0xc00074d380, 0xc005b33e01)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.409Z] testing.tRunner.func1(0xc00074d380)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.409Z] testing.tRunner(0xc00074d380, 0x3d80ce8)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.409Z] created by testing.(*T).Run

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.409Z] 

[2021-11-10T00:09:21.409Z] goroutine 23379 [chan send, 8 minutes]:

[2021-11-10T00:09:21.409Z] testing.tRunner.func1.1(0xc00206ded5, 0x0, 0x0, 0xc00074cc00, 0xc00206de01)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.409Z] testing.tRunner.func1(0xc00074cc00)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.409Z] testing.tRunner(0xc00074cc00, 0x3d80cf8)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.409Z] created by testing.(*T).Run

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.409Z] 

[2021-11-10T00:09:21.409Z] goroutine 24179 [select, 8 minutes]:

[2021-11-10T00:09:21.409Z] github.com/dgraph-io/ristretto.(*defaultPolicy).processItems(0xc003460ae0)

[2021-11-10T00:09:21.409Z] 	/nfs/cache/mod/github.com/dgraph-io/ristretto@v0.0.1/policy.go:96 +0xa5

[2021-11-10T00:09:21.409Z] created by github.com/dgraph-io/ristretto.newDefaultPolicy

[2021-11-10T00:09:21.409Z] 	/nfs/cache/mod/github.com/dgraph-io/ristretto@v0.0.1/policy.go:80 +0x129

[2021-11-10T00:09:21.409Z] 

[2021-11-10T00:09:21.409Z] goroutine 23370 [chan send, 8 minutes]:

[2021-11-10T00:09:21.409Z] testing.tRunner.func1.1(0xc078911ed5, 0x0, 0x0, 0xc004d73c80, 0xc078911e01)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.409Z] testing.tRunner.func1(0xc004d73c80)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.409Z] testing.tRunner(0xc004d73c80, 0x3d80cf0)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.409Z] created by testing.(*T).Run

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.409Z] 

[2021-11-10T00:09:21.409Z] goroutine 23312 [chan send, 8 minutes]:

[2021-11-10T00:09:21.409Z] testing.tRunner.func1.1(0xc00551bed5, 0x0, 0x0, 0xc004d72900, 0xc00551be01)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.409Z] testing.tRunner.func1(0xc004d72900)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.409Z] testing.tRunner(0xc004d72900, 0x3d80cb8)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.409Z] created by testing.(*T).Run

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.409Z] 

[2021-11-10T00:09:21.409Z] goroutine 23313 [chan send, 8 minutes]:

[2021-11-10T00:09:21.409Z] testing.tRunner.func1.1(0xc005bb3ed5, 0x0, 0x0, 0xc004d72a80, 0xc005bb3e01)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.409Z] testing.tRunner.func1(0xc004d72a80)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.409Z] testing.tRunner(0xc004d72a80, 0x3d80cb0)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.409Z] created by testing.(*T).Run

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.409Z] 

[2021-11-10T00:09:21.409Z] goroutine 23365 [chan send, 8 minutes]:

[2021-11-10T00:09:21.409Z] testing.tRunner.func1.1(0xc0746abed5, 0x0, 0x0, 0xc004d73200, 0xc0746abe01)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.409Z] testing.tRunner.func1(0xc004d73200)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.409Z] testing.tRunner(0xc004d73200, 0x3d80c90)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.409Z] created by testing.(*T).Run

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.409Z] 

[2021-11-10T00:09:21.409Z] goroutine 31595 [select]:

[2021-11-10T00:09:21.409Z] github.com/pingcap/tidb/domain.(*Domain).handleEvolvePlanTasksLoop.func1(0xc07ee167e0, 0x41c16d8, 0xc07fb6eda0, 0x41ed530, 0xc0760efc00)

[2021-11-10T00:09:21.409Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:1106 +0xde

[2021-11-10T00:09:21.409Z] created by github.com/pingcap/tidb/domain.(*Domain).handleEvolvePlanTasksLoop

[2021-11-10T00:09:21.409Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:1099 +0x8f

[2021-11-10T00:09:21.409Z] 

[2021-11-10T00:09:21.409Z] goroutine 23369 [chan send, 8 minutes]:

[2021-11-10T00:09:21.409Z] testing.tRunner.func1.1(0xc075ab5ed5, 0x0, 0x0, 0xc004d73b00, 0xc075ab5e01)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.409Z] testing.tRunner.func1(0xc004d73b00)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.409Z] testing.tRunner(0xc004d73b00, 0x3d80c80)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.410Z] created by testing.(*T).Run

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.410Z] 

[2021-11-10T00:09:21.410Z] goroutine 23599 [select]:

[2021-11-10T00:09:21.410Z] github.com/pingcap/badger.(*levelsController).runWorker(0xc001ec62a0, 0xc002982138, 0x0)

[2021-11-10T00:09:21.410Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/levels.go:212 +0x185

[2021-11-10T00:09:21.410Z] created by github.com/pingcap/badger.(*levelsController).startCompact

[2021-11-10T00:09:21.410Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/levels.go:180 +0x93

[2021-11-10T00:09:21.410Z] 

[2021-11-10T00:09:21.410Z] goroutine 23371 [chan send, 8 minutes]:

[2021-11-10T00:09:21.410Z] testing.tRunner.func1.1(0xc00284bed5, 0x0, 0x0, 0xc004d73e00, 0xc00284be01)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.410Z] testing.tRunner.func1(0xc004d73e00)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.410Z] testing.tRunner(0xc004d73e00, 0x3d80c28)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.410Z] created by testing.(*T).Run

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.410Z] 

[2021-11-10T00:09:21.410Z] goroutine 23387 [chan send, 8 minutes]:

[2021-11-10T00:09:21.410Z] testing.tRunner.func1.1(0xc072c1ded5, 0x0, 0x0, 0xc00074d980, 0xc072c1de01)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.410Z] testing.tRunner.func1(0xc00074d980)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.410Z] testing.tRunner(0xc00074d980, 0x3d80bd0)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.410Z] created by testing.(*T).Run

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.410Z] 

[2021-11-10T00:09:21.410Z] goroutine 23368 [chan send, 8 minutes]:

[2021-11-10T00:09:21.410Z] testing.tRunner.func1.1(0xc006759ed5, 0x0, 0x0, 0xc004d73980, 0xc006759e01)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.410Z] testing.tRunner.func1(0xc004d73980)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.410Z] testing.tRunner(0xc004d73980, 0x3d80ca8)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.410Z] created by testing.(*T).Run

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.410Z] 

[2021-11-10T00:09:21.410Z] goroutine 23373 [chan send, 8 minutes]:

[2021-11-10T00:09:21.410Z] testing.tRunner.func1.1(0xc0792abed5, 0x0, 0x0, 0xc00074c300, 0xc0792abe01)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.410Z] testing.tRunner.func1(0xc00074c300)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.410Z] testing.tRunner(0xc00074c300, 0x3d80c60)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.410Z] created by testing.(*T).Run

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.410Z] 

[2021-11-10T00:09:21.410Z] goroutine 23388 [chan send, 8 minutes]:

[2021-11-10T00:09:21.410Z] testing.tRunner.func1.1(0xc07b011ed5, 0x0, 0x0, 0xc00074db00, 0xc07b011e01)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.410Z] testing.tRunner.func1(0xc00074db00)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.410Z] testing.tRunner(0xc00074db00, 0x3d80c20)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.410Z] created by testing.(*T).Run

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.410Z] 

[2021-11-10T00:09:21.410Z] goroutine 23364 [chan send, 8 minutes]:

[2021-11-10T00:09:21.410Z] testing.tRunner.func1.1(0xc073a25ed5, 0x0, 0x0, 0xc004d72f00, 0xc073a25e01)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.410Z] testing.tRunner.func1(0xc004d72f00)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.410Z] testing.tRunner(0xc004d72f00, 0x3d80c30)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.410Z] created by testing.(*T).Run

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.410Z] 

[2021-11-10T00:09:21.410Z] goroutine 23600 [chan receive, 8 minutes]:

[2021-11-10T00:09:21.410Z] github.com/pingcap/badger.(*DB).runFlushMemTable(0xc002c3e900, 0xc00210e168, 0x0, 0x0)

[2021-11-10T00:09:21.410Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/db.go:920 +0x190

[2021-11-10T00:09:21.410Z] created by github.com/pingcap/badger.Open

[2021-11-10T00:09:21.410Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/db.go:361 +0x17b0

[2021-11-10T00:09:21.410Z] 

[2021-11-10T00:09:21.410Z] goroutine 24011 [select, 4 minutes]:

[2021-11-10T00:09:21.410Z] github.com/pingcap/tidb/store/mockstore/unistore/tikv.writeDBWorker.run(0xc002c9f7a0, 0xc002168800)

[2021-11-10T00:09:21.410Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/store/mockstore/unistore/tikv/write.go:96 +0x205

[2021-11-10T00:09:21.410Z] created by github.com/pingcap/tidb/store/mockstore/unistore/tikv.(*dbWriter).Open

[2021-11-10T00:09:21.410Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/store/mockstore/unistore/tikv/write.go:199 +0x8f

[2021-11-10T00:09:21.410Z] 

[2021-11-10T00:09:21.410Z] goroutine 23380 [chan send, 8 minutes]:

[2021-11-10T00:09:21.410Z] testing.tRunner.func1.1(0xc0722cfed5, 0x0, 0x0, 0xc00074cd80, 0xc0722cfe01)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.410Z] testing.tRunner.func1(0xc00074cd80)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.410Z] testing.tRunner(0xc00074cd80, 0x3d80d00)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.410Z] created by testing.(*T).Run

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.410Z] 

[2021-11-10T00:09:21.410Z] goroutine 23598 [select, 8 minutes]:

[2021-11-10T00:09:21.410Z] github.com/pingcap/badger.(*blobGCHandler).run(0xc0011a81c0, 0xc002982120)

[2021-11-10T00:09:21.410Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/blob.go:468 +0xd4

[2021-11-10T00:09:21.410Z] created by github.com/pingcap/badger.(*blobManager).Open

[2021-11-10T00:09:21.410Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/blob.go:292 +0x618

[2021-11-10T00:09:21.410Z] 

[2021-11-10T00:09:21.410Z] goroutine 23375 [chan send, 8 minutes]:

[2021-11-10T00:09:21.410Z] testing.tRunner.func1.1(0xc079371ed5, 0x0, 0x0, 0xc00074c600, 0xc079371e01)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.410Z] testing.tRunner.func1(0xc00074c600)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.410Z] testing.tRunner(0xc00074c600, 0x3d80c48)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.410Z] created by testing.(*T).Run

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.410Z] 

[2021-11-10T00:09:21.410Z] goroutine 23363 [chan send, 8 minutes]:

[2021-11-10T00:09:21.410Z] testing.tRunner.func1.1(0xc00526fed5, 0x0, 0x0, 0xc004d72d80, 0xc00526fe01)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.410Z] testing.tRunner.func1(0xc004d72d80)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.410Z] testing.tRunner(0xc004d72d80, 0x3d80cc0)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.410Z] created by testing.(*T).Run

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.410Z] 

[2021-11-10T00:09:21.410Z] goroutine 32762 [select, 8 minutes]:

[2021-11-10T00:09:21.410Z] github.com/pingcap/tidb/domain.(*Domain).PlanReplayerLoop.func1(0xc07ee167e0)

[2021-11-10T00:09:21.410Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:1191 +0x11b

[2021-11-10T00:09:21.410Z] created by github.com/pingcap/tidb/domain.(*Domain).PlanReplayerLoop

[2021-11-10T00:09:21.410Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:1182 +0x5f

[2021-11-10T00:09:21.410Z] 

[2021-11-10T00:09:21.410Z] goroutine 23385 [chan send, 8 minutes]:

[2021-11-10T00:09:21.410Z] testing.tRunner.func1.1(0xc001377ed5, 0x0, 0x0, 0xc00074d680, 0xc001377e01)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.410Z] testing.tRunner.func1(0xc00074d680)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.410Z] testing.tRunner(0xc00074d680, 0x3d80cd8)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.410Z] created by testing.(*T).Run

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.411Z] 

[2021-11-10T00:09:21.411Z] goroutine 24017 [select]:

[2021-11-10T00:09:21.411Z] github.com/tikv/client-go/v2/tikv.(*KVStore).runSafePointChecker(0xc0025cc480)

[2021-11-10T00:09:21.411Z] 	/nfs/cache/mod/github.com/tikv/client-go/v2@v2.0.0-alpha.0.20211102120533-b8cc5a319d96/tikv/kv.go:264 +0x136

[2021-11-10T00:09:21.411Z] created by github.com/tikv/client-go/v2/tikv.NewKVStore

[2021-11-10T00:09:21.411Z] 	/nfs/cache/mod/github.com/tikv/client-go/v2@v2.0.0-alpha.0.20211102120533-b8cc5a319d96/tikv/kv.go:187 +0x38a

[2021-11-10T00:09:21.411Z] 

[2021-11-10T00:09:21.411Z] goroutine 23381 [chan send, 8 minutes]:

[2021-11-10T00:09:21.411Z] testing.tRunner.func1.1(0xc0068d7ed5, 0x0, 0x0, 0xc00074cf00, 0xc0068d7e01)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.411Z] testing.tRunner.func1(0xc00074cf00)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.411Z] testing.tRunner(0xc00074cf00, 0x3d80cd0)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.411Z] created by testing.(*T).Run

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.411Z] 

[2021-11-10T00:09:21.411Z] goroutine 23378 [chan send, 8 minutes]:

[2021-11-10T00:09:21.411Z] testing.tRunner.func1.1(0xc078f45ed5, 0x0, 0x0, 0xc00074ca80, 0xc078f45e01)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.411Z] testing.tRunner.func1(0xc00074ca80)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.411Z] testing.tRunner(0xc00074ca80, 0x3d80be0)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.411Z] created by testing.(*T).Run

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.411Z] 

[2021-11-10T00:09:21.411Z] goroutine 23382 [chan send, 8 minutes]:

[2021-11-10T00:09:21.411Z] testing.tRunner.func1.1(0xc0776c1ed5, 0x0, 0x0, 0xc00074d080, 0xc0776c1e01)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.411Z] testing.tRunner.func1(0xc00074d080)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.411Z] testing.tRunner(0xc00074d080, 0x3d80d08)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.411Z] created by testing.(*T).Run

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.411Z] 

[2021-11-10T00:09:21.411Z] goroutine 23376 [chan send, 8 minutes]:

[2021-11-10T00:09:21.411Z] testing.tRunner.func1.1(0xc005f77ed5, 0x0, 0x0, 0xc00074c780, 0xc005f77e01)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.411Z] testing.tRunner.func1(0xc00074c780)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.411Z] testing.tRunner(0xc00074c780, 0x3d80c68)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.411Z] created by testing.(*T).Run

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.411Z] 

[2021-11-10T00:09:21.411Z] goroutine 23366 [chan send, 8 minutes]:

[2021-11-10T00:09:21.411Z] testing.tRunner.func1.1(0xc075e53ed5, 0x0, 0x0, 0xc004d73380, 0xc075e53e01)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.411Z] testing.tRunner.func1(0xc004d73380)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.411Z] testing.tRunner(0xc004d73380, 0x3d80c98)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.411Z] created by testing.(*T).Run

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.411Z] 

[2021-11-10T00:09:21.411Z] goroutine 24015 [select]:

[2021-11-10T00:09:21.411Z] github.com/tikv/client-go/v2/oracle/oracles.(*pdOracle).updateTS(0xc0009480e0, 0x41a48a0, 0xc000134018, 0x77359400)

[2021-11-10T00:09:21.411Z] 	/nfs/cache/mod/github.com/tikv/client-go/v2@v2.0.0-alpha.0.20211102120533-b8cc5a319d96/oracle/oracles/pd.go:229 +0x131

[2021-11-10T00:09:21.411Z] created by github.com/tikv/client-go/v2/oracle/oracles.NewPdOracle

[2021-11-10T00:09:21.411Z] 	/nfs/cache/mod/github.com/tikv/client-go/v2@v2.0.0-alpha.0.20211102120533-b8cc5a319d96/oracle/oracles/pd.go:77 +0xd3

[2021-11-10T00:09:21.411Z] 

[2021-11-10T00:09:21.411Z] goroutine 23386 [chan send, 8 minutes]:

[2021-11-10T00:09:21.411Z] testing.tRunner.func1.1(0xc075819ed5, 0x0, 0x0, 0xc00074d800, 0xc075819e01)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.411Z] testing.tRunner.func1(0xc00074d800)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.411Z] testing.tRunner(0xc00074d800, 0x3d80c50)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.411Z] created by testing.(*T).Run

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.411Z] 

[2021-11-10T00:09:21.411Z] goroutine 23389 [chan send, 8 minutes]:

[2021-11-10T00:09:21.411Z] testing.tRunner.func1.1(0xc09421fed5, 0x0, 0x0, 0xc00074dc80, 0xc09421fe01)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.411Z] testing.tRunner.func1(0xc00074dc80)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.411Z] testing.tRunner(0xc00074dc80, 0x3d80d10)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.411Z] created by testing.(*T).Run

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.411Z] 

[2021-11-10T00:09:21.411Z] goroutine 23362 [chan send, 8 minutes]:

[2021-11-10T00:09:21.411Z] testing.tRunner.func1.1(0xc0774abed5, 0x0, 0x0, 0xc004d72c00, 0xc0774abe01)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.411Z] testing.tRunner.func1(0xc004d72c00)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.411Z] testing.tRunner(0xc004d72c00, 0x3d80c38)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.411Z] created by testing.(*T).Run

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.411Z] 

[2021-11-10T00:09:21.411Z] goroutine 31163 [select, 8 minutes]:

[2021-11-10T00:09:21.411Z] github.com/pingcap/tidb/ddl.(*delRange).startEmulator(0xc07c9f31d0)

[2021-11-10T00:09:21.411Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/ddl/delete_range.go:144 +0x128

[2021-11-10T00:09:21.411Z] created by github.com/pingcap/tidb/ddl.(*delRange).start

[2021-11-10T00:09:21.411Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/ddl/delete_range.go:126 +0x6a

[2021-11-10T00:09:21.411Z] 

[2021-11-10T00:09:21.411Z] goroutine 24178 [select]:

[2021-11-10T00:09:21.411Z] github.com/tikv/client-go/v2/tikv.(*KVStore).safeTSUpdater(0xc0025cc480)

[2021-11-10T00:09:21.411Z] 	/nfs/cache/mod/github.com/tikv/client-go/v2@v2.0.0-alpha.0.20211102120533-b8cc5a319d96/tikv/kv.go:544 +0x194

[2021-11-10T00:09:21.411Z] created by github.com/tikv/client-go/v2/tikv.NewKVStore

[2021-11-10T00:09:21.411Z] 	/nfs/cache/mod/github.com/tikv/client-go/v2@v2.0.0-alpha.0.20211102120533-b8cc5a319d96/tikv/kv.go:188 +0x3ac

[2021-11-10T00:09:21.411Z] 

[2021-11-10T00:09:21.411Z] goroutine 23374 [chan send, 8 minutes]:

[2021-11-10T00:09:21.411Z] testing.tRunner.func1.1(0xc004e23ed5, 0x0, 0x0, 0xc00074c480, 0xc004e23e01)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.411Z] testing.tRunner.func1(0xc00074c480)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.411Z] testing.tRunner(0xc00074c480, 0x3d80c40)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.411Z] created by testing.(*T).Run

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.411Z] 

[2021-11-10T00:09:21.411Z] goroutine 23383 [chan send, 8 minutes]:

[2021-11-10T00:09:21.411Z] testing.tRunner.func1.1(0xc0771a3ed5, 0x0, 0x0, 0xc00074d200, 0xc0771a3e01)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.411Z] testing.tRunner.func1(0xc00074d200)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.411Z] testing.tRunner(0xc00074d200, 0x3d80c78)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.411Z] created by testing.(*T).Run

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.411Z] 

[2021-11-10T00:09:21.411Z] goroutine 24012 [select, 4 minutes]:

[2021-11-10T00:09:21.411Z] github.com/pingcap/tidb/store/mockstore/unistore/tikv.writeLockWorker.run(0xc002c9f800, 0xc002168800)

[2021-11-10T00:09:21.412Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/store/mockstore/unistore/tikv/write.go:148 +0x3e5

[2021-11-10T00:09:21.412Z] created by github.com/pingcap/tidb/store/mockstore/unistore/tikv.(*dbWriter).Open

[2021-11-10T00:09:21.412Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/store/mockstore/unistore/tikv/write.go:206 +0xdc

[2021-11-10T00:09:21.412Z] 

[2021-11-10T00:09:21.412Z] goroutine 23367 [chan send, 8 minutes]:

[2021-11-10T00:09:21.412Z] testing.tRunner.func1.1(0xc0721f5ed5, 0x0, 0x0, 0xc004d73680, 0xc0721f5e01)

[2021-11-10T00:09:21.412Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.412Z] testing.tRunner.func1(0xc004d73680)

[2021-11-10T00:09:21.412Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.412Z] testing.tRunner(0xc004d73680, 0x3d80c88)

[2021-11-10T00:09:21.412Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.412Z] created by testing.(*T).Run

[2021-11-10T00:09:21.412Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.412Z] 

[2021-11-10T00:09:21.412Z] goroutine 23390 [chan receive, 8 minutes]:

[2021-11-10T00:09:21.412Z] testing.runTests.func1.1(0xc00058a600)

[2021-11-10T00:09:21.412Z] 	/usr/local/go/src/testing/testing.go:1516 +0x3b

[2021-11-10T00:09:21.412Z] created by testing.runTests.func1

[2021-11-10T00:09:21.412Z] 	/usr/local/go/src/testing/testing.go:1516 +0xac

[2021-11-10T00:09:21.412Z] 

[2021-11-10T00:09:21.412Z] goroutine 24180 [select, 8 minutes]:

[2021-11-10T00:09:21.412Z] github.com/dgraph-io/ristretto.(*Cache).processItems(0xc002c9fbc0)

[2021-11-10T00:09:21.412Z] 	/nfs/cache/mod/github.com/dgraph-io/ristretto@v0.0.1/cache.go:257 +0x9d

[2021-11-10T00:09:21.412Z] created by github.com/dgraph-io/ristretto.NewCache

[2021-11-10T00:09:21.412Z] 	/nfs/cache/mod/github.com/dgraph-io/ristretto@v0.0.1/cache.go:155 +0x205

[2021-11-10T00:09:21.412Z] 

[2021-11-10T00:09:21.412Z] goroutine 24013 [select, 2 minutes]:

[2021-11-10T00:09:21.412Z] github.com/pingcap/tidb/store/mockstore/unistore/tikv.(*MVCCStore).runUpdateSafePointLoop(0xc000ef6380)

[2021-11-10T00:09:21.412Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/store/mockstore/unistore/tikv/mvcc.go:1550 +0x28a

[2021-11-10T00:09:21.412Z] created by github.com/pingcap/tidb/store/mockstore/unistore/tikv.NewMVCCStore

[2021-11-10T00:09:21.412Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/store/mockstore/unistore/tikv/mvcc.go:85 +0x269

[2021-11-10T00:09:21.412Z] 

[2021-11-10T00:09:21.412Z] goroutine 23372 [chan send, 8 minutes]:

[2021-11-10T00:09:21.412Z] testing.tRunner.func1.1(0xc071d07ed5, 0x0, 0x0, 0xc00074c180, 0xc071d07e01)

[2021-11-10T00:09:21.412Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.412Z] testing.tRunner.func1(0xc00074c180)

[2021-11-10T00:09:21.412Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.412Z] testing.tRunner(0xc00074c180, 0x3d80cc8)

[2021-11-10T00:09:21.412Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.412Z] created by testing.(*T).Run

[2021-11-10T00:09:21.412Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.412Z] 

[2021-11-10T00:09:21.412Z] goroutine 24016 [select, 2 minutes]:

[2021-11-10T00:09:21.412Z] github.com/tikv/client-go/v2/internal/locate.(*RegionCache).asyncCheckAndResolveLoop(0xc000ef6480, 0xdf8475800)

[2021-11-10T00:09:21.412Z] 	/nfs/cache/mod/github.com/tikv/client-go/v2@v2.0.0-alpha.0.20211102120533-b8cc5a319d96/internal/locate/region_cache.go:399 +0xdc

[2021-11-10T00:09:21.412Z] created by github.com/tikv/client-go/v2/internal/locate.NewRegionCache

[2021-11-10T00:09:21.412Z] 	/nfs/cache/mod/github.com/tikv/client-go/v2@v2.0.0-alpha.0.20211102120533-b8cc5a319d96/internal/locate/region_cache.go:370 +0x259

[2021-11-10T00:09:21.412Z] 

[2021-11-10T00:09:21.412Z] goroutine 23418 [select, 2 minutes]:

[2021-11-10T00:09:21.412Z] github.com/pingcap/badger.(*DB).updateSize(0xc002c3e900, 0xc0019e0198)

[2021-11-10T00:09:21.412Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/db.go:1039 +0x108

[2021-11-10T00:09:21.412Z] created by github.com/pingcap/badger.Open

[2021-11-10T00:09:21.412Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/db.go:328 +0xe14

[2021-11-10T00:09:21.412Z] 

[2021-11-10T00:09:21.412Z] goroutine 23419 [select]:

[2021-11-10T00:09:21.412Z] github.com/pingcap/badger/epoch.(*ResourceManager).collectLoop(0xc001271740, 0xc0019e01b0)

[2021-11-10T00:09:21.412Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/epoch/manager.go:101 +0xdc

[2021-11-10T00:09:21.412Z] created by github.com/pingcap/badger/epoch.NewResourceManager

[2021-11-10T00:09:21.412Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/epoch/manager.go:79 +0xa7

[2021-11-10T00:09:21.412Z] 

[2021-11-10T00:09:21.412Z] goroutine 23560 [select, 8 minutes]:

[2021-11-10T00:09:21.412Z] github.com/pingcap/badger.Open.func4(0xc005cb0000)

[2021-11-10T00:09:21.412Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/db.go:342 +0x17c

[2021-11-10T00:09:21.412Z] created by github.com/pingcap/badger.Open

[2021-11-10T00:09:21.412Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/db.go:339 +0x10a8

[2021-11-10T00:09:21.412Z] 

[2021-11-10T00:09:21.412Z] goroutine 24010 [chan receive, 4 minutes]:

[2021-11-10T00:09:21.412Z] github.com/pingcap/badger.(*writeWorker).runMergeLSM(0xc002c96460, 0xc001d3c180)

[2021-11-10T00:09:21.412Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/writer.go:154 +0xad

[2021-11-10T00:09:21.412Z] created by github.com/pingcap/badger.startWriteWorker

[2021-11-10T00:09:21.412Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/writer.go:66 +0x1e8

[2021-11-10T00:09:21.412Z] 

[2021-11-10T00:09:21.412Z] goroutine 24009 [chan receive, 4 minutes, locked to thread]:

[2021-11-10T00:09:21.412Z] github.com/pingcap/badger.(*writeWorker).runWriteLSM(0xc002c96460, 0xc001d3c180)

[2021-11-10T00:09:21.412Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/writer.go:141 +0x195

[2021-11-10T00:09:21.412Z] created by github.com/pingcap/badger.startWriteWorker

[2021-11-10T00:09:21.412Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/writer.go:65 +0x1bc

[2021-11-10T00:09:21.412Z] 

[2021-11-10T00:09:21.412Z] goroutine 24008 [select, 4 minutes]:

[2021-11-10T00:09:21.412Z] github.com/pingcap/badger.(*writeWorker).runWriteVLog(0xc002c96460, 0xc001d3c180)

[2021-11-10T00:09:21.412Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/writer.go:94 +0x1dc

[2021-11-10T00:09:21.412Z] created by github.com/pingcap/badger.startWriteWorker

[2021-11-10T00:09:21.412Z] 	/nfs/cache/mod/github.com/pingcap/badger@v1.5.1-0.20210831093107-2f6cb8008145/writer.go:64 +0x190

[2021-11-10T00:09:21.412Z] 

[2021-11-10T00:09:21.412Z] goroutine 33250 [select]:

[2021-11-10T00:09:21.412Z] github.com/pingcap/tidb/domain.(*Domain).LoadSysVarCacheLoop.func1(0xc07ee167e0, 0xc073b59818, 0x6fc23ac00, 0x41ed530, 0xc07ee09a00)

[2021-11-10T00:09:21.412Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:989 +0x125

[2021-11-10T00:09:21.412Z] created by github.com/pingcap/tidb/domain.(*Domain).LoadSysVarCacheLoop

[2021-11-10T00:09:21.412Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:980 +0x10e

[2021-11-10T00:09:21.412Z] 

[2021-11-10T00:09:21.412Z] goroutine 31594 [select]:

[2021-11-10T00:09:21.412Z] github.com/pingcap/tidb/domain.(*Domain).globalBindHandleWorkerLoop.func1(0xc07ee167e0, 0x41c16d8, 0xc07fb6eda0)

[2021-11-10T00:09:21.412Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:1071 +0x157

[2021-11-10T00:09:21.412Z] created by github.com/pingcap/tidb/domain.(*Domain).globalBindHandleWorkerLoop

[2021-11-10T00:09:21.412Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:1058 +0x73

[2021-11-10T00:09:21.412Z] 

[2021-11-10T00:09:21.412Z] goroutine 32760 [select, 8 minutes]:

[2021-11-10T00:09:21.412Z] github.com/pingcap/tidb/domain.(*Domain).TelemetryReportLoop.func1(0xc07ee167e0, 0x41ed530, 0xc086018000)

[2021-11-10T00:09:21.412Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:1140 +0x12f

[2021-11-10T00:09:21.412Z] created by github.com/pingcap/tidb/domain.(*Domain).TelemetryReportLoop

[2021-11-10T00:09:21.412Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:1132 +0x2c6

[2021-11-10T00:09:21.412Z] 

[2021-11-10T00:09:21.412Z] goroutine 31164 [select]:

[2021-11-10T00:09:21.412Z] github.com/pingcap/tidb/ddl.(*worker).start(0xc07cfddc30, 0xc07c9c6300)

[2021-11-10T00:09:21.412Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/ddl/ddl_worker.go:178 +0x315

[2021-11-10T00:09:21.412Z] created by github.com/pingcap/tidb/ddl.(*ddl).Start

[2021-11-10T00:09:21.412Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/ddl/ddl.go:369 +0x6a5

[2021-11-10T00:09:21.412Z] 

[2021-11-10T00:09:21.412Z] goroutine 31162 [select, 8 minutes]:

[2021-11-10T00:09:21.412Z] github.com/pingcap/tidb/ddl.(*ddl).limitDDLJobs(0xc07c9e27e0)

[2021-11-10T00:09:21.412Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/ddl/ddl_worker.go:263 +0x2d9

[2021-11-10T00:09:21.413Z] created by github.com/pingcap/tidb/ddl.(*ddl).Start

[2021-11-10T00:09:21.413Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/ddl/ddl.go:351 +0x2d5

[2021-11-10T00:09:21.413Z] 

[2021-11-10T00:09:21.413Z] goroutine 31169 [select, 8 minutes]:

[2021-11-10T00:09:21.413Z] github.com/pingcap/tidb/domain.(*Domain).globalConfigSyncerKeeper(0xc07ee167e0)

[2021-11-10T00:09:21.413Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:510 +0xf4

[2021-11-10T00:09:21.413Z] created by github.com/pingcap/tidb/domain.(*Domain).Init

[2021-11-10T00:09:21.413Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:830 +0x7af

[2021-11-10T00:09:21.413Z] 

[2021-11-10T00:09:21.413Z] goroutine 32761 [select]:

[2021-11-10T00:09:21.413Z] github.com/pingcap/tidb/domain.(*Domain).TelemetryRotateSubWindowLoop.func1(0xc07ee167e0)

[2021-11-10T00:09:21.413Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:1169 +0xd8

[2021-11-10T00:09:21.413Z] created by github.com/pingcap/tidb/domain.(*Domain).TelemetryRotateSubWindowLoop

[2021-11-10T00:09:21.413Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:1162 +0x7f

[2021-11-10T00:09:21.413Z] 

[2021-11-10T00:09:21.413Z] goroutine 31165 [select]:

[2021-11-10T00:09:21.413Z] github.com/pingcap/tidb/ddl.(*worker).start(0xc07cfddb80, 0xc07c9c6300)

[2021-11-10T00:09:21.413Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/ddl/ddl_worker.go:178 +0x315

[2021-11-10T00:09:21.413Z] created by github.com/pingcap/tidb/ddl.(*ddl).Start

[2021-11-10T00:09:21.413Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/ddl/ddl.go:369 +0x6a5

[2021-11-10T00:09:21.413Z] 

[2021-11-10T00:09:21.413Z] goroutine 31202 [select]:

[2021-11-10T00:09:21.413Z] github.com/pingcap/tidb/domain.(*Domain).topologySyncerKeeper(0xc07ee167e0)

[2021-11-10T00:09:21.413Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:533 +0x193

[2021-11-10T00:09:21.413Z] created by github.com/pingcap/tidb/domain.(*Domain).Init

[2021-11-10T00:09:21.413Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:834 +0x815

[2021-11-10T00:09:21.413Z] 

[2021-11-10T00:09:21.413Z] goroutine 31168 [select]:

[2021-11-10T00:09:21.413Z] github.com/pingcap/tidb/domain.(*Domain).infoSyncerKeeper(0xc07ee167e0)

[2021-11-10T00:09:21.413Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:487 +0x138

[2021-11-10T00:09:21.413Z] created by github.com/pingcap/tidb/domain.(*Domain).Init

[2021-11-10T00:09:21.413Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:829 +0x78a

[2021-11-10T00:09:21.413Z] 

[2021-11-10T00:09:21.413Z] goroutine 31167 [select, 8 minutes]:

[2021-11-10T00:09:21.413Z] github.com/pingcap/tidb/domain.(*Domain).topNSlowQueryLoop(0xc07ee167e0)

[2021-11-10T00:09:21.413Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:455 +0x1e5

[2021-11-10T00:09:21.413Z] created by github.com/pingcap/tidb/domain.(*Domain).Init

[2021-11-10T00:09:21.413Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:828 +0x765

[2021-11-10T00:09:21.413Z] 
[2021-11-10T00:09:21.413Z] goroutine 32495 [select, 4 minutes]:
[2021-11-10T00:09:21.413Z] github.com/pingcap/tidb/domain.(*Domain).LoadPrivilegeLoop.func1(0xc07ee167e0, 0xc073b596b8, 0x45d964b800, 0x41ed530, 0xc07a9ca200)
[2021-11-10T00:09:21.413Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:939 +0x125
[2021-11-10T00:09:21.413Z] created by github.com/pingcap/tidb/domain.(*Domain).LoadPrivilegeLoop
[2021-11-10T00:09:21.413Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:930 +0x1f8
[2021-11-10T00:09:21.413Z] FAIL	github.com/pingcap/tidb/table/tables	540.138s

https://ci.pingcap.net/blue/organizations/jenkins/tidb_ghpr_check_2/detail/tidb_ghpr_check_2/44348/pipeline/
https://ci.pingcap.net/blue/organizations/jenkins/tidb_ghpr_check_2/detail/tidb_ghpr_check_2/44340/pipeline/

4. What is your TiDB version? (Required)

master, commit dbea989

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions