Skip to content

DATA RACE found in transaction asynchronous clean up and union store data reset #24560

New issue
New issue
Closed
#24505
Closed
DATA RACE found in transaction asynchronous clean up and union store data reset#24560
#24505
Labels
severity/criticalsig/transactionSIG:Transactiontype/bugThe issue is confirmed as a bug.
@tiancaiamao

Description

@tiancaiamao
tiancaiamao
opened on May 12, 2021

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

It's observed in the CI race test.

[2021-05-11T10:42:06.180Z] ==================
[2021-05-11T10:42:06.180Z] WARNING: DATA RACE
[2021-05-11T10:42:06.180Z] Write at 0x00c030589050 by goroutine 65:
[2021-05-11T10:42:06.180Z]   github.com/pingcap/tidb/store/tikv/unionstore.(*memdbVlog).revertToCheckpoint()
[2021-05-11T10:42:06.180Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/store/tikv/unionstore/memdb_arena.go:311 +0x260
[2021-05-11T10:42:06.180Z]   github.com/pingcap/tidb/store/tikv/unionstore.(*MemDB).Cleanup()
[2021-05-11T10:42:06.180Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/store/tikv/unionstore/memdb.go:128 +0x2f5
[2021-05-11T10:42:06.180Z]   github.com/pingcap/tidb/store/driver/txn.(*memBuffer).Cleanup()
[2021-05-11T10:42:06.180Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/store/driver/txn/unionstore_driver.go:60 +0x55
[2021-05-11T10:42:06.180Z]   github.com/pingcap/tidb/session.(*TxnState).cleanupStmtBuf()
[2021-05-11T10:42:06.180Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/session/txn.go:105 +0xef
[2021-05-11T10:42:06.180Z]   github.com/pingcap/tidb/session.(*TxnState).cleanup()
[2021-05-11T10:42:06.180Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/session/txn.go:273 +0x50
[2021-05-11T10:42:06.180Z]   github.com/pingcap/tidb/session.(*TxnState).reset()
[2021-05-11T10:42:06.180Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/session/txn.go:268 +0x38
[2021-05-11T10:42:06.180Z]   github.com/pingcap/tidb/session.(*TxnState).Commit()
[2021-05-11T10:42:06.180Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/session/txn.go:258 +0x861
[2021-05-11T10:42:06.180Z]   github.com/pingcap/tidb/session.(*session).doCommit()
[2021-05-11T10:42:06.180Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/session/session.go:534 +0xb03
[2021-05-11T10:42:06.180Z]   github.com/pingcap/tidb/session.(*session).doCommitWithRetry()
[2021-05-11T10:42:06.180Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/session/session.go:565 +0x1ae
[2021-05-11T10:42:06.180Z]   github.com/pingcap/tidb/session.(*session).CommitTxn()
[2021-05-11T10:42:06.180Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/session/session.go:627 +0x163
[2021-05-11T10:42:06.180Z]   github.com/pingcap/tidb/session.autoCommitAfterStmt()
[2021-05-11T10:42:06.180Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/session/tidb.go:254 +0x4cb
[2021-05-11T10:42:06.180Z]   github.com/pingcap/tidb/session.finishStmt()
[2021-05-11T10:42:06.180Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/session/tidb.go:220 +0xf6
[2021-05-11T10:42:06.180Z]   github.com/pingcap/tidb/session.runStmt()
[2021-05-11T10:42:06.180Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/session/session.go:1599 +0x58c
[2021-05-11T10:42:06.181Z]   github.com/pingcap/tidb/session.(*session).ExecuteStmt()
[2021-05-11T10:42:06.181Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/session/session.go:1484 +0xb27
[2021-05-11T10:42:06.181Z]   github.com/pingcap/tidb/util/testkit.(*TestKit).Exec()
[2021-05-11T10:42:06.181Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/util/testkit/testkit.go:170 +0x2f1
[2021-05-11T10:42:06.181Z]   github.com/pingcap/tidb/util/testkit.(*TestKit).MustExec()
[2021-05-11T10:42:06.181Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/util/testkit/testkit.go:216 +0x91
[2021-05-11T10:42:06.181Z]   github.com/pingcap/tidb/session_test.(*testSessionSuite).TestRetryGlobalTempTable()
[2021-05-11T10:42:06.181Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/session/session_test.go:815 +0x648
[2021-05-11T10:42:06.181Z]   runtime.call32()
[2021-05-11T10:42:06.181Z]       /usr/local/go/src/runtime/asm_amd64.s:539 +0x3a
[2021-05-11T10:42:06.181Z]   reflect.Value.Call()
[2021-05-11T10:42:06.181Z]       /usr/local/go/src/reflect/value.go:321 +0xd3
[2021-05-11T10:42:06.181Z]   github.com/pingcap/check.(*suiteRunner).forkTest.func1()
[2021-05-11T10:42:06.181Z]       /nfs/cache/gopath/pkg/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:850 +0x9aa
[2021-05-11T10:42:06.181Z]   github.com/pingcap/check.(*suiteRunner).forkCall.func1()
[2021-05-11T10:42:06.181Z]       /nfs/cache/gopath/pkg/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:739 +0x113
[2021-05-11T10:42:06.181Z] 
[2021-05-11T10:42:06.181Z] Previous read at 0x00c030589050 by goroutine 461:
[2021-05-11T10:42:06.181Z]   github.com/pingcap/tidb/store/tikv/unionstore.(*MemDB).GetValueByHandle()
[2021-05-11T10:42:06.181Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/store/tikv/unionstore/memdb.go:254 +0xb8
[2021-05-11T10:42:06.181Z]   github.com/pingcap/tidb/store/tikv.(*memBufferMutations).GetValue()
[2021-05-11T10:42:06.181Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/store/tikv/2pc.go:140 +0xc5
[2021-05-11T10:42:06.181Z]   github.com/pingcap/tidb/store/tikv.(*batched).appendBatchMutationsBySize()
[2021-05-11T10:42:06.181Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/store/tikv/2pc.go:1490 +0x1c1
[2021-05-11T10:42:06.181Z]   github.com/pingcap/tidb/store/tikv.(*twoPhaseCommitter).doActionOnGroupMutations()
[2021-05-11T10:42:06.181Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/store/tikv/2pc.go:560 +0x509
[2021-05-11T10:42:06.181Z]   github.com/pingcap/tidb/store/tikv.(*twoPhaseCommitter).doActionOnMutations()
[2021-05-11T10:42:06.181Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/store/tikv/2pc.go:499 +0x11d
[2021-05-11T10:42:06.181Z]   github.com/pingcap/tidb/store/tikv.(*twoPhaseCommitter).cleanup.func1()
[2021-05-11T10:42:06.181Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/store/tikv/cleanup.go:70 +0x12a7

2. What did you expect to see? (Required)

No DATA RACE

3. What did you see instead (Required)

DATA RACE detected

When 2PC commit fails, the cleanup processing is in another goroutine, it needs to read the union store to get the KVs.
Since the 2PC has finished and returned, the caller in the session package would reset the union store at the same time.
The asynchronous clean up and the reset operation conflicts and lead to the DATA RACE.

4. What is your TiDB version? (Required)

master branch

commit f13f696 (HEAD -> master, origin/master, origin/HEAD)
Author: ZhuoZhi 517770911@qq.com
Date: Tue May 11 10:57:38 2021 +0800

executor: add correctness tests about PointGet and BatchGet (#24467)

Metadata

Metadata

Assignees

No one assigned

    Labels

    severity/criticalsig/transactionSIG:Transactiontype/bugThe issue is confirmed as a bug.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions