tidb occurred fatal error: concurrent map iteration and map write when memory grow quickly #38703
Description
Bug Report
Please answer these questions before submitting your issue. Thanks!
1. Minimal reproduce step (Required)
- tidb cluster in k8s, including tidb 8c 30g *6
- running ch workload:
/go-tpc ch run --host *** -P 4000 --warehouses 2000 -T 16 -D longrun_tpcc --acThreads 0 --ignore-errorand randomly execute query inq1,q2,q3,q4,q5,q6,q7,q8,q10,q11,q12,q13,q14,q15,q16,q17,q18,q19,q20,q21,q22,q23 - running go-ycsb workload:
/go-ycsb run mysql -P "./workloads/betting" --threads=500 -p "mysql.host=***" -p "mysql.port=4000" -p "operationcount=9223362036854775807" -p "dbnameprefix=betting_" -p "tablecount=2000" -p "unitnameprefix=unit2_" -p "databaseproportions=1.0" -p "unitscount=10" - these workloads make tidb memory usage grow quickly
2. What did you expect to see? (Required)
- tidb running with no error
3. What did you see instead (Required)
- tidb occurred fatal error
fatal error: concurrent map iteration and map write
goroutine 21310 [running]:
github.com/pingcap/tidb/util/execdetails.(*CopRuntimeStats).GetActRows(...)
/home/root/tidb/util/execdetails/execdetails.go:389
github.com/pingcap/tidb/planner/core.getRuntimeInfoStr({0x4d95480?, 0xc00234e000?}, {0x4d4d150?, 0xc153822780?}, 0xc153822780?)
/home/root/tidb/planner/core/common_plans.go:826 +0x28e
github.com/pingcap/tidb/planner/core.(*Explain).prepareOperatorInfo(0xc01601d960, {0x4d4d150, 0xc153822780}, {0xc36f902257, 0x9}, {0xc13104e000, 0x21})
/home/root/tidb/planner/core/common_plans.go:878 +0x3c9
github.com/pingcap/tidb/planner/core.(*Explain).explainFlatOpInRowFormat(0xc01601d960, 0xc2f476a5a0)
/home/root/tidb/planner/core/common_plans.go:798 +0x21d
github.com/pingcap/tidb/planner/core.(*Explain).explainFlatPlanInRowFormat(0x4d4cd90?, 0xc051b3c0f0)
/home/root/tidb/planner/core/common_plans.go:779 +0x5f
github.com/pingcap/tidb/planner/core.(*Explain).RenderResult(0xc01601d960)
/home/root/tidb/planner/core/common_plans.go:746 +0x548
github.com/pingcap/tidb/planner/core.GetExplainAnalyzeRowsForPlan(0xc01601d960)
/home/root/tidb/planner/core/common_plans.go:664 +0x2f
github.com/pingcap/tidb/session.(*session).getCurrentAnalyzePlan(0xc00234e000, {0x459be40?, 0xc1532bb040}, 0xc002336300)
/home/root/tidb/session/session.go:1619 +0xd7
github.com/pingcap/tidb/util/memoryusagealarm.getCurrentAnalyzePlan(0x5869ff6103a?)
/home/root/tidb/util/memoryusagealarm/memoryusagealarm.go:243 +0x53
github.com/pingcap/tidb/util/memoryusagealarm.(*memoryUsageAlarm).getTop10SqlInfo(0xc01601ff20, 0x0?, {0xc20d1b1920?, 0x3, 0x38?})
/home/root/tidb/util/memoryusagealarm/memoryusagealarm.go:285 +0x9e6
github.com/pingcap/tidb/util/memoryusagealarm.(*memoryUsageAlarm).getTop10SqlInfoByMemoryUsage(0x16a4f67?, {0xc20d1b1920?, 0x38?, 0xc01601dfd0?})
/home/root/tidb/util/memoryusagealarm/memoryusagealarm.go:305 +0x2e
github.com/pingcap/tidb/util/memoryusagealarm.(*memoryUsageAlarm).printTop10SqlInfo(0xc316900190?, {0xc20d1b1920, 0x3, 0xb}, 0x16a4365?)
/home/root/tidb/util/memoryusagealarm/memoryusagealarm.go:255 +0x227
github.com/pingcap/tidb/util/memoryusagealarm.(*memoryUsageAlarm).recordSQL(0x3f73ac0?, {0x4d498d0?, 0xc002afe7e0?}, {0xc30dfa3b00, 0x3c})
/home/root/tidb/util/memoryusagealarm/memoryusagealarm.go:336 +0x471
github.com/pingcap/tidb/util/memoryusagealarm.(*memoryUsageAlarm).doRecord(0xc01601ff20, 0x4c6925868?, 0x37?, {0x4d498d0, 0xc002afe7e0}, 0x1)
/home/root/tidb/util/memoryusagealarm/memoryusagealarm.go:222 +0xc78
github.com/pingcap/tidb/util/memoryusagealarm.(*memoryUsageAlarm).alarm4ExcessiveMemUsage(0xc01601ff20, {0x4d498d0, 0xc002afe7e0})
/home/root/tidb/util/memoryusagealarm/memoryusagealarm.go:163 +0x393
github.com/pingcap/tidb/util/memoryusagealarm.(*Handle).Run(0xc000e9a438)
/home/root/tidb/util/memoryusagealarm/memoryusagealarm.go:67 +0x113
created by main.createServer
/home/root/tidb/tidb-server/main.go:759 +0x4d0
....
4. What is your TiDB version? (Required)
Release Version: v6.4.0-alpha
Edition: Community
Git Commit Hash: 85b53e1
Git Branch: heads/refs/tags/v6.4.0-alpha
UTC Build Time: 2022-10-22 11:09:50
GoVersion: go1.19.1
Race Enabled: false
TiKV Min Version: 6.2.0-alpha
Check Table Before Drop: false
Store: tikv