stale read QPS is affected by DDL load #46524
Description
Bug Report
Please answer these questions before submitting your issue. Thanks!
1. Minimal reproduce step (Required)
-
Deploy a TiDB cluster.
-
run stale read load.
sysbench --config-file=sysbench.conf mussel_minimal.lua --threads=20 --read_staleness=-5 run
- run DDL load by loadgen. Following load will use 1 thread to continuously create tables, from t1 to t100000.
loadgen bench --sql 'create table t#seq-val (id int key, b int)' --thread=1 --valmin=1 --valmax=100000 --port 23400 --host=192.168.78.2 --db=test2
2. What did you expect to see? (Required)
The stale read load QPS shouldn't be affected by irrelevant DDL load.
3. What did you see instead (Required)
The stale read load QPS is drop from 17200 to 250.
4. What is your TiDB version? (Required)
v6.5.4
master: 187573b
5. Analyze
I add some metrics in TiDB to debug: crazycs520@06bf109
The stale-read request is slow on the following execution stack
github.com/pingcap/tidb/domain.(*Domain).loadInfoSchema+0xdef /data/cs/tidb/domain/domain.go:227
github.com/pingcap/tidb/domain.(*Domain).GetSnapshotInfoSchema+0x44 /data/cs/tidb/domain/domain.go:390
github.com/pingcap/tidb/sessiontxn/staleread.GetSessionSnapshotInfoSchema+0x53 /data/cs/tidb/sessiontxn/staleread/processor.go:320
github.com/pingcap/tidb/sessiontxn/staleread.(*staleReadProcessor).evaluateFromStmtTSOrSysVariable+0xdd /data/cs/tidb/sessiontxn/staleread/processor.go:245
github.com/pingcap/tidb/sessiontxn/staleread.(*staleReadProcessor).OnSelectTable+0x17a /data/cs/tidb/sessiontxn/staleread/processor.go:185
github.com/pingcap/tidb/planner/core.(*preprocessor).handleTableName+0xb6 /data/cs/tidb/planner/core/preprocess.go:1559
github.com/pingcap/tidb/planner/core.(*preprocessor).Leave+0x198 /data/cs/tidb/planner/core/preprocess.go:577
github.com/pingcap/tidb/parser/ast.(*TableName).Accept+0x164 /data/cs/tidb/parser/ast/dml.go:443
github.com/pingcap/tidb/parser/ast.(*TableSource).Accept+0x66 /data/cs/tidb/parser/ast/dml.go:599
github.com/pingcap/tidb/parser/ast.(*Join).Accept+0x66 /data/cs/tidb/parser/ast/dml.go:241
github.com/pingcap/tidb/parser/ast.(*TableRefsClause).Accept+0x64 /data/cs/tidb/parser/ast/dml.go:800
github.com/pingcap/tidb/parser/ast.(*SelectStmt).Accept+0x1fb /data/cs/tidb/parser/ast/dml.go:1460
github.com/pingcap/tidb/parser/ast.(*TableSource).Accept+0x66 /data/cs/tidb/parser/ast/dml.go:599
github.com/pingcap/tidb/parser/ast.(*Join).Accept+0x66 /data/cs/tidb/parser/ast/dml.go:241
github.com/pingcap/tidb/parser/ast.(*TableRefsClause).Accept+0x64 /data/cs/tidb/parser/ast/dml.go:800
github.com/pingcap/tidb/parser/ast.(*SelectStmt).Accept+0x1fb /data/cs/tidb/parser/ast/dml.go:1460
github.com/pingcap/tidb/planner/core.Preprocess+0x205 /data/cs/tidb/planner/core/preprocess.go:136
github.com/pingcap/tidb/executor.(*Compiler).Compile+0x3ed /data/cs/tidb/executor/compiler.go:81
github.com/pingcap/tidb/session.(*session).ExecuteStmt+0x5d3 /data/cs/tidb/session/session.go:2214
github.com/pingcap/tidb/server.(*TiDBContext).ExecuteStmt+0xa6 /data/cs/tidb/server/driver_tidb.go:294
github.com/pingcap/tidb/server.(*clientConn).handleStmt+0x152 /data/cs/tidb/server/conn.go:2122
github.com/pingcap/tidb/server.(*clientConn).handleQuery+0x8f4 /data/cs/tidb/server/conn.go:1972
github.com/pingcap/tidb/server.(*clientConn).dispatch+0xf84 /data/cs/tidb/server/conn.go:1403
github.com/pingcap/tidb/server.(*clientConn).Run+0x232 /data/cs/tidb/server/conn.go:1152
github.com/pingcap/tidb/server.(*Server).onConn+0xa86 /data/cs/tidb/server/server.go:648
As you can see, the stale read query costs too much time to get snapshot infoschema. This is the main cause of QPS degradation.