planner: change the content of AnalyzeTableID to build global-stats by Reminiscent · Pull Request #22554 · pingcap/tidb

Reminiscent · 2021-01-27T03:40:46Z

What problem does this PR solve?

sub-PR for the PR#22472

Problem Summary:
We change the content for the AnalyzeTableID to build global-level stats. We use TableID to represent the ID of the table. And use the PartitionID to represent the ID of the partition. If the table is not the partition table, the PartitionID will be -1.

What is changed and how it works?

As the above described.

Related changes

PR to update pingcap/docs/pingcap/docs-cn:
Need to cherry-pick to the release branch

Check List

Tests

Unit test
Integration test

Side effects

We can not use the global-stats in 'dynamic-only' mode until the PR#22472 merged.

Release note

No release note

…obalStats-0

Reminiscent · 2021-01-27T05:55:42Z

executor/builder.go

-				if task.TableID.StoreAsCollectID() && len(task.TableID.CollectIDs) > 1 && !task.IndexInfo.Global && !task.IndexInfo.Unique {
-					b.buildAnalyzeFastIndex(e, task, v.Opts)
-				} else {
-					e.tasks = append(e.tasks, b.buildAnalyzeIndexPushdown(task, v.Opts, autoAnalyze))
-				}


TODO: need more investigate here

It seems OK since now we don't need fast analyze to collect partition statistics.

Reminiscent · 2021-01-27T05:56:55Z

/run-all-tests

rebelice

In the description, you say 'If the table is not the partition table, the PartitionID will be zero'. But in fact, it is -1 in the code.
Rest LGTM.

Reminiscent · 2021-01-27T10:16:44Z

In the description, you say 'If the table is not the partition table, the PartitionID will be zero'. But in fact, it is -1 in the code.
Rest LGTM.

Sry, it's a typo in description. I have fixed it now.

Reminiscent · 2021-01-27T10:24:01Z

/run-all-tests

Reminiscent · 2021-01-27T11:03:17Z

TODO: need more investigation on the change of the execution plan.

qw4990 · 2021-01-27T11:38:07Z

util/ranger/testdata/ranger_suite_out.json

-          "TableReader_5 4.00 root partition:all data:TableFullScan_4",
-          "└─TableFullScan_4 4.00 cop[tikv] table:t keep order:false"
+          "TableReader_5 10000.00 root partition:all data:TableFullScan_4",
+          "└─TableFullScan_4 10000.00 cop[tikv] table:t keep order:false, stats:pseudo"


Why use pseudo here?

…obalStats-0

Reminiscent · 2021-01-28T09:22:42Z

/run-all-tests

…obalStats-0

Reminiscent · 2021-01-29T03:04:37Z

/run-all-tests

util/ranger/testdata/ranger_suite_in.json

xuyifangreeneyes · 2021-01-29T11:14:43Z

planner/core/planbuilder.go

+						TblInfo:     tbl.TableInfo,
+					})
+				}
 			}


Assume table t has three partitions p0, p1, p2. When executing analyze table t in StaticButPrepareDynamic mode, here we generate 6 IdxTasks while 3 may be enough since we save partition table stats for dynamic pruning now.

…obalStats-0

Reminiscent · 2021-02-01T02:28:35Z

util/ranger/ranger_test.go

 	testKit := testkit.NewTestKit(c, store)
 	testKit.MustExec("use test;")
+	testKit.MustExec("set @@tidb_partition_prune_mode = 'static-only';")
+	testKit.MustExec("set @@tidb_executor_concurrency = 1;")


The reason why we set @@tidb_executor_concurrency = 1 is due to the concurrent execution of PartitionUnion, we need the results to stabilize.

Reminiscent · 2021-02-01T03:47:11Z

planner/core/planbuilder.go

-			if pruneMode == variable.StaticOnly || (pruneMode == variable.StaticButPrepareDynamic && !b.ctx.GetSessionVars().InRestrictedSQL) {
-				// static mode or static-but-prepare-dynamic mode not belong auto analyze need analyze each partition
-				// for static-but-prepare-dynamic mode with auto analyze, echo partition will be check before analyze partition.
-				for i, id := range physicalIDs {
-					info := analyzeInfo{DBName: tbl.Schema.O, TableName: tbl.Name.O, PartitionName: names[i], TableID: AnalyzeTableID{PersistID: id, CollectIDs: []int64{id}}, Incremental: as.Incremental}
-					p.IdxTasks = append(p.IdxTasks, AnalyzeIndexTask{
-						IndexInfo:   idx,
-						analyzeInfo: info,
-						TblInfo:     tbl.TableInfo,
-					})


We do not need to distinguish between different pruneMode here. Because no matter which mode it is, we need to build partition-level stats for each partitions. As for whether we need to build global-stats, we only need to check the pruneMode before building global-stats.

xuyifangreeneyes

LGTM

ti-srebot · 2021-02-01T09:11:57Z

@xuyifangreeneyes, Thanks for your review. The bot only counts LGTMs from Reviewers and higher roles, but you're still welcome to leave your comments. See the corresponding SIG page for more information. Related SIGs: planner(slack),execution(slack).

…obalStats-0

Reminiscent · 2021-02-02T03:46:25Z

/run-all-tests

Reminiscent · 2021-02-02T03:46:57Z

@qw4990 @rebelice @winoros PTAL. Thanks!

rebelice

LGTM

qw4990

LGTM

qw4990 · 2021-02-02T07:24:29Z

/merge

ti-srebot · 2021-02-02T07:26:43Z

/run-all-tests

planner: change the AnalyzeTableID when we build the analyze task

9ac2061

Reminiscent requested review from a team as code owners January 27, 2021 03:40

Reminiscent requested review from lzmhhh123 and removed request for a team January 27, 2021 03:40

github-actions bot added the sig/execution SIG execution label Jan 27, 2021

Reminiscent added 3 commits January 27, 2021 12:22

fix UT

faaea09

fix UT

b9cf0a4

Merge branch 'master' of https://github.com/pingcap/tidb into buildGl…

b45ee97

…obalStats-0

Reminiscent commented Jan 27, 2021

View reviewed changes

Reminiscent mentioned this pull request Jan 27, 2021

planner, statistics: build the global statistics for the partition table #22472

Closed

qw4990 self-requested a review January 27, 2021 06:37

rebelice reviewed Jan 27, 2021

View reviewed changes

fix UT

f94a9a8

qw4990 reviewed Jan 27, 2021

View reviewed changes

Reminiscent added 3 commits January 28, 2021 12:13

add the logical name 'statisticsID'

8cd694d

Merge branch 'master' of https://github.com/pingcap/tidb into buildGl…

5701cc8

…obalStats-0

refine the code

60bffc5

Reminiscent added 4 commits January 28, 2021 17:54

fix UT

14d07eb

fix UT

1fa2b25

update test results

975956f

Merge branch 'master' of https://github.com/pingcap/tidb into buildGl…

dee32ca

…obalStats-0

github-actions bot added the sig/planner SIG: Planner label Jan 29, 2021

update test results

0029317

Reminiscent commented Jan 29, 2021

View reviewed changes

util/ranger/testdata/ranger_suite_in.json Outdated Show resolved Hide resolved

xuyifangreeneyes reviewed Jan 29, 2021

View reviewed changes

Reminiscent added 2 commits February 1, 2021 10:25

update test results

6eb1ee4

Merge branch 'master' of https://github.com/pingcap/tidb into buildGl…

bd6beea

…obalStats-0

Reminiscent commented Feb 1, 2021

View reviewed changes

Unify the value of AnalyzeTableID in analyzeInfo in each pruneMode mode

2e73639

Reminiscent commented Feb 1, 2021

View reviewed changes

Reminiscent requested review from qw4990 and xuyifangreeneyes February 1, 2021 03:50

fix UT

26e0267

xuyifangreeneyes reviewed Feb 1, 2021

View reviewed changes

Reminiscent added 2 commits February 2, 2021 11:40

update the comments

214f3bb

Merge branch 'master' of https://github.com/pingcap/tidb into buildGl…

3f02417

…obalStats-0

rebelice approved these changes Feb 2, 2021

View reviewed changes

ti-srebot added the status/LGT1 Indicates that a PR has LGTM 1. label Feb 2, 2021

qw4990 approved these changes Feb 2, 2021

View reviewed changes

ti-srebot removed the status/LGT1 Indicates that a PR has LGTM 1. label Feb 2, 2021

ti-srebot approved these changes Feb 2, 2021

View reviewed changes

ti-srebot added the status/LGT2 Indicates that a PR has LGTM 2. label Feb 2, 2021

ti-srebot added the status/can-merge Indicates a PR has been approved by a committer. label Feb 2, 2021

Merge branch 'master' into buildGlobalStats-0

ba63b81

ti-srebot merged commit 44641c3 into pingcap:master Feb 2, 2021

This was referenced Feb 23, 2021

statistics: merge partition-level FMSketch to global-level FMSketch and update the NDV #22878

Merged

statistics: enables global-level stats to be generated in fast analyze mode #22931

Merged

Reminiscent deleted the buildGlobalStats-0 branch August 5, 2021 01:57

Conversation

Reminiscent commented Jan 27, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What is changed and how it works?

Related changes

Check List

Release note

Uh oh!

Reminiscent Jan 27, 2021

Choose a reason for hiding this comment

Uh oh!

qw4990 Jan 27, 2021

Choose a reason for hiding this comment

Uh oh!

Reminiscent commented Jan 27, 2021

Uh oh!

rebelice left a comment

Choose a reason for hiding this comment

Uh oh!

Reminiscent commented Jan 27, 2021

Uh oh!

Reminiscent commented Jan 27, 2021

Uh oh!

Reminiscent commented Jan 27, 2021

Uh oh!

qw4990 Jan 27, 2021

Choose a reason for hiding this comment

Uh oh!

Reminiscent commented Jan 28, 2021

Uh oh!

Reminiscent commented Jan 29, 2021

Uh oh!

Uh oh!

xuyifangreeneyes Jan 29, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reminiscent Feb 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reminiscent Feb 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xuyifangreeneyes left a comment

Choose a reason for hiding this comment

Uh oh!

ti-srebot commented Feb 1, 2021

Uh oh!

Reminiscent commented Feb 2, 2021

Uh oh!

Reminiscent commented Feb 2, 2021

Uh oh!

rebelice left a comment

Choose a reason for hiding this comment

Uh oh!

qw4990 left a comment

Choose a reason for hiding this comment

Uh oh!

qw4990 commented Feb 2, 2021

Uh oh!

ti-srebot commented Feb 2, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Reminiscent commented Jan 27, 2021 •

edited

Loading

xuyifangreeneyes Jan 29, 2021 •

edited

Loading

Reminiscent Feb 1, 2021 •

edited

Loading

Reminiscent Feb 1, 2021 •

edited

Loading