planner: update cost model2 by qw4990 · Pull Request #39438 · pingcap/tidb

qw4990 · 2022-11-28T17:34:30Z

What problem does this PR solve?

Issue Number: ref #35240

Problem Summary: planner: update cost model2

What is changed and how it works?

planner: update cost model2

add some start-costs to HashJoin/IndexJoin to let the optimizer prefers to use MergeJoin if data size is small;
add some start-costs to HashAgg to let the optimizer prefers to use StreamAgg and not push HashAgg down if the data-size is small;

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

ti-chi-bot · 2022-11-28T17:34:32Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

chrysan
time-and-fate

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Details

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

qw4990 · 2022-11-29T02:05:32Z

/run-build-arm64 comment=true

sre-bot · 2022-11-29T02:16:00Z

download tidb binary(linux arm64) at http://fileserver.pingcap.net/download/builds/pingcap/test/tidb/3be764e21287d8b1314fdbe5d8508032ad92921e/centos7/tidb-linux-arm64.tar.gz

qw4990 · 2022-11-29T06:48:53Z

executor/explain_test.go

 		{
 			sql:      "select count(*) from t_unistore_act_rows group by b",
-			expected: []string{"2", "2", "2", "4"},
+			expected: []string{"2", "4", "4"},


Expected, not push Agg down.

qw4990 · 2022-11-29T06:49:12Z

executor/explainfor_test.go

-		`  └─TableReader_13(Probe) 0.00 root  data:Selection_12`,
-		`    └─Selection_12 0.00 cop[tikv]  eq(test.t1.a, 1)`,
-		`      └─TableRangeScan_11 0.80 cop[tikv] table:t1 range: decided by [eq(test.t1.a, test.t2.a)], keep order:false, stats:pseudo`))
+		`└─MergeJoin_10 1.00 root  inner join, left key:test.t2.a, right key:test.t1.a`,


Expected, MergeJoin is more cpu-effective if data-size is small.

qw4990 · 2022-11-29T06:49:38Z

executor/join_test.go

 	tk.MustQuery("select * from t l1 where exists ( select * from t l2 where l2.l_orderkey = l1.l_orderkey and l2.l_suppkey <> l1.l_suppkey )order by `l_orderkey`,`l_linenumber`;").Check(testkit.Rows("0 0 0 0", "0 1 0 1", "0 2 0 0", "1 0 1 0", "1 1 1 1", "1 2 1 0", "2 0 0 0", "2 1 0 1", "2 2 0 0"))
 	tk.MustQuery("desc format = 'brief' select count(*) from t l1 where exists ( select * from t l2 where l2.l_orderkey = l1.l_orderkey and l2.l_suppkey <> l1.l_suppkey );").Check(testkit.Rows(
-		"HashAgg 1.00 root  funcs:count(1)->Column#11",
+		"StreamAgg 1.00 root  funcs:count(1)->Column#11",


Expected, StreamAgg is more cpu-effective if data-size is small.

qw4990 · 2022-11-29T06:50:55Z

planner/core/testdata/binary_plan_suite_out.json

                        "operator_info": "data:TableFullScan_16"
                      }
                    ],
-                    "cost": 975351.9825195674,


Expected, just update cost values.

qw4990 · 2022-11-29T06:51:21Z

planner/core/testdata/expression_rewriter_suite_out.json

          "HashJoin 2.25 root  inner join, equal:[eq(test.t1.a, test.t2.a) eq(test.t1.b, test.t2.b)]",
          "├─HashAgg(Build) 1.69 root  group by:test.t2.a, test.t2.b, funcs:firstrow(test.t2.a)->test.t2.a, funcs:firstrow(test.t2.b)->test.t2.b",
-          "│ └─TableReader 1.69 root  data:HashAgg",
-          "│   └─HashAgg 1.69 cop[tikv]  group by:test.t2.a, test.t2.b, ",


Expected, not push Agg down if data-size is small.

planner/core/plan_cost_ver2.go

qw4990 · 2022-11-29T14:40:47Z

/merge

ti-chi-bot · 2022-11-29T14:40:52Z

This pull request has been accepted and is ready to merge.

Details

Commit hash: 83c2880

qw4990 · 2022-11-29T14:46:28Z

/merge

qw4990 · 2022-11-29T15:10:21Z

/merge

hawkingrei · 2022-11-29T15:31:33Z

/run-all-tests

qw4990 · 2022-11-29T15:31:36Z

/run-all-tests

qw4990 · 2022-11-29T15:31:44Z

/run-build-arm64 comment=true

sre-bot · 2022-11-29T15:42:10Z

download tidb binary(linux arm64) at http://fileserver.pingcap.net/download/builds/pingcap/test/tidb/e5a7dc8aebe6daef9c5c82875819789bb87357c4/centos7/tidb-linux-arm64.tar.gz

sre-bot · 2022-11-29T16:57:47Z

TiDB MergeCI notify

🔴 Bad News! New failing [1] after this pr merged.
These new failed integration tests seem to be caused by the current PR, please try to fix these new failed integration tests, thanks!

CI Name	Result	Duration	Compare with Parent commit
idc-jenkins-ci-tidb/integration-ddl-test	🟥 failed 2, success 4, total 6	5 min 21 sec	New failing
idc-jenkins-ci/integration-cdc-test	✅ all 40 tests passed	22 min	Fixed
idc-jenkins-ci-tidb/integration-common-test	🟢 all 17 tests passed	18 min	Existing passed
idc-jenkins-ci-tidb/common-test	🟢 all 11 tests passed	9 min 31 sec	Existing passed
idc-jenkins-ci-tidb/sqllogic-test-2	🟢 all 28 tests passed	6 min 54 sec	Existing passed
idc-jenkins-ci-tidb/mybatis-test	🟢 all 1 tests passed	5 min 51 sec	Existing passed
idc-jenkins-ci-tidb/tics-test	🟢 all 1 tests passed	5 min 41 sec	Existing passed
idc-jenkins-ci-tidb/sqllogic-test-1	🟢 all 26 tests passed	5 min 32 sec	Existing passed
idc-jenkins-ci-tidb/integration-compatibility-test	🟢 all 1 tests passed	3 min 13 sec	Existing passed
idc-jenkins-ci-tidb/plugin-test	🟢 build success, plugin test success	4min	Existing passed

ti-chi-bot added do-not-merge/invalid-title release-note-none Denotes a PR that doesn't merit a release note. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Nov 28, 2022

fixup

f958c5a

qw4990 force-pushed the tpcc-tune3 branch from 3be764e to f958c5a Compare November 29, 2022 02:54

qw4990 changed the title ~~wip~~ planner: update cost model2 Nov 29, 2022

ti-chi-bot removed the do-not-merge/invalid-title label Nov 29, 2022

qw4990 added sig/planner SIG: Planner epic/cost-model the optimizer cost model labels Nov 29, 2022

Merge branch 'master' into tpcc-tune3

f7e0c06

ti-chi-bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Nov 29, 2022

fixup

a36e38e

qw4990 force-pushed the tpcc-tune3 branch from e9f7fdf to a36e38e Compare November 29, 2022 05:17

qw4990 and others added 5 commits November 29, 2022 13:28

fixup

a7150a0

Merge branch 'master' into tpcc-tune3

5550b49

Merge branch 'master' into tpcc-tune3

1a5c4fe

fixup

3ed014f

Merge branch 'master' into tpcc-tune3

2c65189

qw4990 commented Nov 29, 2022

View reviewed changes

Merge branch 'master' into tpcc-tune3

bc1e3ec

qw4990 requested review from chrysan and time-and-fate November 29, 2022 07:58

chrysan approved these changes Nov 29, 2022

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Nov 29, 2022

time-and-fate approved these changes Nov 29, 2022

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Nov 29, 2022

time-and-fate reviewed Nov 29, 2022

View reviewed changes

planner/core/plan_cost_ver2.go Outdated Show resolved Hide resolved

qw4990 added 4 commits November 29, 2022 17:56

fiuxp

847a5d0

Merge branch 'master' into tpcc-tune3

40059a9

fixup

6dc1374

Merge branch 'master' into tpcc-tune3

83c2880

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Nov 29, 2022

Merge branch 'master' into tpcc-tune3

6fa4954

Merge branch 'master' into tpcc-tune3

e5a7dc8

Merge branch 'master' into tpcc-tune3

bf3006a

ti-chi-bot merged commit 4476173 into pingcap:master Nov 29, 2022

Conversation

qw4990 commented Nov 28, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

Uh oh!

ti-chi-bot commented Nov 28, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

qw4990 commented Nov 29, 2022

Uh oh!

sre-bot commented Nov 29, 2022

Uh oh!

qw4990 Nov 29, 2022

Choose a reason for hiding this comment

Uh oh!

qw4990 Nov 29, 2022

Choose a reason for hiding this comment

Uh oh!

qw4990 Nov 29, 2022

Choose a reason for hiding this comment

Uh oh!

qw4990 Nov 29, 2022

Choose a reason for hiding this comment

Uh oh!

qw4990 Nov 29, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

qw4990 commented Nov 29, 2022

Uh oh!

ti-chi-bot commented Nov 29, 2022

Uh oh!

qw4990 commented Nov 29, 2022

Uh oh!

qw4990 commented Nov 29, 2022

Uh oh!

hawkingrei commented Nov 29, 2022

Uh oh!

qw4990 commented Nov 29, 2022

Uh oh!

qw4990 commented Nov 29, 2022

Uh oh!

sre-bot commented Nov 29, 2022

Uh oh!

sre-bot commented Nov 29, 2022

TiDB MergeCI notify

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

qw4990 commented Nov 28, 2022 •

edited

Loading

ti-chi-bot commented Nov 28, 2022 •

edited

Loading