Skip to content

Support finalize for join#8366

Merged
ti-chi-bot[bot] merged 17 commits intopingcap:masterfrom
windtalker:support_finalize_for_join
Dec 28, 2023
Merged

Support finalize for join#8366
ti-chi-bot[bot] merged 17 commits intopingcap:masterfrom
windtalker:support_finalize_for_join

Conversation

@windtalker
Copy link
Contributor

@windtalker windtalker commented Nov 13, 2023

What problem does this PR solve?

Issue Number: close #8296

Problem Summary:
Support finalize for join

What is changed and how it works?

  • Support finalize for PhysicalJoin
  • Make FinalizeRule works for the whole planner tree(before this pr, FinalizeRule will stop at the First PhyscialJoin node)
  • Some refine in ProbeProcessInfo and Join.cpp

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
mysql> desc t;
+---------+---------+------+------+---------+-------+
| Field   | Type    | Null | Key  | Default | Extra |
+---------+---------+------+------+---------+-------+
| from_id | int(11) | YES  |      | NULL    |       |
| to_id   | int(11) | YES  |      | NULL    |       |
| value   | int(11) | YES  |      | NULL    |       |
+---------+---------+------+------+---------+-------+
mysql> select count(*) from t;
+----------+
| count(*) |
+----------+
|   393216 |
+----------+
mysql> select count(*), from_id, to_id, value from t group by from_id, to_id, value;
+----------+---------+-------+-------+
| count(*) | from_id | to_id | value |
+----------+---------+-------+-------+
|   131072 |       2 |     3 |     2 |
|   131072 |       3 |     4 |     3 |
|   131072 |       1 |     2 |     1 |
+----------+---------+-------+-------+

Before this pr

mysql>  select count(*) from t t1 join t t2 on t1.from_id = t2.to_id and t1.value > t2.value;
+-------------+
| count(*)    |
+-------------+
| 34359738368 |
+-------------+
1 row in set (1 min 5.94 sec)

After this pr

mysql>  select count(*) from t t1 join t t2 on t1.from_id = t2.to_id and t1.value > t2.value;
+-------------+
| count(*)    |
+-------------+
| 34359738368 |
+-------------+
1 row in set (52.86 sec)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

None

@ti-chi-bot ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Nov 13, 2023
@windtalker windtalker force-pushed the support_finalize_for_join branch from 0f310f2 to b0ffb43 Compare November 14, 2023 08:17
@windtalker
Copy link
Contributor Author

/run-all-tests

3 similar comments
@windtalker
Copy link
Contributor Author

/run-all-tests

@windtalker
Copy link
Contributor Author

/run-all-tests

@windtalker
Copy link
Contributor Author

/run-all-tests

@windtalker windtalker force-pushed the support_finalize_for_join branch from 2984d89 to cfd5ea5 Compare November 17, 2023 08:46
@windtalker
Copy link
Contributor Author

/run-all-tests

@windtalker windtalker force-pushed the support_finalize_for_join branch from cfd5ea5 to 030353b Compare November 20, 2023 01:39
@windtalker
Copy link
Contributor Author

/run-all-tests

@ti-chi-bot ti-chi-bot bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 22, 2023
@windtalker windtalker force-pushed the support_finalize_for_join branch from 030353b to 9c2a1fe Compare November 23, 2023 02:26
@ti-chi-bot ti-chi-bot bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 23, 2023
@windtalker
Copy link
Contributor Author

/run-all-tests

@windtalker windtalker force-pushed the support_finalize_for_join branch from 9c2a1fe to 3b9af83 Compare November 28, 2023 03:29
@windtalker
Copy link
Contributor Author

/run-all-tests

@windtalker windtalker force-pushed the support_finalize_for_join branch 2 times, most recently from 76262cf to efca8d5 Compare December 8, 2023 06:38
@windtalker
Copy link
Contributor Author

/run-all-tests

@SeaRise SeaRise self-requested a review December 8, 2023 08:44
@windtalker windtalker force-pushed the support_finalize_for_join branch from 085934e to b154d85 Compare December 12, 2023 07:51
@windtalker windtalker force-pushed the support_finalize_for_join branch 2 times, most recently from a4d7d5f to 16bb393 Compare December 15, 2023 03:23
@windtalker
Copy link
Contributor Author

/run-all-tests

@windtalker windtalker force-pushed the support_finalize_for_join branch 2 times, most recently from 1f99459 to aa3896d Compare December 19, 2023 02:29
@ti-chi-bot ti-chi-bot bot added the approved label Dec 22, 2023
@windtalker windtalker force-pushed the support_finalize_for_join branch from a86db27 to cd918f7 Compare December 26, 2023 08:24
windtalker and others added 16 commits December 27, 2023 17:55
Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>
Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>
Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>
Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>
Signed-off-by: xufei <xufei@pingcap.com>
Signed-off-by: xufei <xufei@pingcap.com>
Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>
Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>
Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>
Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>
Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>
Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>
Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>
Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>
Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>
Co-authored-by: Liqi Geng <gengliqiii@gmail.com>
@windtalker windtalker force-pushed the support_finalize_for_join branch from 95fee18 to e8087e9 Compare December 27, 2023 09:56
@ti-chi-bot ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Dec 28, 2023
@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Dec 28, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gengliqi, SeaRise

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Dec 28, 2023

[LGTM Timeline notifier]

Timeline:

  • 2023-12-22 08:21:19.68050828 +0000 UTC m=+1208370.717735203: ☑️ agreed by SeaRise.
  • 2023-12-28 09:51:00.600892092 +0000 UTC m=+1732151.638119019: ☑️ agreed by gengliqi.

@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Dec 28, 2023

@windtalker: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

trigger some heavy tests which will not run always when PR updated.

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@windtalker
Copy link
Contributor Author

/run-integration-tests

@ti-chi-bot ti-chi-bot bot merged commit 58c5531 into pingcap:master Dec 28, 2023
@windtalker windtalker deleted the support_finalize_for_join branch March 1, 2024 01:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved lgtm release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Join only construct joined columns that is needed by its parent operator

3 participants