Feature Branch: merge master to `planner_refactory` branch by SeaRise · Pull Request #5353 · pingcap/tiflash

SeaRise · 2022-07-12T01:38:44Z

What problem does this PR solve?

Issue Number: ref #4739
close #5350

Problem Summary:

What is changed and how it works?

We can use #5350 to check if the modification is correct

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

None

close pingcap#4830

* Add to_seconds support for tiflash Signed-off-by: yibin <huyibin@pingcap.com> * Fix format issue Signed-off-by: yibin <huyibin@pingcap.com> * Add mutex lock to protect async reciever async reader Signed-off-by: yibin <huyibin@pingcap.com> * Fix a rebase issue Signed-off-by: yibin <huyibin@pingcap.com> * Change raw pointer to unique ptr Signed-off-by: yibin <huyibin@pingcap.com> * Fix format issue Signed-off-by: yibin <huyibin@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

close pingcap#5016

close pingcap#3157

close pingcap#4922

close pingcap#5023

close pingcap#5007

ref pingcap#4646

ref pingcap#4897

…gionKVStoreTest*'` (pingcap#4903) close pingcap#4904

close pingcap#4998

close pingcap#5028

ref pingcap#3594

ref pingcap#4830

close pingcap#4962

ref pingcap#4879

) * fix decoding error under heavy ddl operation * small fix

…ingcap#5053) ref pingcap#5052

close pingcap#5057

close pingcap#5054

close pingcap#5060

close pingcap#4606

…release build (pingcap#5047) ref pingcap#4909

ref pingcap#4609

) close pingcap#4474

close pingcap#5085

…ingcap#5094) ref pingcap#5076, close pingcap#5093

close pingcap#5037

close pingcap#4879

ref pingcap#5095

update for planner update for tests fix tests

ti-chi-bot · 2022-07-12T01:38:45Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

guo-shaoge
ywqzzy

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Details

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

SeaRise · 2022-07-12T01:42:36Z

/rebuild

SeaRise · 2022-07-12T01:45:08Z

dbms/src/Flash/Planner/Planner.cpp

+    /// todo support fine grained shuffle
+    static auto disable_fine_frained_shuffle = [](const DAGQueryBlock & query_block) {
+        return !enableFineGrainedShuffle(query_block.source->fine_grained_shuffle_stream_count())
+            && (!query_block.exchange_sender || !enableFineGrainedShuffle(query_block.exchange_sender->fine_grained_shuffle_stream_count()));
+    };
    return query_block.source
        && (query_block.source->tp() == tipb::ExecType::TypeProjection
-            || query_block.source->tp() == tipb::ExecType::TypeExchangeReceiver);
+            || query_block.source->tp() == tipb::ExecType::TypeExchangeReceiver)
+        && disable_fine_frained_shuffle(query_block);


for fine grained shuffle
#5048

SeaRise · 2022-07-12T01:46:09Z

dbms/src/Flash/Planner/plans/PhysicalExchangeReceiver.cpp

    for (size_t i = 0; i < max_streams; ++i)
    {
-        BlockInputStreamPtr stream = std::make_shared<ExchangeReceiverInputStream>(mpp_exchange_receiver, log->identifier(), executor_id);
+        BlockInputStreamPtr stream = std::make_shared<ExchangeReceiverInputStream>(mpp_exchange_receiver, log->identifier(), executor_id, /*stream_id=*/0);


for fine grained shuffle
#5048

SeaRise · 2022-07-12T01:46:21Z

dbms/src/Flash/Planner/plans/PhysicalExchangeSender.cpp

+    /// todo support fine grained shuffle
    int stream_id = 0;
    pipeline.transform([&](auto & stream) {
        // construct writer
-        std::unique_ptr<DAGResponseWriter> response_writer = std::make_unique<StreamingDAGResponseWriter<MPPTunnelSetPtr>>(
+        std::unique_ptr<DAGResponseWriter> response_writer = std::make_unique<StreamingDAGResponseWriter<MPPTunnelSetPtr, false>>(
            dag_context.tunnel_set,
            partition_col_ids,
            partition_col_collators,
            exchange_type,
            context.getSettingsRef().dag_records_per_chunk,
            context.getSettingsRef().batch_send_min_limit,
            stream_id++ == 0, /// only one stream needs to sending execution summaries for the last response
-            dag_context);
+            dag_context,
+            0,
+            0);


for fine grained shuffle
#5048

SeaRise · 2022-07-12T01:47:30Z

dbms/src/Flash/tests/gtest_planner_interpreter.cpp

+}
+CATCH
+
+/// todo support FineGrainedShuffle


for fine grained shuffle
#5048

SeaRise · 2022-07-12T01:48:40Z

dbms/src/Flash/Planner/plans/PhysicalAggregation.cpp

+    if (pipeline.streams.size() > 1 || pipeline.streams_with_non_joined_data.size() > 1)
    {
        const Settings & settings = context.getSettingsRef();
-        BlockInputStreamPtr stream_with_non_joined_data = combinedNonJoinedDataStream(pipeline, max_streams, log);
-        pipeline.firstStream() = std::make_shared<ParallelAggregatingBlockInputStream>(
+        BlockInputStreamPtr stream = std::make_shared<ParallelAggregatingBlockInputStream>(
            pipeline.streams,
-            stream_with_non_joined_data,
+            pipeline.streams_with_non_joined_data,
            params,
            context.getFileProvider(),
            true,
            max_streams,
            settings.aggregation_memory_efficient_merge_threads ? static_cast<size_t>(settings.aggregation_memory_efficient_merge_threads) : static_cast<size_t>(settings.max_threads),
            log->identifier());
+
        pipeline.streams.resize(1);
+        pipeline.streams_with_non_joined_data.clear();
+        pipeline.firstStream() = std::move(stream);
+
        // should record for agg before restore concurrency. See #3804.
        recordProfileStreams(pipeline, context);
        restoreConcurrency(pipeline, context.getDAGContext()->final_concurrency, log);
    }
    else
    {
-        BlockInputStreamPtr stream_with_non_joined_data = combinedNonJoinedDataStream(pipeline, max_streams, log);
        BlockInputStreams inputs;
        if (!pipeline.streams.empty())
            inputs.push_back(pipeline.firstStream());
-        else
-            pipeline.streams.resize(1);
-        if (stream_with_non_joined_data)
-            inputs.push_back(stream_with_non_joined_data);
+
+        if (!pipeline.streams_with_non_joined_data.empty())
+            inputs.push_back(pipeline.streams_with_non_joined_data.at(0));
+
+        pipeline.streams.resize(1);
+        pipeline.streams_with_non_joined_data.clear();
+


SeaRise · 2022-07-12T03:11:42Z

/rebuild

SeaRise · 2022-07-12T03:57:43Z

/merge

ti-chi-bot · 2022-07-12T03:57:44Z

@SeaRise: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2022-07-12T03:57:46Z

This pull request has been accepted and is ready to merge.

Details

Commit hash: b8a6178

SeaRise · 2022-07-12T03:59:02Z

/rebuild

SeaRise · 2022-07-12T03:59:50Z

/run-integration-test

sre-bot · 2022-07-12T04:38:31Z

Coverage for changed files

too many lines from llvm-cov, please refer to full report instead

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18664      9499             49.11%    211011  95872        54.57%

full coverage report (for internal network access only)

SeaRise · 2022-07-12T05:20:19Z

/run-integration-test

SeaRise · 2022-07-12T05:34:14Z

/run-integration-test

SeaRise · 2022-07-12T06:04:40Z

/run-integration-test

SeaRise · 2022-07-12T06:30:52Z

/run-integration-test

SeaRise · 2022-07-12T06:57:18Z

/run-integration-test

SeaRise · 2022-07-12T08:01:39Z

/run-integration-test

SeaRise · 2022-07-12T08:34:10Z

/run-integration-test

windtalker and others added 30 commits May 26, 2022 21:12

Refine function test framework (pingcap#4861)

973de13

close pingcap#4830

Optimize blobstore gc when current stat total size is 0 (pingcap#5015)

d84f273

close pingcap#5016

Fix an invalid default value cause bootstrap failed (pingcap#4916)

ca3e1c6

close pingcap#3157

PageStorage: Report the file usage of BlobStore (pingcap#4999)

d1ecebd

close pingcap#4922

Fix unstable ut in the master branch (pingcap#5024)

7dff463

close pingcap#5023

refine get type from decimal literal (pingcap#5014)

c8624a0

close pingcap#5007

Refactor: add new module schema (pingcap#4932)

516fa15

ref pingcap#4646

docs: Add rfc for compact table via sql (pingcap#4929)

335b124

ref pingcap#4897

Fix broken unit test for `--gtest_filter='*StorageDeltaMergeTest*:*Re…

580821a

…gionKVStoreTest*'` (pingcap#4903) close pingcap#4904

Remove useless code (pingcap#5004)

6afdd74

close pingcap#4998

Add mix mode UT (pingcap#5012)

a0ed1a6

close pingcap#5028

support mix_mode in dt_workload (pingcap#5011)

800715c

ref pingcap#3594

enable new function test framework by default (pingcap#5029)

380f6cd

ref pingcap#4830

Tiflash pagectl support encrypted data (pingcap#5003)

187a591

close pingcap#4962

update tiflash proxy (pingcap#5043)

226dff6

ref pingcap#4879

Fix potential data inconsistency under heavy ddl operation (pingcap#5044

2ce9529

) * fix decoding error under heavy ddl operation * small fix

*: fix invalid fmt format string in CreatingSetsBlockInputStream.cpp (p…

20d2198

…ingcap#5053) ref pingcap#5052

increase bg gc check interval (pingcap#5056)

677ad75

close pingcap#5057

PageStorage: Fix pages are not deleted under some cases (pingcap#5069)

b49a787

close pingcap#5054

Fix unstable drop table unit test (pingcap#5059)

a66c082

close pingcap#5060

Fix broken ut SegmentDeletionRelevantPlaceTest (pingcap#4607)

e3a4412

close pingcap#4606

Add Precompiled header for modules & Refine by PImpl to accelerate …

5847f1c

…release build (pingcap#5047) ref pingcap#4909

Test: Mock Input columns for operator tests (pingcap#5041)

fdab3f5

ref pingcap#4609

Improve the performance of partition table in extreme case (pingcap#4988

5b61ae7

) close pingcap#4474

Fix some fail cases when enable TASN (pingcap#5086)

167d39f

close pingcap#5085

PageStorage: Fix entry.tag after full gc && add more debug message (p…

ba725cc

…ingcap#5094) ref pingcap#5076, close pingcap#5093

feat: add page_stress_testing as a subcommand of tiflash (pingcap#5038)

a9b322a

close pingcap#5037

feat: add hardware information of server as an attribute (pingcap#5090)

1e3207d

close pingcap#4879

move tunnel_map to MPPTunnelSet (pingcap#5123)

8916afe

ref pingcap#5095

f

b8a6178

update for planner update for tests fix tests

ti-chi-bot added do-not-merge/needs-linked-issue release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jul 12, 2022

SeaRise changed the title ~~merge master to feature branch(planner_refactory)~~ Feature Branch: merge master to planner_refactory branch Jul 12, 2022

ti-chi-bot removed the do-not-merge/needs-linked-issue label Jul 12, 2022

SeaRise commented Jul 12, 2022

View reviewed changes

ywqzzy approved these changes Jul 12, 2022

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Jul 12, 2022

guo-shaoge approved these changes Jul 12, 2022

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Jul 12, 2022

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Jul 12, 2022

ti-chi-bot merged commit 05fd9bd into pingcap:planner_refactory Jul 12, 2022

Conversation

SeaRise commented Jul 12, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

Uh oh!

ti-chi-bot commented Jul 12, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SeaRise commented Jul 12, 2022

Uh oh!

SeaRise Jul 12, 2022

Choose a reason for hiding this comment

Uh oh!

SeaRise Jul 12, 2022

Choose a reason for hiding this comment

Uh oh!

SeaRise Jul 12, 2022

Choose a reason for hiding this comment

Uh oh!

SeaRise Jul 12, 2022

Choose a reason for hiding this comment

Uh oh!

SeaRise Jul 12, 2022

Choose a reason for hiding this comment

Uh oh!

SeaRise commented Jul 12, 2022

Uh oh!

SeaRise commented Jul 12, 2022

Uh oh!

ti-chi-bot commented Jul 12, 2022

Uh oh!

ti-chi-bot commented Jul 12, 2022

Uh oh!

SeaRise commented Jul 12, 2022

Uh oh!

SeaRise commented Jul 12, 2022

Uh oh!

sre-bot commented Jul 12, 2022

Coverage for changed files

Coverage summary

Uh oh!

SeaRise commented Jul 12, 2022

Uh oh!

SeaRise commented Jul 12, 2022

Uh oh!

SeaRise commented Jul 12, 2022

Uh oh!

SeaRise commented Jul 12, 2022

Uh oh!

SeaRise commented Jul 12, 2022

Uh oh!

SeaRise commented Jul 12, 2022

Uh oh!

SeaRise commented Jul 12, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

SeaRise commented Jul 12, 2022 •

edited

Loading

ti-chi-bot commented Jul 12, 2022 •

edited

Loading