Skip to content

Test: Mock multiple compute service to run MPPTasks.#5573

Merged
ti-chi-bot merged 57 commits intopingcap:masterfrom
ywqdev:mock_multiple_service
Aug 23, 2022
Merged

Test: Mock multiple compute service to run MPPTasks.#5573
ti-chi-bot merged 57 commits intopingcap:masterfrom
ywqdev:mock_multiple_service

Conversation

@ywqzzy
Copy link
Contributor

@ywqzzy ywqzzy commented Aug 9, 2022

What problem does this PR solve?

Issue Number: ref #4609

Problem Summary:
We want to start multiple compute services to run real MPP tasks across grpc.

What is changed and how it works?

  1. Add MockStorage to hold user input columns to pass to each compute service.
  2. Add MockComputeServerManager to maintain compute server info and computer service instance in gtest.
  3. Add executeMPPQueryWithMultipleServer in dbgFuncCoprocessor.cpp to dispatch MPP tasks to different compute services.
  4. Extract common code for executeMPPQuery and executeMPPQueryWithMultipleServer.

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

None

@ti-chi-bot
Copy link
Member

ti-chi-bot commented Aug 9, 2022

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • Lloyd-Pottiger
  • SeaRise

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Details

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Aug 9, 2022
@ywqzzy ywqzzy changed the title Test: Mock multiple compute service to run MPPTasks. [DNM]Test: Mock multiple compute service to run MPPTasks. Aug 9, 2022
@ywqzzy ywqzzy changed the title [DNM]Test: Mock multiple compute service to run MPPTasks. [DNM] Test: Mock multiple compute service to run MPPTasks. Aug 9, 2022
@ywqzzy ywqzzy added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 9, 2022
@ywqzzy
Copy link
Contributor Author

ywqzzy commented Aug 22, 2022

/unhold

@ti-chi-bot ti-chi-bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 22, 2022
@ywqzzy ywqzzy requested review from SeaRise and Willendless and removed request for Willendless August 22, 2022 06:12
mpp_ctx->sender_target_task_ids = current_task_ids;
auto sub_fragments = mppQueryToQueryFragments(exchange.second.second, executor_index, properties, false, mpp_ctx);
receiver_source_task_ids_map[exchange.first] = sub_fragments.cbegin()->task_ids;
receiver_source_task_ids_map[exchange.first] = sub_fragments[sub_fragments.size() - 1].task_ids;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The exchangeSender will have multiple sub_fragments if it is the parent of another exchangeSender. The last fragment of the sub_fragment is the exchangeSender itself.

@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Aug 22, 2022
@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Aug 23, 2022
@ywqzzy
Copy link
Contributor Author

ywqzzy commented Aug 23, 2022

/merge

@ti-chi-bot
Copy link
Member

@ywqzzy: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

DetailsCommit hash: 68e429b

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Aug 23, 2022
@ywqzzy ywqzzy force-pushed the mock_multiple_service branch from 68e429b to 7296707 Compare August 23, 2022 04:22
@ti-chi-bot ti-chi-bot removed the status/can-merge Indicates a PR has been approved by a committer. label Aug 23, 2022
@ywqzzy
Copy link
Contributor Author

ywqzzy commented Aug 23, 2022

/run-all-tests

@sre-bot
Copy link
Collaborator

sre-bot commented Aug 23, 2022

Coverage for changed files

Filename                                                 Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Debug/MockComputeServerManager.cpp                            20                 1    95.00%           9                 0   100.00%          57                 0   100.00%          12                 1    91.67%
Debug/MockStorage.cpp                                         40                 6    85.00%          15                 0   100.00%         101                 6    94.06%          20                 6    70.00%
Debug/MockStorage.h                                            1                 0   100.00%           1                 0   100.00%           3                 0   100.00%           0                 0         -
Debug/astToExecutor.cpp                                      587               133    77.34%          54                 3    94.44%        1571               357    77.28%         590               178    69.83%
Debug/astToExecutor.h                                         30                 8    73.33%          22                 4    81.82%          48                12    75.00%           8                 5    37.50%
Debug/dbgFuncCoprocessor.cpp                                 431               323    25.06%          42                28    33.33%         893               606    32.14%         292               221    24.32%
Debug/dbgFuncCoprocessor.h                                     1                 0   100.00%           1                 0   100.00%           1                 0   100.00%           0                 0         -
Flash/Coprocessor/DAGQueryBlockInterpreter.cpp               266                94    64.66%          33                 5    84.85%         605               138    77.19%         142                41    71.13%
Flash/Coprocessor/MockSourceStream.h                          27                 3    88.89%           1                 0   100.00%          39                 0   100.00%          16                 2    87.50%
Flash/FlashService.cpp                                       399               306    23.31%          20                11    45.00%         327               237    27.52%         102                85    16.67%
Flash/FlashService.h                                           5                 3    40.00%           4                 2    50.00%          14                 7    50.00%           0                 0         -
Flash/Mpp/MPPTask.cpp                                        511               203    60.27%          27                 8    70.37%         457               171    62.58%         186               114    38.71%
Flash/Planner/plans/PhysicalMockExchangeReceiver.cpp          13                 0   100.00%           6                 0   100.00%          45                 4    91.11%           6                 0   100.00%
Flash/Planner/plans/PhysicalMockTableScan.cpp                 12                 0   100.00%           6                 0   100.00%          45                 4    91.11%           4                 0   100.00%
Flash/Planner/tests/gtest_physical_plan.cpp                  124                70    43.55%          15                 0   100.00%         515                 1    99.81%          42                40     4.76%
Flash/tests/gtest_compute_server.cpp                         104                21    79.81%           4                 0   100.00%         240                 0   100.00%          30                12    60.00%
Flash/tests/gtest_window_executor.cpp                        101                21    79.21%           9                 0   100.00%         315                 0   100.00%          30                12    60.00%
Interpreters/Context.cpp                                     534               305    42.88%         170                78    54.12%        1136               626    44.89%         284               205    27.82%
Interpreters/Context.h                                        13                 5    61.54%          13                 5    61.54%          13                 5    61.54%           0                 0         -
Server/FlashGrpcServerHolder.cpp                             220                71    67.73%           7                 0   100.00%         156                32    79.49%          60                28    53.33%
TestUtils/ExecutorTestUtils.cpp                               33                 2    93.94%          12                 1    91.67%          99                 5    94.95%          16                 0   100.00%
TestUtils/ExecutorTestUtils.h                                  3                 1    66.67%           3                 1    66.67%          18                13    27.78%           0                 0         -
TestUtils/MPPTaskTestUtils.h                                  15                 2    86.67%           9                 1    88.89%          46                 6    86.96%           4                 1    75.00%
TestUtils/mockExecutor.cpp                                    71                 3    95.77%          43                 1    97.67%         272                24    91.18%          32                 2    93.75%
TestUtils/mockExecutor.h                                      12                 4    66.67%          12                 4    66.67%          26                 6    76.92%           0                 0         -
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                       3573              1585    55.64%         538               152    71.75%        7042              2260    67.91%        1876               953    49.20%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18417      8335             54.74%    212754  86102        59.53%

full coverage report (for internal network access only)

@ywqzzy
Copy link
Contributor Author

ywqzzy commented Aug 23, 2022

/merge

@ti-chi-bot
Copy link
Member

@ywqzzy: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

DetailsCommit hash: 7296707

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Aug 23, 2022
@ywqzzy
Copy link
Contributor Author

ywqzzy commented Aug 23, 2022

/run-unit-tests

@sre-bot
Copy link
Collaborator

sre-bot commented Aug 23, 2022

Coverage for changed files

Filename                                                 Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Debug/MockComputeServerManager.cpp                            20                 1    95.00%           9                 0   100.00%          57                 0   100.00%          12                 1    91.67%
Debug/MockStorage.cpp                                         40                 6    85.00%          15                 0   100.00%         101                 6    94.06%          20                 6    70.00%
Debug/MockStorage.h                                            1                 0   100.00%           1                 0   100.00%           3                 0   100.00%           0                 0         -
Debug/astToExecutor.cpp                                      587               133    77.34%          54                 3    94.44%        1571               357    77.28%         590               178    69.83%
Debug/astToExecutor.h                                         30                 8    73.33%          22                 4    81.82%          48                12    75.00%           8                 5    37.50%
Debug/dbgFuncCoprocessor.cpp                                 431               323    25.06%          42                28    33.33%         893               606    32.14%         292               221    24.32%
Debug/dbgFuncCoprocessor.h                                     1                 0   100.00%           1                 0   100.00%           1                 0   100.00%           0                 0         -
Flash/Coprocessor/DAGQueryBlockInterpreter.cpp               266                94    64.66%          33                 5    84.85%         605               138    77.19%         142                41    71.13%
Flash/Coprocessor/MockSourceStream.h                          27                 3    88.89%           1                 0   100.00%          39                 0   100.00%          16                 2    87.50%
Flash/FlashService.cpp                                       399               306    23.31%          20                11    45.00%         327               237    27.52%         102                85    16.67%
Flash/FlashService.h                                           5                 3    40.00%           4                 2    50.00%          14                 7    50.00%           0                 0         -
Flash/Mpp/MPPTask.cpp                                        511               203    60.27%          27                 8    70.37%         457               171    62.58%         186               114    38.71%
Flash/Planner/plans/PhysicalMockExchangeReceiver.cpp          13                 0   100.00%           6                 0   100.00%          45                 4    91.11%           6                 0   100.00%
Flash/Planner/plans/PhysicalMockTableScan.cpp                 12                 0   100.00%           6                 0   100.00%          45                 4    91.11%           4                 0   100.00%
Flash/Planner/tests/gtest_physical_plan.cpp                  124                70    43.55%          15                 0   100.00%         515                 1    99.81%          42                40     4.76%
Flash/tests/gtest_compute_server.cpp                         104                21    79.81%           4                 0   100.00%         240                 0   100.00%          30                12    60.00%
Flash/tests/gtest_window_executor.cpp                        101                21    79.21%           9                 0   100.00%         315                 0   100.00%          30                12    60.00%
Interpreters/Context.cpp                                     534               305    42.88%         170                78    54.12%        1136               626    44.89%         284               205    27.82%
Interpreters/Context.h                                        13                 5    61.54%          13                 5    61.54%          13                 5    61.54%           0                 0         -
Server/FlashGrpcServerHolder.cpp                             220                71    67.73%           7                 0   100.00%         156                32    79.49%          60                28    53.33%
TestUtils/ExecutorTestUtils.cpp                               33                 2    93.94%          12                 1    91.67%          99                 5    94.95%          16                 0   100.00%
TestUtils/ExecutorTestUtils.h                                  3                 1    66.67%           3                 1    66.67%          18                13    27.78%           0                 0         -
TestUtils/MPPTaskTestUtils.h                                  15                 2    86.67%           9                 1    88.89%          46                 6    86.96%           4                 1    75.00%
TestUtils/mockExecutor.cpp                                    71                 3    95.77%          43                 1    97.67%         272                24    91.18%          32                 2    93.75%
TestUtils/mockExecutor.h                                      12                 4    66.67%          12                 4    66.67%          26                 6    76.92%           0                 0         -
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                       3573              1585    55.64%         538               152    71.75%        7042              2260    67.91%        1876               953    49.20%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18417      8333             54.75%    212754  86072        59.54%

full coverage report (for internal network access only)

@ti-chi-bot ti-chi-bot merged commit 112a51e into pingcap:master Aug 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants