Test: Mock window function, refactor window function tests by ywqzzy · Pull Request #5021 · pingcap/tiflash

ywqzzy · 2022-05-27T10:30:38Z

What problem does this PR solve?

Issue Number: ref #4609 close #5081

Problem Summary:
Mock window and sort executors.
Rewrite window function tests in new test framework.

What is changed and how it works?

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

None

…_window_function

ti-chi-bot · 2022-05-27T10:30:40Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

Lloyd-Pottiger
SeaRise

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Details

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

Lloyd-Pottiger · 2022-05-27T15:13:54Z

you can try gmock to do mock test which has embedded in gtest

dbms/src/Debug/astToExecutor.cpp

dbms/src/Flash/Coprocessor/collectOutputFieldTypes.cpp

dbms/src/Flash/Coprocessor/DAGQueryBlock.cpp

ywqzzy · 2022-05-30T08:31:24Z

you can try gmock to do mock test which has embedded in gtest

It's not my intention to this pr.
But great idea, I will feature out a way to mock data using gmock for future.

dbms/src/Debug/astToExecutor.cpp

SeaRise · 2022-06-10T05:56:51Z

maybe we can remove #ifndef DBMS_PUBLIC_GTEST in

tiflash/dbms/src/Flash/Coprocessor/DAGQueryBlockInterpreter.h

Lines 57 to 59 in 1e3207d

    
           #ifndef DBMS_PUBLIC_GTEST 
        
           private: 
        
           #endif

and

tiflash/dbms/src/Flash/Coprocessor/DAGExpressionAnalyzer.h

Lines 156 to 158 in 1e3207d

    
           #ifndef DBMS_PUBLIC_GTEST 
        
           private: 
        
           #endif

dbms/src/Debug/astToExecutor.cpp

dbms/src/Interpreters/WindowDescription.cpp

dbms/src/Interpreters/WindowDescription.h

dbms/src/TestUtils/ExecutorTestUtils.h

dbms/src/Flash/Coprocessor/DAGQueryBlockInterpreter.h

dbms/src/Flash/Coprocessor/DAGExpressionAnalyzer.h

dbms/src/Debug/astToExecutor.cpp

dbms/src/Flash/tests/gtest_interpreter.cpp

Lloyd-Pottiger · 2022-06-10T11:57:20Z

dbms/src/DataStreams/WindowBlockInputStream.cpp

+        window_description.window_functions_descriptions.begin(),
+        window_description.window_functions_descriptions.end(),
+        [&](const auto & func, FmtBuffer & b) {
+            b.fmtAppend("{}", func.window_function->getName());


Suggested change

b.fmtAppend("{}", func.window_function->getName());

b.append(func.window_function->getName());

dbms/src/Debug/astToExecutor.cpp

dbms/src/TestUtils/ExecutorTestUtils.h

dbms/src/TestUtils/FunctionTestUtils.cpp

Lloyd-Pottiger

LGTM

ywqzzy · 2022-06-13T07:47:07Z

/merge

ti-chi-bot · 2022-06-13T07:47:09Z

@ywqzzy: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2022-06-13T07:47:10Z

This pull request has been accepted and is ready to merge.

Details

Commit hash: 958722d

sre-bot · 2022-06-13T08:08:23Z

Coverage for changed files

Filename                                             Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
DataStreams/WindowBlockInputStream.cpp                   162                34    79.01%          20                 0   100.00%         397                96    75.82%         124                33    73.39%
DataStreams/WindowBlockInputStream.h                      26                 2    92.31%          15                 2    86.67%          58                15    74.14%          16                 2    87.50%
Debug/astToExecutor.cpp                                  576               215    62.67%          53                 9    83.02%        1516               553    63.52%         568               254    55.28%
Debug/astToExecutor.h                                     33                12    63.64%          22                 7    68.18%          50                18    64.00%          10                 6    40.00%
Flash/Coprocessor/DAGExpressionAnalyzer.h                  2                 1    50.00%           2                 1    50.00%           2                 1    50.00%           0                 0         -
Flash/Coprocessor/DAGQueryBlockInterpreter.h               2                 0   100.00%           2                 0   100.00%           2                 0   100.00%           0                 0         -
Flash/Coprocessor/DAGUtils.cpp                           347               155    55.33%          45                15    66.67%         638               294    53.92%         412               197    52.18%
Flash/Coprocessor/collectOutputFieldTypes.cpp             59                 8    86.44%          14                 0   100.00%         143                21    85.31%          56                11    80.36%
Flash/tests/gtest_interpreter.cpp                        110                57    48.18%           4                 0   100.00%         415                 0   100.00%          12                12     0.00%
Interpreters/WindowDescription.cpp                        23                11    52.17%           5                 0   100.00%          63                24    61.90%          28                11    60.71%
Interpreters/WindowDescription.h                           1                 1     0.00%           1                 1     0.00%          11                11     0.00%           0                 0         -
TestUtils/ExecutorTestUtils.cpp                           69                 8    88.41%          12                 1    91.67%         101                 5    95.05%          26                 6    76.92%
TestUtils/ExecutorTestUtils.h                              6                 2    66.67%           3                 0   100.00%          18                 6    66.67%           6                 3    50.00%
TestUtils/FunctionTestUtils.cpp                          138                23    83.33%          14                 0   100.00%         155                19    87.74%          62                21    66.13%
TestUtils/FunctionTestUtils.h                            116                 9    92.24%          36                 0   100.00%         273                 7    97.44%          48                 6    87.50%
TestUtils/executorSerializer.cpp                          71                 8    88.73%          29                 1    96.55%         360                31    91.39%          50                10    80.00%
TestUtils/mockExecutor.cpp                                79                15    81.01%          49                12    75.51%         307                63    79.48%          36                 3    91.67%
TestUtils/mockExecutor.h                                   6                 1    83.33%           6                 1    83.33%          15                 3    80.00%           0                 0         -
TestUtils/tests/gtest_mock_executors.cpp                 262               190    27.48%          11                 0   100.00%         215                 0   100.00%          40                40     0.00%
WindowFunctions/tests/gtest_window_functions.cpp          30                19    36.67%           2                 0   100.00%         143                 0   100.00%           4                 4     0.00%
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                   2118               771    63.60%         345                50    85.51%        4882              1167    76.10%        1498               619    58.68%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18322      9640             47.39%    205718  96567        53.06%

full coverage report (for internal network access only)

ywqzzy added 6 commits May 11, 2022 10:46

half way store.

c52af87

tmp save.

0773d7e

Merge branch 'master' of https://github.com/pingcap/tiflash into mock…

1e2a41f

…_window_function

mock frame.

6260500

support sort, run window function ut.

2e2c1af

update.

58be5d2

ywqzzy added 2 commits May 27, 2022 20:36

run interpreter.

88f2844

serailize window and sort.

23d8057

ywqzzy mentioned this pull request May 27, 2022

Interpreter & Executor & MPP unit test framework #4609

Closed

54 tasks

ti-chi-bot removed the do-not-merge/needs-linked-issue label May 28, 2022

SeaRise reviewed May 30, 2022

View reviewed changes

dbms/src/Debug/astToExecutor.cpp Outdated Show resolved Hide resolved

dbms/src/Flash/Coprocessor/collectOutputFieldTypes.cpp Outdated Show resolved Hide resolved

dbms/src/Flash/Coprocessor/DAGQueryBlock.cpp Outdated Show resolved Hide resolved

print window stream.

db8974c

ti-chi-bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels May 30, 2022

more tests with different types

fd10ccb

ywqzzy added 4 commits May 30, 2022 22:33

rewrite all tests.

e638f98

fix columnprune bug

b043a98

refine code

cd68a59

little refine.

721e471

ywqzzy changed the title ~~[WIP]Interpreter: Mock window function~~ Interpreter: Mock window function Jun 1, 2022

ti-chi-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 1, 2022

ywqzzy requested a review from SeaRise June 1, 2022 03:42

fix.

e60e35f

modify testname

a89cb84

SeaRise reviewed Jun 10, 2022

View reviewed changes

dbms/src/Debug/astToExecutor.cpp Outdated Show resolved Hide resolved

dbms/src/Debug/astToExecutor.cpp Show resolved Hide resolved

remove some useless data.

53a8a1f

SeaRise reviewed Jun 10, 2022

View reviewed changes

dbms/src/Debug/astToExecutor.cpp Show resolved Hide resolved

SeaRise reviewed Jun 10, 2022

View reviewed changes

dbms/src/Interpreters/WindowDescription.cpp Show resolved Hide resolved

dbms/src/Interpreters/WindowDescription.cpp Show resolved Hide resolved

dbms/src/Interpreters/WindowDescription.h Outdated Show resolved Hide resolved

dbms/src/TestUtils/ExecutorTestUtils.h Show resolved Hide resolved

address comments.

5f336d2

SeaRise reviewed Jun 10, 2022

View reviewed changes

dbms/src/Flash/Coprocessor/DAGQueryBlockInterpreter.h Show resolved Hide resolved

dbms/src/Flash/Coprocessor/DAGExpressionAnalyzer.h Show resolved Hide resolved

dbms/src/Debug/astToExecutor.cpp Outdated Show resolved Hide resolved

ywqzzy added 2 commits June 10, 2022 15:26

address comments.

3f0fbc1

format.

2589c3b

ywqzzy commented Jun 10, 2022

View reviewed changes

dbms/src/Flash/tests/gtest_interpreter.cpp Outdated Show resolved Hide resolved

ywqzzy added 2 commits June 10, 2022 15:54

address comments.

60b2d55

address comments

596b766

SeaRise approved these changes Jun 10, 2022

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Jun 10, 2022

Lloyd-Pottiger reviewed Jun 10, 2022

View reviewed changes

address comments.

11f11a7

ywqzzy requested a review from Lloyd-Pottiger June 13, 2022 04:34

Lloyd-Pottiger approved these changes Jun 13, 2022

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Jun 13, 2022

mengxin9014 approved these changes Jun 13, 2022

View reviewed changes

ywqzzy added 2 commits June 13, 2022 15:03

Merge branch 'master' into mock_window_function

9813118

Merge branch 'master' into mock_window_function

958722d

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Jun 13, 2022

ti-chi-bot merged commit f4c2e01 into pingcap:master Jun 13, 2022

	b.fmtAppend("{}", func.window_function->getName());
	b.append(func.window_function->getName());

Conversation

ywqzzy commented May 27, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

Uh oh!

ti-chi-bot commented May 27, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Lloyd-Pottiger commented May 27, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ywqzzy commented May 30, 2022

Uh oh!

Uh oh!

Uh oh!

SeaRise commented Jun 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Lloyd-Pottiger Jun 10, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Lloyd-Pottiger left a comment

Choose a reason for hiding this comment

Uh oh!

ywqzzy commented Jun 13, 2022

Uh oh!

ti-chi-bot commented Jun 13, 2022

Uh oh!

ti-chi-bot commented Jun 13, 2022

Uh oh!

sre-bot commented Jun 13, 2022

Coverage for changed files

Coverage summary

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

ywqzzy commented May 27, 2022 •

edited

Loading

ti-chi-bot commented May 27, 2022 •

edited

Loading

SeaRise commented Jun 10, 2022 •

edited

Loading