Refine function test framework by windtalker · Pull Request #4861 · pingcap/tiflash

windtalker · 2022-05-11T06:53:18Z

What problem does this PR solve?

Issue Number: close #4830

Problem Summary:

We have a function test framework used by unit test of functions, however, the current implementation of the test framework use the following code to test the function:

    auto & factory = FunctionFactory::instance();

    Block block(columns);
    ColumnNumbers cns;
    for (size_t i = 0; i < columns.size(); ++i)
        cns.push_back(i);

    auto bp = factory.tryGet(func_name, context);
    if (!bp)
        throw TiFlashTestException(fmt::format("Function {} not found!", func_name));
    auto func = bp->build(columns, collator);
    block.insert({nullptr, func->getReturnType(), "res"});
    func->execute(block, cns, columns.size());
    return block.getByPosition(columns.size());

This is not the same in the real world when we executing functions, in the real world, we first build the function:

 const String & func_name = getFunctionName(expr);
 if (DAGExpressionAnalyzerHelper::function_builder_map.count(func_name) != 0)
 {
     ret = DAGExpressionAnalyzerHelper::function_builder_map[func_name](this, expr, actions);
 }
 else
 {
     ret = buildFunction(expr, actions);
 }

Then execute the function later. When building the function, we may rewrite functions, take ifNull(col1, col2) for example, we rewrite it to
multiIf(isNull(col1), col2, assumeNotNull(col1))

Current function test framework does not take this rewrite into account, it will either testing ifNull(col1, col2) by executing ifNull(col1, col2) directly, or rewrite it manually. Both solutions can't guarantee the testing logic is the same as the real world executing logic, which makes function unit test meaningless for functions that will be rewritten during buildFunction.(#4829 is an example that we already add ut for ifNull, but not catch the real world bug)

What is changed and how it works?

add a new argument raw_function_test in FunctionTest::executeFunction, and if raw_function_test == true, will use the original way to test function, otherwise, will convert columns into TiPB expr first, then test the function after DAGExpressionAnalyzerHelper::buildFunction, raw_function_test is true by default

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

None

ti-chi-bot · 2022-05-11T06:53:19Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

SeaRise
ywqzzy

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Details

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

dbms/src/Flash/Coprocessor/FunctionBuildHelper.cpp

dbms/src/TestUtils/FunctionTestUtils.cpp

dbms/src/TestUtils/FunctionTestUtils.h

dbms/src/Flash/Coprocessor/FunctionBuildHelper.cpp

dbms/src/TestUtils/FunctionTestUtils.cpp

dbms/src/Flash/Coprocessor/FunctionBuildHelper.cpp

dbms/src/Flash/Coprocessor/DAGUtils.cpp

dbms/src/TestUtils/FunctionTestUtils.cpp

windtalker · 2022-05-14T04:47:35Z

/run-all-tests

windtalker · 2022-05-14T09:05:18Z

/run-all-tests

sre-bot · 2022-05-14T09:34:41Z

Coverage for changed files

too many lines from llvm-cov, please refer to full report instead

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18108      9813             45.81%    201878  98120        51.40%

full coverage report (for internal network access only)

windtalker · 2022-05-14T10:26:03Z

/run-all-tests

sre-bot · 2022-05-14T10:53:35Z

Coverage for changed files

too many lines from llvm-cov, please refer to full report instead

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18109      9814             45.81%    201802  98129        51.37%

full coverage report (for internal network access only)

SeaRise · 2022-05-16T02:43:03Z

There are many changes to the gtest case.
Can you give some comments

dbms/src/Core/Field.h

dbms/src/DataTypes/FieldToDataType.h

dbms/src/TestUtils/ColumnsToTiPBExpr.cpp

windtalker · 2022-05-16T05:59:28Z

There are many changes to the gtest case. Can you give some comments

Ok, add the main changes of unit test in issue description.

dbms/src/DataTypes/FieldToDataType.h

dbms/src/Functions/tests/gtest_bitand.cpp

dbms/src/Functions/tests/gtest_datetime_daymonthyear.cpp

dbms/src/Functions/tests/gtest_ifnull.cpp

dbms/src/Functions/tests/gtest_regexp.cpp

SeaRise · 2022-05-16T08:52:10Z

how about making raw_function_test be true by default and just fix gtest_is_null in this pr?
Fix other gtest in other pr.
and make raw_function_test be false after all gtest are fixed.

Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>

…r/tiflash into refine_function_test_framework

dbms/src/Core/Field.h

dbms/src/DataTypes/FieldToDataType.h

dbms/src/TestUtils/FunctionTestUtils.cpp

dbms/src/TestUtils/ColumnsToTiPBExpr.cpp

Co-authored-by: SeaRise <hhssearise@foxmail.com>

Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>

…r/tiflash into refine_function_test_framework

SeaRise

other LGTM

dbms/src/TestUtils/ColumnsToTiPBExpr.cpp

ywqzzy

LGTM

Co-authored-by: SeaRise <hhssearise@foxmail.com>

windtalker · 2022-05-26T12:24:25Z

/merge

ti-chi-bot · 2022-05-26T12:24:26Z

@windtalker: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2022-05-26T12:24:28Z

This pull request has been accepted and is ready to merge.

Details

Commit hash: 1b0239d

Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>

…r/tiflash into refine_function_test_framework

windtalker · 2022-05-26T12:42:07Z

/merge

ti-chi-bot · 2022-05-26T12:42:07Z

@windtalker: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2022-05-26T12:42:09Z

This pull request has been accepted and is ready to merge.

Details

Commit hash: 5e81690

sre-bot · 2022-05-26T13:04:16Z

Coverage for changed files

Filename                                              Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Debug/astToExecutor.cpp                                   514               213    58.56%          49                 9    81.63%        1347               580    56.94%         512               246    51.95%
Debug/astToExecutor.h                                      29                10    65.52%          18                 5    72.22%          44                16    63.64%          10                 6    40.00%
Flash/Coprocessor/DAGExpressionAnalyzer.cpp               383               177    53.79%          50                10    80.00%         903               392    56.59%         298               166    44.30%
Flash/Coprocessor/DAGExpressionAnalyzer.h                   2                 2     0.00%           2                 2     0.00%           2                 2     0.00%           0                 0         -
Flash/Coprocessor/DAGExpressionAnalyzerHelper.cpp          92                87     5.43%          14                12    14.29%         244               226     7.38%          68                65     4.41%
Flash/Coprocessor/DAGUtils.cpp                            344               204    40.70%          45                18    60.00%         634               386    39.12%         410               239    41.71%
Flash/Coprocessor/DAGUtils.h                                3                 0   100.00%           1                 0   100.00%          12                 0   100.00%           2                 0   100.00%
TestUtils/ColumnsToTiPBExpr.cpp                            55                55     0.00%           6                 6     0.00%         122               122     0.00%          40                40     0.00%
TestUtils/FunctionTestUtils.cpp                           131                28    78.63%          12                 2    83.33%         126                44    65.08%          56                25    55.36%
TestUtils/FunctionTestUtils.h                             113                 9    92.04%          33                 0   100.00%         264                 7    97.35%          48                 6    87.50%
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                    1666               785    52.88%         230                64    72.17%        3698              1775    52.00%        1444               793    45.08%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18277      9786             46.46%    204780  98132        52.08%

full coverage report (for internal network access only)

ti-chi-bot added the release-note-none Denotes a PR that doesn't merit a release note. label May 11, 2022

windtalker requested a review from SeaRise May 11, 2022 06:53

ti-chi-bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label May 11, 2022

ywqzzy self-requested a review May 11, 2022 06:56

ywqzzy reviewed May 11, 2022

View reviewed changes

dbms/src/Flash/Coprocessor/FunctionBuildHelper.cpp Outdated Show resolved Hide resolved

dbms/src/Flash/Coprocessor/FunctionBuildHelper.cpp Outdated Show resolved Hide resolved

dbms/src/TestUtils/FunctionTestUtils.cpp Outdated Show resolved Hide resolved

ywqzzy self-requested a review May 11, 2022 07:24

SeaRise reviewed May 11, 2022

View reviewed changes

SeaRise requested review from SeaRise May 11, 2022 11:14

windtalker force-pushed the refine_function_test_framework branch 2 times, most recently from 0c6e6a4 to 5c7eeb6 Compare May 13, 2022 09:08

ti-chi-bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels May 13, 2022

SeaRise reviewed May 13, 2022

View reviewed changes

ti-chi-bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels May 13, 2022

windtalker force-pushed the refine_function_test_framework branch from 3f9f33a to c66539b Compare May 16, 2022 00:12

SeaRise reviewed May 16, 2022

View reviewed changes

dbms/src/Core/Field.h Outdated Show resolved Hide resolved

dbms/src/DataTypes/FieldToDataType.h Outdated Show resolved Hide resolved

dbms/src/TestUtils/ColumnsToTiPBExpr.cpp Outdated Show resolved Hide resolved

SeaRise self-requested a review May 16, 2022 08:04

SeaRise reviewed May 16, 2022

View reviewed changes

SeaRise self-requested a review May 26, 2022 07:55

windtalker added 2 commits May 26, 2022 17:02

fix clang-tidy

fb94ca2

Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>

Merge branch 'refine_function_test_framework' of github.com:windtalke…

321f437

…r/tiflash into refine_function_test_framework

SeaRise reviewed May 26, 2022

View reviewed changes

dbms/src/Core/Field.h Outdated Show resolved Hide resolved

dbms/src/DataTypes/FieldToDataType.h Outdated Show resolved Hide resolved

dbms/src/TestUtils/FunctionTestUtils.cpp Outdated Show resolved Hide resolved

dbms/src/TestUtils/ColumnsToTiPBExpr.cpp Outdated Show resolved Hide resolved

windtalker and others added 3 commits May 26, 2022 18:00

Update dbms/src/TestUtils/ColumnsToTiPBExpr.cpp

2aa78c1

Co-authored-by: SeaRise <hhssearise@foxmail.com>

address comments

61f24e0

Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>

Merge branch 'refine_function_test_framework' of github.com:windtalke…

e32adee

…r/tiflash into refine_function_test_framework

SeaRise self-requested a review May 26, 2022 10:10

SeaRise approved these changes May 26, 2022

View reviewed changes

dbms/src/TestUtils/ColumnsToTiPBExpr.cpp Outdated Show resolved Hide resolved

dbms/src/TestUtils/ColumnsToTiPBExpr.cpp Outdated Show resolved Hide resolved

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label May 26, 2022

ywqzzy approved these changes May 26, 2022

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels May 26, 2022

windtalker and others added 2 commits May 26, 2022 20:23

Update dbms/src/TestUtils/ColumnsToTiPBExpr.cpp

f6ed2be

Co-authored-by: SeaRise <hhssearise@foxmail.com>

Update dbms/src/TestUtils/ColumnsToTiPBExpr.cpp

1b0239d

Co-authored-by: SeaRise <hhssearise@foxmail.com>

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label May 26, 2022

windtalker added 2 commits May 26, 2022 20:41

fix clang-tidy

e3640bd

Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>

Merge branch 'refine_function_test_framework' of github.com:windtalke…

5e81690

…r/tiflash into refine_function_test_framework

ti-chi-bot removed the status/can-merge Indicates a PR has been approved by a committer. label May 26, 2022

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label May 26, 2022

ti-chi-bot merged commit 973de13 into pingcap:master May 26, 2022

windtalker mentioned this pull request May 30, 2022

enable new function test framework by default #5029

Merged

12 tasks

windtalker deleted the refine_function_test_framework branch January 30, 2023 12:50

Conversation

windtalker commented May 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

Uh oh!

ti-chi-bot commented May 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

windtalker commented May 14, 2022

Uh oh!

windtalker commented May 14, 2022

Uh oh!

sre-bot commented May 14, 2022

Coverage for changed files

Coverage summary

Uh oh!

windtalker commented May 14, 2022

Uh oh!

sre-bot commented May 14, 2022

Coverage for changed files

Coverage summary

Uh oh!

SeaRise commented May 16, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

windtalker commented May 16, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SeaRise commented May 16, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SeaRise left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ywqzzy left a comment

Choose a reason for hiding this comment

Uh oh!

windtalker commented May 26, 2022

Uh oh!

ti-chi-bot commented May 26, 2022

Uh oh!

ti-chi-bot commented May 26, 2022

Uh oh!

windtalker commented May 11, 2022 •

edited

Loading

ti-chi-bot commented May 11, 2022 •

edited

Loading