Skip to content

Test: Bind delta merge storage with executor ut#6561

Merged
ti-chi-bot merged 14 commits intopingcap:masterfrom
ywqzzy:bind_delta_merge_storage_with_executor_ut
Jan 3, 2023
Merged

Test: Bind delta merge storage with executor ut#6561
ti-chi-bot merged 14 commits intopingcap:masterfrom
ywqzzy:bind_delta_merge_storage_with_executor_ut

Conversation

@ywqzzy
Copy link
Contributor

@ywqzzy ywqzzy commented Dec 29, 2022 •
edited
Loading

What problem does this PR solve?

Issue Number: ref #4609

Problem Summary:
We want to use DeltaMergeStorage to run the query in ut framework.
Then we can test the whole query process in single node(not including mpp with multiple nodes, we assume that data are stored in one node).
It will be helpful for features that change both storage code and compute code. Eg: late materialization and pipeline model.
Maybe RSFilter can use ut framework too.
Here is one simple case:

context.addMockDeltaMerge({"test_db", "t0"},
                                  {{"col0", TiDB::TP::TypeLongLong}},
                                  {{toVec<Int64>("col0", {0, 1, 2, 3, 4, 5, 6, 7})}});

auto request = context
                       .scan("test_db", "t0")
                       .build(context);
executeAndAssertColumnsEqual(
        request,
        {{toNullableVec<Int64>("col0", {0, 1, 2, 3, 4, 5, 6, 7})}});

What is changed and how it works?

  1. We use MockStorage to hold DeltaMergeStorage.
    std::unordered_map<String, Int64> name_to_id_map_for_delta_merge; /// <table_name, table_id>
    std::unordered_map<Int64, MockColumnInfoVec> table_schema_for_delta_merge; /// <table_id, columnInfo>
    std::unordered_map<Int64, std::shared_ptr<StorageDeltaMerge>> storage_delta_merge_map; // <table_id, StorageDeltaMerge>
    std::unordered_map<String, TableInfo> table_infos_for_delta_merge; /// <table_name, table_info>
    std::unordered_map<Int64, NamesAndTypes> names_and_types_map_for_delta_merge; /// <table_id, NamesAndTypes>
  1. When writing the test, we must call
MockDAGRequestContext::addMockDeltaMerge(const MockTableName & name, const MockColumnInfoVec & columnInfos, ColumnsWithTypeAndName columns);

to define the table schema and insert table columns.
3. addMockDeltaMerge will call

addMockDeltaMergeSchema(db, table, columnInfos);
addMockDeltaMergeData(db, table, columns);

to add schema and data into corresponding DeltaMergeStorage
4. When running a test query, planner will choose to use the DeltaMergeStorage to generate source streams.
5. BTW, I have moved a bunch of code like

using ColMyDateTimeNullableType = std::optional<typename TypeTraits<MyDateTime>::FieldType>;
using ColDecimalNullableType = std::optional<typename TypeTraits<Decimal32>::FieldType>;

into dbms/src/TestUtils/ExecutorTestUtils.h to make the test code cleaner.

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

None

@ti-chi-bot
Copy link
Member

ti-chi-bot commented Dec 29, 2022 •
edited
Loading

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • Lloyd-Pottiger
  • SeaRise

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Details

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Dec 29, 2022
@ywqzzy
Copy link
Contributor Author

ywqzzy commented Dec 30, 2022

/run-all-tests

@ywqzzy ywqzzy changed the title [WIP]Test: Bind delta merge storage with executor ut Test: Bind delta merge storage with executor ut Dec 30, 2022
@ti-chi-bot ti-chi-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 30, 2022
@ywqzzy
Copy link
Contributor Author

ywqzzy commented Dec 30, 2022

/run-all-tests

@ywqzzy
Copy link
Contributor Author

ywqzzy commented Dec 30, 2022

/run-all-tests

SeaRise
SeaRise approved these changes Dec 30, 2022
View reviewed changes
Copy link
Contributor

@SeaRise SeaRise left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Others lgtm

dbms/src/Debug/MockStorage.cpp

BlockInputStreamPtr MockStorage::getStreamFromDeltaMerge(Context & context, Int64 id)
{
auto storage = storage_delta_merge_map[id];
Copy link
Contributor

@SeaRise SeaRise Dec 30, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add some assert for function xxxFromDeltaMerge

@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Dec 30, 2022
@SeaRise
Copy link
Contributor

SeaRise commented Dec 30, 2022

/cc @JaySon-Huang

@SeaRise
Copy link
Contributor

SeaRise commented Dec 30, 2022

/run-unit-test

@ywqzzy
Copy link
Contributor Author

ywqzzy commented Dec 30, 2022

/run-unit-tests

@ywqzzy
Copy link
Contributor Author

ywqzzy commented Dec 31, 2022

/run-unit-tests

@ywqzzy
Copy link
Contributor Author

ywqzzy commented Jan 3, 2023

/run-all-tests

@ywqzzy
Copy link
Contributor Author

ywqzzy commented Jan 3, 2023

/run-all-tests

@ywqzzy ywqzzy force-pushed the bind_delta_merge_storage_with_executor_ut branch from 434b53a to 59fd171 Compare January 3, 2023 05:04
@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Jan 3, 2023
@ywqzzy
Copy link
Contributor Author

ywqzzy commented Jan 3, 2023

/merge

@ti-chi-bot
Copy link
Member

ti-chi-bot commented Jan 3, 2023

@ywqzzy: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot
Copy link
Member

ti-chi-bot commented Jan 3, 2023

This pull request has been accepted and is ready to merge.

DetailsCommit hash: b67b85a

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Jan 3, 2023
@ti-chi-bot ti-chi-bot merged commit 78fa108 into pingcap:master Jan 3, 2023
windtalker pushed a commit to windtalker/tiflash that referenced this pull request Jan 4, 2023
@ywqzzy @windtalker
Test: Bind delta merge storage with executor ut (pingcap#6561)
4ae3b8b 
ref pingcap#4609
@ywqzzy ywqzzy mentioned this pull request Jan 4, 2023
Closed
54 tasks
JaySon-Huang pushed a commit to JaySon-Huang/tiflash that referenced this pull request Jan 4, 2023
@ywqzzy @JaySon-Huang
Test: Bind delta merge storage with executor ut (pingcap#6561)
3887b91 
ref pingcap#4609
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Lloyd-Pottiger Lloyd-Pottiger Lloyd-Pottiger approved these changes

@JaySon-Huang JaySon-Huang Awaiting requested review from JaySon-Huang

+1 more reviewer

@SeaRise SeaRise SeaRise approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@ywqzzy @ti-chi-bot @SeaRise @Lloyd-Pottiger