Skip to content

KVStore: avoid writting committed data into storage in doLearnerRead#9896

Merged
ti-chi-bot[bot] merged 2 commits intopingcap:masterfrom
JinheLin:resolve_key
Mar 7, 2025
Merged

KVStore: avoid writting committed data into storage in doLearnerRead#9896
ti-chi-bot[bot] merged 2 commits intopingcap:masterfrom
JinheLin:resolve_key

Conversation

@JinheLin
Copy link
Contributor

@JinheLin JinheLin commented Feb 20, 2025 •
edited
Loading

What problem does this PR solve?

Issue Number: close #9943

Problem Summary:

  • doLeanerRead will call RegionTable::resolveLocksAndWriteRegion to write committed data into storage.
  • This is unnecessary and may lead to data duplication or even disorder.

tiflash/dbms/src/Storages/KVStore/Decode/PartitionStreams.cpp

Lines 459 to 493 in 6fec03d

RegionTable::ResolveLocksAndWriteRegionRes RegionTable::resolveLocksAndWriteRegion(
TMTContext & tmt,
const TiDB::TableID table_id,
const RegionPtr & region,
const Timestamp start_ts,
const std::unordered_set<UInt64> * bypass_lock_ts,
RegionVersion region_version,
RegionVersion conf_version,
const LoggerPtr & log)
{
auto region_data_lock = resolveLocksAndReadRegionData(
table_id,
region,
start_ts,
bypass_lock_ts,
region_version,
conf_version,
/* resolve_locks */ true,
/* need_data_value */ true);
return std::visit(
variant_op::overloaded{
[&](RegionDataReadInfoList & data_list_read) -> ResolveLocksAndWriteRegionRes {
if (data_list_read.empty())
return RegionException::RegionReadStatus::OK;
auto & context = tmt.getContext();
// There is no raft input here, so we can just ignore the fg flush request.
writeRegionDataToStorage(context, region, data_list_read, log);
RemoveRegionCommitCache(region, data_list_read);
return RegionException::RegionReadStatus::OK;
},
[](auto & r) -> ResolveLocksAndWriteRegionRes { return std::move(r); },
},
region_data_lock);
}

What is changed and how it works?

  • Remove writeRegionDataToStorage in resolveLocksAndWriteRegion.

Check List

Tests

  • Unit test

  • Integration test

  • Manual test (add detailed scripts or steps below)
    image

  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

None

@ti-chi-bot ti-chi-bot bot added do-not-merge/needs-linked-issue release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Feb 20, 2025
@JinheLin
Copy link
Contributor Author

JinheLin commented Feb 20, 2025

/retest

@JinheLin JinheLin changed the title KVStore: remove write storage in doLearnerRead WIP: KVStore: remove write storage in doLearnerRead Feb 27, 2025
@ti-chi-bot ti-chi-bot bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 27, 2025
@JinheLin JinheLin changed the title WIP: KVStore: remove write storage in doLearnerRead KVStore: remove write storage in doLearnerRead Mar 5, 2025
@ti-chi-bot ti-chi-bot bot removed do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. do-not-merge/needs-linked-issue labels Mar 5, 2025
@JinheLin JinheLin changed the title KVStore: remove write storage in doLearnerRead KVStore: avoid writring committed data into storage in doLearnerRead Mar 5, 2025
@JinheLin JinheLin changed the title KVStore: avoid writring committed data into storage in doLearnerRead KVStore: avoid writting committed data into storage in doLearnerRead Mar 5, 2025
JaySon-Huang
JaySon-Huang approved these changes Mar 6, 2025
View reviewed changes
Copy link
Contributor

@JaySon-Huang JaySon-Huang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ti-chi-bot ti-chi-bot bot added needs-1-more-lgtm Indicates a PR needs 1 more LGTM. approved labels Mar 6, 2025
@JinheLin JinheLin requested a review from CalvinNeo March 7, 2025 01:55
CalvinNeo
CalvinNeo approved these changes Mar 7, 2025
View reviewed changes
Copy link
Member

@CalvinNeo CalvinNeo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Mar 7, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: CalvinNeo, JaySon-Huang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:
  • OWNERS [CalvinNeo,JaySon-Huang]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Mar 7, 2025
@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Mar 7, 2025

[LGTM Timeline notifier]

Timeline:

  • 2025-03-06 10:39:25.740176803 +0000 UTC m=+524478.869096543: ☑️ agreed by JaySon-Huang.
  • 2025-03-07 02:54:30.56629947 +0000 UTC m=+582983.695219209: ☑️ agreed by CalvinNeo.

@ti-chi-bot ti-chi-bot bot merged commit db85ffa into pingcap:master Mar 7, 2025
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@JaySon-Huang JaySon-Huang JaySon-Huang approved these changes

@CalvinNeo CalvinNeo CalvinNeo approved these changes

Assignees

No one assigned

Labels

approved lgtm release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Avoid write storage in doLearnerRead

3 participants

@JinheLin @JaySon-Huang @CalvinNeo