Skip to content

Fix the wrong result of reading single format DTFile & misplaced Grafana panel#1803

Merged
ti-srebot merged 6 commits intopingcap:masterfrom
JaySon-Huang:small_fix
Apr 21, 2021
Merged

Fix the wrong result of reading single format DTFile & misplaced Grafana panel#1803
ti-srebot merged 6 commits intopingcap:masterfrom
JaySon-Huang:small_fix

Conversation

@JaySon-Huang
Copy link
Contributor

@JaySon-Huang JaySon-Huang commented Apr 21, 2021

What problem does this PR solve?

Issue Number: close #1787

Problem Summary:

  1. Sometime the panel Region write Duration (write blocks) appear in weird places. It is because the panel share a same "id=77" with the panel Handling Request Number
  2. After restored DTFiles (single file mode), we may get inconsistent rows from a DTFile even if we read with a same key range. Because the min-max indexes can not be loaded successfully.

What is changed and how it works?

  1. Fix the id of panel and re-export the json file for TiFlash-Summary
  2. Do DMFile::initializeSubFileStatIfNeeded inside DMFilePackFilter::init to ensure we can load min-max indexes if they exist.
  3. Do DMFile::initializeSubFileStatIfNeeded inside DMFile::readMeta.
    @lidezhu split the functions from readMeta to reduce the number of bytes read from the disk while initiating. But it brings another problem: We need to acquire a lock for checking whether the file is single-mode and sub_file_stats is empty every time we create a DMFileFilter. No matter the file is single-mode or directory-mode.
    Those data are small and we can optimize the initlizating by lazy init DeltaMergeStore.

Related changes

  • PR to update pingcap/docs/pingcap/docs-cn:
  • Need to cherry-pick to the release branch: 5.0

Check List

Tests

  • Unit test

Side effects

  • N/A

Release note

  • Fix the bug that the Grafana panel Region write Duration (write blocks) may be shown in wrong place

Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
@JaySon-Huang JaySon-Huang added the type/bugfix This PR fixes a bug. label Apr 21, 2021
@JaySon-Huang JaySon-Huang self-assigned this Apr 21, 2021
@JaySon-Huang JaySon-Huang requested a review from lidezhu April 21, 2021 07:56
@JaySon-Huang JaySon-Huang added the needs-cherry-pick-release-5.0 PR which needs to be cherry-picked to release-5.0 label Apr 21, 2021
@JaySon-Huang
Copy link
Contributor Author

/run-all-tests

JaySon-Huang and others added 2 commits April 21, 2021 17:31
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
@JaySon-Huang
Copy link
Contributor Author

/run-all-tests

@ti-srebot ti-srebot added the status/LGT1 Indicates that a PR has LGTM 1. label Apr 21, 2021
@JaySon-Huang
Copy link
Contributor Author

/merge

@ti-srebot ti-srebot added the status/can-merge Indicates a PR has been approved by a committer. label Apr 21, 2021
@ti-srebot
Copy link
Collaborator

/run-all-tests

@ti-srebot
Copy link
Collaborator

cherry pick to release-5.0 in PR #1806

@JaySon-Huang JaySon-Huang deleted the small_fix branch April 21, 2021 14:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-cherry-pick-release-5.0 PR which needs to be cherry-picked to release-5.0 status/can-merge Indicates a PR has been approved by a committer. status/LGT1 Indicates that a PR has LGTM 1. type/bugfix This PR fixes a bug.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ingesting DTFile to DeltaTree storage make it unable to read data

3 participants