Skip to content

[FLASH-549/550/626] Remove obsoleted data in Engine DeltaMerge#292

Merged
JaySon-Huang merged 9 commits intopingcap:DeltaMergeEnginefrom
JaySon-Huang:dm-remove-obsoleted-data
Nov 12, 2019
Merged

[FLASH-549/550/626] Remove obsoleted data in Engine DeltaMerge#292
JaySon-Huang merged 9 commits intopingcap:DeltaMergeEnginefrom
JaySon-Huang:dm-remove-obsoleted-data

Conversation

@JaySon-Huang
Copy link
Contributor

@JaySon-Huang JaySon-Huang commented Oct 23, 2019

  • Update GC safe point to remove obsoleted data when doing Segment split / merge / delta-merge.
  • When region removed, write a deleteRange to Engine DeltaMerge for deleting obsoleted data.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should keep this value here, and use a min_version_last_upate_time to prevent frequently update this value.

Copy link
Contributor Author

@JaySon-Huang JaySon-Huang Oct 29, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok.
And I add dm_safe_point_update_interval_seconds in Settings with default 60 seconds (which is the same interval of TiKV)

TiKV safe point update interval

@JaySon-Huang JaySon-Huang force-pushed the dm-remove-obsoleted-data branch from 5d16c9a to 72155c7 Compare October 29, 2019 06:48
@JaySon-Huang
Copy link
Contributor Author

/run-integration-tests

2 similar comments
@JaySon-Huang
Copy link
Contributor Author

/run-integration-tests

@JaySon-Huang
Copy link
Contributor Author

/run-integration-tests

@JaySon-Huang JaySon-Huang force-pushed the dm-remove-obsoleted-data branch 5 times, most recently from 8cd9c93 to 4e99b57 Compare October 30, 2019 09:34
@JaySon-Huang
Copy link
Contributor Author

/rebuild

1 similar comment
@JaySon-Huang
Copy link
Contributor Author

/rebuild

@JaySon-Huang JaySon-Huang force-pushed the dm-remove-obsoleted-data branch from 4e99b57 to 766a9f9 Compare October 30, 2019 11:00
@JaySon-Huang JaySon-Huang changed the title [FLASH-549/550] Remove obsoleted data in Engine DeltaMerge [WIP][FLASH-549/550/626] Remove obsoleted data in Engine DeltaMerge Oct 31, 2019
@JaySon-Huang JaySon-Huang force-pushed the dm-remove-obsoleted-data branch from 766a9f9 to a8bdbd7 Compare October 31, 2019 06:08
@JaySon-Huang JaySon-Huang changed the title [WIP][FLASH-549/550/626] Remove obsoleted data in Engine DeltaMerge [FLASH-549/550/626] Remove obsoleted data in Engine DeltaMerge Nov 7, 2019
@JaySon-Huang JaySon-Huang force-pushed the dm-remove-obsoleted-data branch from a8bdbd7 to 54c51ec Compare November 7, 2019 03:33
@JaySon-Huang JaySon-Huang force-pushed the dm-remove-obsoleted-data branch from 54c51ec to 5e9b62f Compare November 11, 2019 03:28
@flowbehappy
Copy link
Contributor

LGTM

@JaySon-Huang JaySon-Huang merged commit 05fb49c into pingcap:DeltaMergeEngine Nov 12, 2019
@JaySon-Huang JaySon-Huang deleted the dm-remove-obsoleted-data branch September 27, 2020 09:22
CalvinNeo pushed a commit to CalvinNeo/tiflash that referenced this pull request Sep 23, 2024
Signed-off-by: Wish <breezewish@outlook.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants