Skip to content

Storages: Shutdown the LocalIndexScheduler before shutting down PageStorage/DeltaMergeStore#9712

Merged
ti-chi-bot[bot] merged 5 commits intopingcap:masterfrom
JaySon-Huang:fix_index_scheduler_when_shutdown
Dec 13, 2024
Merged

Storages: Shutdown the LocalIndexScheduler before shutting down PageStorage/DeltaMergeStore#9712
ti-chi-bot[bot] merged 5 commits intopingcap:masterfrom
JaySon-Huang:fix_index_scheduler_when_shutdown

Conversation

@JaySon-Huang
Copy link
Contributor

@JaySon-Huang JaySon-Huang commented Dec 10, 2024 •
edited
Loading

What problem does this PR solve?

Issue Number: close #9714

Problem Summary:

Checkout the logging in issue.

From "Invalid page id, entry not exist [page_id=451.26] [resolve_id=451.26]" and the StackTrace DB::DM::Segment::restoreSegment(std::__1::shared_ptr<DB::Logger> const&, DB::DM::DMContext&, unsigned long) dbms/src/Storages/DeltaMerge/Segment.cpp:427 we can know that something is wrong when restoring the StableValueSpace of Segment. After adding some debugging message, we know that the error comes from restoring table_id=451, segment_id=8822.

Then dump the BlobData from PageStorage "meta". We found that the last two records persisted on disk is invalid as a SegmentMetaInfo.

Before the error happen, we saw that the table_id=451, segment_id=8822 had applied a SegmentUpdateMeta task after DeltaMergeStore::shutdown. Segment::replaceStableMetaVersion will access to a already shutdown PageStorage. Causing the WriteBatch wrote broken data.

[2024/12/10 00:04:03.027 +08:00] [INFO] [Server.cpp:1589] ["Shutting down storages."] [thread_id=1]
...
-- called by `DeltaMergeStore::shutdown` on table_id=451
[2024/12/10 00:04:03.050 +08:00] [INFO] [LocalIndexerScheduler.cpp:170] ["Removed 0 tasks, keyspace_id=4294967295 table_id=451"] [thread_id=1]
[2024/12/10 00:04:03.050 +08:00] [INFO] [RegionTable.cpp:152] ["remove table from RegionTable success, keyspace=4294967295 table_id=451"] [thread_id=1]
...
[2024/12/10 00:04:03.065 +08:00] [INFO] [LocalIndexerScheduler.cpp:69] ["LocalIndexerScheduler is destroying. Waiting scheduler and tasks to finish..."] [thread_id=1]
-- the running task on table_id=451, segment_id=8822, file_id=13172 is not removed. And 
[2024/12/10 00:05:28.470 +08:00] [INFO] [DMFileV3IncrementWriter.cpp:104] ["Write incremental update for DMFile, local_path=/data1/gengliqi/tidb/data/tiflash-9000/data/t_451/stable/dmf_13172 dmfile_path=/data1/gengliqi/tidb/data/tiflash-9000/data/t_451/stable/dmf_13172 old_meta_version=0 new_meta_version=1"] [thread_id=403]
[2024/12/10 00:05:28.474 +08:00] [INFO] [DeltaMergeStore_InternalSegment.cpp:772] ["EnsureStableLocalIndex - Finish building index, dm_files=[dmf_13172(v=0)]"] [source="keyspace=4294967295 table_id=451 segmentEnsureStableLocalIndex source_segment=<segment_id=8822 epoch=4 range=[249948,499989)>"] [thread_id=403]
-- `Segment::replaceStableMetaVersion` will access to a already shutdown PageStorage. Causing the WriteBatch wrote broken data.
[2024/12/10 00:05:28.479 +08:00] [INFO] [DeltaMergeStore_InternalSegment.cpp:690] ["SegmentUpdateMeta - Finish, old_segment=<segment_id=8822 epoch=4 range=[249948,499989)> new_segment=<segment_id=8822 epoch=5 range=[249948,499989)>"] [source="keyspace=4294967295 table_id=451"] [thread_id=403]
-- `DeltaMergeStore::~DeltaMergeStore` is called
[2024/12/10 00:05:28.479 +08:00] [INFO] [DeltaMergeStore.cpp:372] ["Release DeltaMerge Store start"] [source="keyspace=4294967295 table_id=451"] [thread_id=403]
[2024/12/10 00:05:28.479 +08:00] [INFO] [DeltaMergeStore.cpp:376] ["Release DeltaMerge Store end"] [source="keyspace=4294967295 table_id=451"] [thread_id=403]
[2024/12/10 00:05:28.479 +08:00] [INFO] [LocalIndexerScheduler.cpp:85] ["LocalIndexerScheduler is destroyed"] [thread_id=1]

What is changed and how it works?

Storages: Shutdown the LocalIndexScheduler before shutting down PageStorage/DeltaMergeStore
* Add a method `LocalIndexerScheduler::shutdown()` and ensure the running task are all finished before shutting down the GlobalPageStorage in `ContextShared::shutdown()`.
  • Add a method LocalIndexerScheduler::shutdown() and ensure the running task are all finished before shutting down the GlobalPageStorage in ContextShared::shutdown().
  • Add error message about the segment_id when restoreSegment failed
  • Add debugging tool command for reading the BlobData from PageStorage instance

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
1. Deploy a cluster with vector data
2. Create a script to run add/drop vector index repeatly 
`alter table vector_bench_test drop index idx_emb_l2;`
`alter table vector_bench_test add vector index idx_emb_l2 ((VEC_L2_DISTANCE(embedding)));`
3. Create a script to restart tiflash repeatly
4. Check whether tiflash restart successfully
5. TiFlash restart successfully after running those 2 scripts for 7 hours
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

Fix an issue that TiFlash may fail to restart after applying vector index build jobs

@ti-chi-bot ti-chi-bot bot added do-not-merge/needs-linked-issue release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed do-not-merge/needs-linked-issue release-note-none Denotes a PR that doesn't merit a release note. labels Dec 10, 2024
@JaySon-Huang JaySon-Huang changed the title Storages: Shutdown the LocalIndexScheduler before shutting down PageStorage/DeltaMergeStore [WIP] Storages: Shutdown the LocalIndexScheduler before shutting down PageStorage/DeltaMergeStore Dec 10, 2024
@ti-chi-bot ti-chi-bot bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 10, 2024
@JaySon-Huang JaySon-Huang mentioned this pull request Dec 11, 2024
Merged
12 tasks
@ti-chi-bot ti-chi-bot bot added the needs-cherry-pick-release-8.5 Should cherry pick this PR to release-8.5 branch. label Dec 11, 2024
@JaySon-Huang
Shutdown the LocalIndexScheduler before shutting down PageStorage/Del…
e69d4dd 
…taMergeStore

Signed-off-by: JaySon-Huang <tshent@qq.com>
@JaySon-Huang JaySon-Huang force-pushed the fix_index_scheduler_when_shutdown branch from d477249 to e69d4dd Compare December 13, 2024 03:29
@JaySon-Huang
fix
342f5f4 
Signed-off-by: JaySon-Huang <tshent@qq.com>
@JaySon-Huang
Use debug level for useless logging
3d80db6 
Signed-off-by: JaySon-Huang <tshent@qq.com>
@ti-chi-bot ti-chi-bot bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Dec 13, 2024
@JaySon-Huang JaySon-Huang changed the title [WIP] Storages: Shutdown the LocalIndexScheduler before shutting down PageStorage/DeltaMergeStore Storages: Shutdown the LocalIndexScheduler before shutting down PageStorage/DeltaMergeStore Dec 13, 2024
@ti-chi-bot ti-chi-bot bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 13, 2024
@JaySon-Huang
Add test case
1093e34 
Signed-off-by: JaySon-Huang <tshent@qq.com>
@ti-chi-bot ti-chi-bot bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Dec 13, 2024
@JaySon-Huang
Remove test code
9a7ab39 
Signed-off-by: JaySon-Huang <tshent@qq.com>
@ti-chi-bot ti-chi-bot bot added needs-1-more-lgtm Indicates a PR needs 1 more LGTM. approved labels Dec 13, 2024
@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Dec 13, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: JinheLin, Lloyd-Pottiger

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:
  • OWNERS [JinheLin,Lloyd-Pottiger]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Dec 13, 2024
@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Dec 13, 2024

[LGTM Timeline notifier]

Timeline:

  • 2024-12-13 08:11:55.832021108 +0000 UTC m=+598905.920823650: ☑️ agreed by Lloyd-Pottiger.
  • 2024-12-13 08:14:11.084851816 +0000 UTC m=+599041.173654354: ☑️ agreed by JinheLin.

@ti-chi-bot ti-chi-bot bot merged commit b16a5f9 into pingcap:master Dec 13, 2024
ti-chi-bot pushed a commit to ti-chi-bot/tiflash that referenced this pull request Dec 13, 2024
@JaySon-Huang @ti-chi-bot
Storages: Shutdown the LocalIndexScheduler before shutting down PageS…
26aa2bd 
…torage/DeltaMergeStore (pingcap#9712)

close pingcap#9714

Storages: Shutdown the LocalIndexScheduler before shutting down PageStorage/DeltaMergeStore
* Add a method `LocalIndexerScheduler::shutdown()` and ensure the running task are all finished before shutting down the GlobalPageStorage in `ContextShared::shutdown()`.

Signed-off-by: JaySon-Huang <tshent@qq.com>
@ti-chi-bot
Copy link
Member

ti-chi-bot commented Dec 13, 2024

In response to a cherrypick label: new pull request created to branch release-8.5: #9728.

@ti-chi-bot ti-chi-bot mentioned this pull request Dec 13, 2024
Closed
12 tasks
@JaySon-Huang JaySon-Huang deleted the fix_index_scheduler_when_shutdown branch December 13, 2024 08:21
yongman pushed a commit to yongman/tiflash that referenced this pull request Jan 23, 2025
@JaySon-Huang
Storages: Shutdown the LocalIndexScheduler before shutting down PageS…
ad9aeeb 
…torage/DeltaMergeStore (pingcap#9712)

close pingcap#9714

Storages: Shutdown the LocalIndexScheduler before shutting down PageStorage/DeltaMergeStore
* Add a method `LocalIndexerScheduler::shutdown()` and ensure the running task are all finished before shutting down the GlobalPageStorage in `ContextShared::shutdown()`.

Signed-off-by: JaySon-Huang <tshent@qq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@JinheLin JinheLin JinheLin approved these changes

@Lloyd-Pottiger Lloyd-Pottiger Lloyd-Pottiger approved these changes

@CalvinNeo CalvinNeo Awaiting requested review from CalvinNeo

Assignees

No one assigned

Labels

approved lgtm needs-cherry-pick-release-8.5 Should cherry pick this PR to release-8.5 branch. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

TiFlash failed to restart with "Invalid page id, entry not exist [page_id=...]"

4 participants

@JaySon-Huang @ti-chi-bot @JinheLin @Lloyd-Pottiger