storage: Support adding vector index in background#9411
storage: Support adding vector index in background#9411ti-chi-bot[bot] merged 5 commits intopingcap:feature/vector-indexfrom
Conversation
Signed-off-by: Wish <breezewish@outlook.com> Signed-off-by: Lloyd-Pottiger <yan1579196623@gmail.com> Co-authored-by: Wish <breezewish@outlook.com> Co-authored-by: JaySon-Huang <tshent@qq.com>
f9422af to
707e91c
Compare
| } | ||
| else if (global_context->getSharedContextDisagg()->isDisaggregatedStorageMode()) | ||
| { | ||
| global_context->initializeGlobalLocalIndexerScheduler( |
There was a problem hiding this comment.
initialize the LocalIndexerScheduler introduce by #9383, all vector index build task will be schedule by this scheduler.
There was a problem hiding this comment.
I checked that PR, and I can not figure out why it is * 8 / 10
There was a problem hiding this comment.
// There is no compute task in write node.
// Set the pool size to 80% of logical cores and 60% of memory
// to take full advantage of the resources and avoid blocking other tasks like writes and compactions.
Add some comments.
Signed-off-by: Lloyd-Pottiger <yan1579196623@gmail.com>
| { | ||
| RUNTIME_CHECK(segment != nullptr); | ||
|
|
||
| // TODO(local index): There could be some indexes are built while some indexes is not yet built after DDL |
There was a problem hiding this comment.
There could be some indexes are being built?
There was a problem hiding this comment.
Just ignore it, will remove it in a later PR.
| for (const auto & seg_id : segment_ids) | ||
| { | ||
| auto segment = id_to_segment[seg_id]; | ||
| auto new_segment = segmentUpdateMeta(lock, dm_context, segment, new_dmfiles); |
There was a problem hiding this comment.
In segmentUpdateMeta, we will acquire a lock for using Lock = std::unique_lock<std::recursive_mutex>; of a DeltaValueSpace for a Segment, while we are holding also a unique_lock of DM.
I wonder if the lock is too grain? AFAIK, the segment lock is only used in replaceStableMetaVersion. So I wonder if we could do replaceStableMetaVersion without DM lock?
There was a problem hiding this comment.
Accquiring DM lock to make sure segment will not be abandoned by other thread.
Signed-off-by: Lloyd-Pottiger <yan1579196623@gmail.com>
JaySon-Huang
left a comment
There was a problem hiding this comment.
Directly bump the version for supporting building index on both DeltaVS and StableVS
breezewish
left a comment
There was a problem hiding this comment.
The change looks minimal compared as CSE version.
Signed-off-by: Lloyd-Pottiger <yan1579196623@gmail.com>
dbms/src/Storages/DeltaMerge/DeltaMergeStore_InternalSegment.cpp
Outdated
Show resolved
Hide resolved
dbms/src/Storages/DeltaMerge/DeltaMergeStore_InternalSegment.cpp
Outdated
Show resolved
Hide resolved
dbms/src/Storages/DeltaMerge/DeltaMergeStore_InternalSegment.cpp
Outdated
Show resolved
Hide resolved
|
|
||
| LOG_DEBUG(log, "ReplaceStableMetaVersion - Finish, new_stable_files={}", new_stable_files_str()); | ||
|
|
||
| LOG_DEBUG( |
There was a problem hiding this comment.
We will print info log in DeltaMergeStore::segmentUpdateMeta
Signed-off-by: Lloyd-Pottiger <yan1579196623@gmail.com>
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: breezewish, JaySon-Huang The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
[LGTM Timeline notifier]Timeline:
|
What problem does this PR solve?
Issue Number: ref #9032
Problem Summary:
What is changed and how it works?
Check List
Tests
Side effects
STORAGE_FORMAT_V7Documentation
Release note