This repository was archived by the owner on Jul 24, 2024. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 100
This repository was archived by the owner on Jul 24, 2024. It is now read-only.
Pipelining split and import #419
Copy link
Copy link
Closed
Labels
difficulty/2-mediumMedium-difficulty issueMedium-difficulty issuedo-not-merge/work-in-progressstatus/WIPstatus/WIPpriority/P2Medium priority issueMedium priority issuetype/feature-requestNew feature or requestNew feature or request
Description
Feature Request
After #266, our main progress of restoring is pipelined, but at the split-scatter and import stage, it's sequential, which will waste many time during restoring.
Describe your feature request related problem:
N/A
Describe the feature you'd like:
Pipelining split and import.
Describe alternatives you've considered:
N/A
Teachability, Documentation, Adoption, Migration Strategy:
Unfortunately, we must change the interface of BatchSender to make it able to run asynchronously, instead of returning a nil which presenting all tables are restored, every time a table has been restored(or, some error happens), we should emit them to a sink:
// TableSink is a sink for asynchronous task.
type TableSink interface {
EmitTable(utils.Table)
EmitError(error)
}
// BatchSender is the abstract of how the batcher send a batch.
type BatchSender interface {
// RestoreBatch will send the restore request.
RestoreBatch(ctx context.Context, ranges []rtree.Range, rewriteRules *RewriteRules, sink TableSink)
Close()
}Fortunately, the batcher itself can run asynchronously, so we don't need modify it greatly.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
difficulty/2-mediumMedium-difficulty issueMedium-difficulty issuedo-not-merge/work-in-progressstatus/WIPstatus/WIPpriority/P2Medium priority issueMedium priority issuetype/feature-requestNew feature or requestNew feature or request