Skip to content
This repository was archived by the owner on Jul 24, 2024. It is now read-only.
This repository was archived by the owner on Jul 24, 2024. It is now read-only.

Pipelining split and import #419

@YuJuncen

Description

@YuJuncen

Feature Request

After #266, our main progress of restoring is pipelined, but at the split-scatter and import stage, it's sequential, which will waste many time during restoring.

Describe your feature request related problem:

N/A

Describe the feature you'd like:

Pipelining split and import.

Describe alternatives you've considered:

N/A

Teachability, Documentation, Adoption, Migration Strategy:

Unfortunately, we must change the interface of BatchSender to make it able to run asynchronously, instead of returning a nil which presenting all tables are restored, every time a table has been restored(or, some error happens), we should emit them to a sink:

// TableSink is a sink for asynchronous task.
type TableSink interface {
	EmitTable(utils.Table)
	EmitError(error)
}

// BatchSender is the abstract of how the batcher send a batch. 
type BatchSender interface {
	// RestoreBatch will send the restore request.
	RestoreBatch(ctx context.Context, ranges []rtree.Range, rewriteRules *RewriteRules, sink TableSink)
	Close()
}

Fortunately, the batcher itself can run asynchronously, so we don't need modify it greatly.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions