Skip to content

Multi-thread upload for different backends #4798

@NyaMisty

Description

@NyaMisty

What is your current rclone version (output from rclone version)?

rclone v1.53.3 :)

What problem are you are trying to solve?

Currently, backend can only upload file in one thread, or do some buffering and implement multi thread in backend's own logic.

If we use only one thread, when faced with super big file (like 700GB~2TB), the uploading would be very unstable and slow due to TCP flow control & various QoS.

But if we implement buffering in backend, that'll mean each backend would need to implement their own multi thread management logic, which I think is redundant and not that elegant.

How do you think rclone should be changed to solve that?

For most backend, they split the file into chunks of various size before uploading. So, apparently we should implement multi-thread transfer by directly offering chunks to backend (block-level sequential access), instead of using OpenWriterAt interface (totally random access), which could seldomly be implemented on a remote fs.
So personally I suggest we add a special Put interface to receive a ChunkGetter, which reads chunks from the source file object in a multi-threaded manner, and then reimplement a multi-thread-streams for uploading process.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions