-
-
Notifications
You must be signed in to change notification settings - Fork 5k
Description
What is your current rclone version (output from rclone version)?
rclone v1.53.3 :)
What problem are you are trying to solve?
Currently, backend can only upload file in one thread, or do some buffering and implement multi thread in backend's own logic.
If we use only one thread, when faced with super big file (like 700GB~2TB), the uploading would be very unstable and slow due to TCP flow control & various QoS.
But if we implement buffering in backend, that'll mean each backend would need to implement their own multi thread management logic, which I think is redundant and not that elegant.
How do you think rclone should be changed to solve that?
For most backend, they split the file into chunks of various size before uploading. So, apparently we should implement multi-thread transfer by directly offering chunks to backend (block-level sequential access), instead of using OpenWriterAt interface (totally random access), which could seldomly be implemented on a remote fs.
So personally I suggest we add a special Put interface to receive a ChunkGetter, which reads chunks from the source file object in a multi-threaded manner, and then reimplement a multi-thread-streams for uploading process.