[WIP] Tus Uploader by innovate-invent · Pull Request #3850 · rclone/rclone

innovate-invent · 2020-01-04T07:30:08Z

What is the purpose of this change?

Integrate the tus upload handler

Was the change discussed in an issue or in the forum before?

https://forum.rclone.org/t/http-upload-contribution/13271/35

Checklist

I have read the contribution guidelines.
I have added tests for all changes in this PR if appropriate.
I have added documentation for the changes if appropriate.
All commit messages are in house style.
I'm done, this Pull Request is ready for review :-)

innovate-invent · 2020-06-25T15:45:39Z

Hi @ncw, I noticed you are active on rclone again.
Do you have time to provide feedback so that I can move these PRs along?
https://forum.rclone.org/t/http-upload-contribution/13271/69

ncw · 2020-06-26T11:20:55Z

Hi @ncw, I noticed you are active on rclone again.
Do you have time to provide feedback so that I can move these PRs along?
https://forum.rclone.org/t/http-upload-contribution/13271/69

I'm a bit behind in PRs sorry.

What I would like to achieve most with this work is a framework for rclone copy to do resumable uploads. I think the TUS work can fit on the back of that, but I'm concerned that TUS itself will force too much policy onto rclone which will make it unsupportable by most backends.

Maybe you could start by rebasing this on master then I'll review it?

ivandeex · 2020-11-17T14:01:39Z

wow it's a big patch 🙈

ivandeex · 2020-11-17T14:03:19Z

Related to #4547

Related to #87

ivandeex · 2020-12-03T22:12:05Z

@innovate-invent
May I help you with rebasing this on master?

innovate-invent · 2020-12-03T22:17:38Z

@ivandeex absolutely, thanks

innovate-invent · 2021-01-06T00:36:59Z

PR #3842 refactors the http server code. This PR is based off of that. If you want to roll both PRs into one that is ok.

ivandeex · 2021-01-06T01:27:17Z

If you want to roll both PRs into one that is ok.

No. That's up to you.

ivandeex · 2021-01-06T01:52:48Z

@ivandeex Are named return values not allowed?

Allowed, unless shadowed by local variables. In your code you frequently name return value err and local variable err as well. You'll have to rename local variable to use pure returns.

innovate-invent · 2021-01-06T01:56:27Z

Ah!, that was written back when I misunderstood the functionality of the := operator

ivandeex · 2021-01-06T01:58:38Z

Ah!, that was written back when I misunderstood the functionality of the := operator

x := expr is similar to C++'s auto x = expr;

- Change Dir to interface - Add NewLazyDir() - Rename variable in ResumeUpload to camel case

- Implement generic Uploader for Fs that implement Concatenatable - Change ConcatUploader to store total size in file and limit the number of fragments per folder by putting them in subfolders - Utilise fs.operations for fs comparison and cleanup - Use path.Join for path building - Add ConcatReader - Refactor ConcatUploader into fragmentUploader for reuse in fallback uploader - Add FallbackUploader - Add interface checks for uploaders - Refactor cleanDir() to generic CleanUploads() for use by any Fs that uses fragmentUploader

- Implement ResumableUpload and Concatenator - Implement ResumableCleanup() for local - Update mTime of upload paths for cleanup - Refactor cleanDir() to generic CleanUploads() for use by any Fs that uses fragmentuploader - Implement lazy dir for local

- Implement ResumableUpload and Concatenator - Implement upload lifecycle policy - Utilise fs.operations for fs comparison and cleanup - Avoid calling applyUploadLifecycle on every request - Use path.Join for path building - Handle edge conditions while concatenating fragments - Remove upload policy from s3 upload bucket - Fix part count for normal upload - Refactor cleanDir() to generic CleanUploads() for use by any Fs that uses fragmentuploader

ivandeex · 2021-05-03T08:26:30Z

Note: PR #4547 will introduce the Resumer interface in rclone, which is closely related to the changes here.

innovate-invent · 2021-05-03T08:32:25Z

@ivandeex yes, this PR will compete with that one
I believe this PR provides better overall functionality

ivandeex · 2021-07-12T15:06:30Z

This pull request was last updated 3 months ago.
Please be aware that after 6 months of no code changes pull requests become eligible for closing.

innovate-invent · 2021-07-13T17:27:45Z

@ivandeex This is waiting on the Resumer interface. @ncw needs to respond before this can proceed. Please do not close until then.

ivandeex · 2021-10-22T09:56:45Z

@innovate-invent @ncw
Hopefully we can sort the resume stuff in 2-3 months by 1.58 ?

ivandeex · 2021-11-19T17:30:42Z

Hopefully we can resolve all problems around the resume code spun off from here.
After that we can take decisions on this ancient PR.

innovate-invent force-pushed the tus branch from 34ba2c6 to fdeaae7 Compare January 4, 2020 07:44

ivandeex added waiting for reply... needs rebase labels Nov 5, 2020

ivandeex added breaking change help wanted maintenance needs retest labels Nov 17, 2020

ivandeex added Remote: S3 Serving refactoring and removed maintenance labels Nov 17, 2020

ivandeex added the concurrency label Dec 1, 2020

ivandeex removed waiting for reply... help wanted labels Dec 3, 2020

ivandeex marked this pull request as ready for review January 5, 2021 21:26

ivandeex force-pushed the tus branch from 1dc4ec3 to 63b93f5 Compare January 6, 2021 00:17

This comment has been minimized.

Sign in to view

ivandeex removed concurrency needs rebase labels Jan 6, 2021

innovate-invent and others added 7 commits May 2, 2021 13:09

lib/http: prototype tus upload handler

372105a

fs: define intefaces: Uploader Resumable Concatenator Dir

c6360f3

- Change Dir to interface - Add NewLazyDir() - Rename variable in ResumeUpload to camel case

tus: fix building

36501dd

tus: update go.mod

bf4173b

innovate-invent force-pushed the tus branch from 4525f31 to bf4173b Compare May 2, 2021 23:48

ivandeex removed breaking change needs rebase needs retest waiting for reply... labels May 3, 2021

ivandeex marked this pull request as ready for review May 3, 2021 08:34

ivandeex added concurrency and removed Remote: S3 labels May 3, 2021

innovate-invent marked this pull request as draft May 3, 2021 09:33

innovate-invent mentioned this pull request May 3, 2021

feat: Add ResumableUpload interface #5299

Open

5 tasks

ivandeex added the resume label Oct 16, 2021

ivandeex added needs rebase and removed concurrency labels Oct 22, 2021

ivandeex mentioned this pull request Nov 17, 2021

chunker: implement Resumer interface #5185

Closed

5 tasks

ivandeex modified the milestones: Known Problem, Help Wanted Nov 19, 2021

Uh oh!

Conversation

innovate-invent commented Jan 4, 2020

What is the purpose of this change?

Was the change discussed in an issue or in the forum before?

Checklist

Uh oh!

innovate-invent commented Jun 25, 2020

Uh oh!

ncw commented Jun 26, 2020

Uh oh!

ivandeex commented Nov 17, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ivandeex commented Nov 17, 2020

Uh oh!

ivandeex commented Dec 3, 2020

Uh oh!

innovate-invent commented Dec 3, 2020

Uh oh!

This comment has been minimized.

innovate-invent commented Jan 6, 2021

Uh oh!

ivandeex commented Jan 6, 2021

Uh oh!

ivandeex commented Jan 6, 2021

Uh oh!

innovate-invent commented Jan 6, 2021

Uh oh!

ivandeex commented Jan 6, 2021

Uh oh!

ivandeex commented May 3, 2021

Uh oh!

innovate-invent commented May 3, 2021

Uh oh!

ivandeex commented Jul 12, 2021

Uh oh!

innovate-invent commented Jul 13, 2021

Uh oh!

ivandeex commented Oct 22, 2021

Uh oh!

ivandeex commented Nov 19, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ivandeex commented Nov 17, 2020 •

edited

Loading