Synchronize first credentials access

It might be cool to restructure lfs.Upload so it can take a number of files. Then, it could post an array of objects and get back multiple link relations. One Git LFS API request for every n files.

Probably out of scope for this PR. Just thinking....

How will upload progress work on the cli output? We almost need something like the pb package that can track multiple, separate progress bars.

I think posting an array would be good, that'll save a lot of round trips when there's a lot of files. I would agree that's out of scope for this PR, that would require an API change because I don't think we currently say that is allowed.

For progress, it probably will have to be something with pb showing multiple bars. That should have been added to the TODO list.

A progress bar that shows multiple entries is going to be problematic. It's not too complicated on unixy environments with ansi codes, but windows is a different story. I think you can access some terminal manipulation routines through the windows syscalls and kernel32, but that's a bit beyond my knowledge. There are a couple cross platform terminal ui libraries (e.g. go-termbox), but those want to take over the entire screen, and with git also dumping output to the screen that's not very practical.

We have a few options we can do:

• Don't show the progress bar(s) for concurrent uploads
• Show a single progress bar that covers the entire upload set
• Show a multi-bar on unix, and nothing or the single bar on windows (perhaps someone can contribute a windows multi-bar)

Show a single progress bar that covers the entire upload set

I like this idea. Instead of a single "uploading foo.bar (100MB)" message per file, have a single message showing the number of files and total storage, as well as a single progress bar for the whole thing.

That's my leaning as well, because it's the simplest :).

pb doesn't really know about progress bars that span multiple items, but we can sneak it in with the newer Prefix() method by updating the prefix as transfers complete:

Super cool 👍

This is now ready for 👀

The number of uploads that will happen concurrently is configured via lfs.concurrentuploads (better name?). It defaults to 3 (better default?).

When push or pre-push happens, the files to be uploaded are calculated as before and added to a queue. The queue is then processed. In order to prevent being prompted for authentication n times simultaneously, the queue is processed sequentially until an API call completes successfully. This should happen right after the first API call. Once an API call succeeds the upload workers will wake up and begin processing the queue concurrently until it's finished.

Progress output is now a single bar that covers the combined sizes of all uploads (see gif above).

Edit to add: any upload errors that occur are collected and displayed after processing is complete.

Found a small issue:

So I used the new git lfs push command twice in row, without a git push in between. The progress counter stayed at "0 of 100 files" until the first upload. It skipped the first 50 since the first git lfs push call already uploaded them. It'd be cool if it could update the file counter and the byte counter on an upload that can be skipped because the remote server already has it.

$ git lfs push origin master 
(61 of 100 files) 3.91 MB / 19.53 MB  20.00 % 1m56s     

I'm using this to generate 50 Git LFS files to upload.

require "time"
require "fileutils"

now = Time.now.strftime("%Y%m%d-%I%M%S")
FileUtils.mkdir_p(now)

50.times do |i|
  File.open("#{now}/#{i}.dat", "w") do |f|
    f.write(`openssl rand -hex 102400`)
  end
end

puts `git add #{now}`
puts `git commit -m "files for #{now}"`

Here's how it looks now:

# create 50 files and commit

$ g lfs push origin master
(50 of 50 files) 9.77 MB / 9.77 MB  100.00 % 34s                                                                                                                                                                                                                                                                                                                            

# create 50 more files and another commit

$ git push origin master
(100 of 100 files) 19.53 MB / 19.53 MB  100.00 % 49s

The progress bar looks good. There's a lot of overhead, but I think we'll need to make some major API changes to solve.