Skip to content

ssh auth and credential helper caching#2080

Merged
ttaylorr merged 12 commits intomasterfrom
auth-caching
Mar 28, 2017
Merged

ssh auth and credential helper caching#2080
ttaylorr merged 12 commits intomasterfrom
auth-caching

Conversation

@technoweenie
Copy link
Contributor

This is another approach to #2045. It adds implementations of lfsapi.CredentialHelper and lfsapi.SSHResolver that cache the results in-memory to prevent multiple calls over the span of a single git lfs call. While this is the same idea as #2045, it's implemented by wrapping some go interfaces, instead of changing lfsapi.Client call signatures. This is more idiomatic go, and doesn't require any changes to callers.

In the interest of testing this out for v2.0.2, the cachers are disabled until git config lfs.cachecredentials 1 is run.

To highlight the difference, I hacked the batch size to just 1 to simulate multiple batches. Output has important lines in bold to show the difference.

HTTP REMOTE

Here's the uncached output for an HTTP remote fetch:

$ git config --unset lfs.cachecredentials
$ rm -rf .git/lfs/objects && GIT_TRACE=1 git lfs fetch
...
trace git-lfs: creds: git credential fill ("https", "github.com", "github/git-lfs-test")
trace git-lfs: Filled credentials for https://github.com/github/git-lfs-test
trace git-lfs: HTTP: POST https://github.com/github/git-lfs-test.git/info/lfs/objects/batch
trace git-lfs: HTTP: 200
trace git-lfs: HTTP: LOTS OF JSON
trace git-lfs: tq: starting transfer adapter "basic"
Git LFS: (0 of 6 files) 0 B / 818.95 KB trace git-lfs: HTTP: GET https://github-cloud.s3.amazonaws.com/alambic/media/111976139/02/63/0263829989b6fd954f72baaf2fc64bc2e2f01d692d4de72986ea808f6e99813f
Git LFS: (0 of 6 files) 0 B / 818.95 KB trace git-lfs: HTTP: 200
trace git-lfs: tq: sending batch of size 1
trace git-lfs: fetch gif/droidtocat.gif [d1c8fab51418ef587fcf5dd8e73c60b9700704e2f8f5292ea12ec27c285b23a3]
trace git-lfs: api: batch 1 files
trace git-lfs: creds: git credential fill ("https", "github.com", "github/git-lfs-test")
trace git-lfs: Filled credentials for https://github.com/github/git-lfs-test
trace git-lfs: HTTP: POST https://github.com/github/git-lfs-test.git/info/lfs/objects/batch
trace git-lfs: HTTP: 200
trace git-lfs: HTTP: LOTS OF JSON

Here's the cached output:

$ git config lfs.cachecredentials 1
$ rm -rf .git/lfs/objects && GIT_TRACE=1 git lfs fetch
...
trace git-lfs: creds: git credential fill ("https", "github.com", "github/git-lfs-test")
trace git-lfs: Filled credentials for https://github.com/github/git-lfs-test
trace git-lfs: HTTP: POST https://github.com/github/git-lfs-test.git/info/lfs/objects/batch
trace git-lfs: HTTP: 200
trace git-lfs: HTTP: LOTS OF JSON
trace git-lfs: tq: starting transfer adapter "basic"
Git LFS: (0 of 6 files) 0 B / 818.95 KB trace git-lfs: HTTP: GET https://github-cloud.s3.amazonaws.com/alambic/media/111976139/02/63/0263829989b6fd954f72baaf2fc64bc2e2f01d692d4de72986ea808f6e99813f
Git LFS: (0 of 6 files) 0 B / 818.95 KB trace git-lfs: HTTP: 200
trace git-lfs: tq: sending batch of size 1
trace git-lfs: fetch gif/droidtocat.gif [d1c8fab51418ef587fcf5dd8e73c60b9700704e2f8f5292ea12ec27c285b23a3]
trace git-lfs: api: batch 1 files
trace git-lfs: creds: git credential cache ("https", "github.com", "github/git-lfs-test")
trace git-lfs: Filled credentials for https://github.com/github/git-lfs-test
trace git-lfs: HTTP: POST https://github.com/github/git-lfs-test.git/info/lfs/objects/batch
trace git-lfs: HTTP: 200
trace git-lfs: HTTP: LOTS OF JSON

Notice the git credential cache call here ^

SSH REMOTE

Here's the uncached output for an SSH remote:

$ git config --unset lfs.cachecredentials
$ rm -rf .git/lfs/objects && GIT_TRACE=1 git lfs fetch
...
trace git-lfs: tq: sending batch of size 1
trace git-lfs: fetch gif/atom-undo.gif [b9f86fab477109565871ced361ba69f2425a91fbe6057fa7a9629a8d536d7c71]
trace git-lfs: ssh: git@github.com git-lfs-authenticate github/git-lfs-test.git download
trace git-lfs: api: batch 1 files
trace git-lfs: HTTP: POST https://lfs.github.com/github/git-lfs-test/objects/batch
trace git-lfs: HTTP: 200
trace git-lfs: HTTP: LOTS OF JSON
trace git-lfs: tq: starting transfer adapter "basic"
Git LFS: (0 of 6 files) 0 B / 818.95 KB trace git-lfs: HTTP: GET https://github-cloud.s3.amazonaws.com/alambic/media/111976139/02/63/0263829989b6fd954f72baaf2fc64bc2e2f01d692d4de72986ea808f6e99813f
Git LFS: (0 of 6 files) 0 B / 818.95 KB trace git-lfs: HTTP: 200
trace git-lfs: tq: sending batch of size 1
trace git-lfs: fetch gif/droidtocat.gif [d1c8fab51418ef587fcf5dd8e73c60b9700704e2f8f5292ea12ec27c285b23a3]
trace git-lfs: ssh: git@github.com git-lfs-authenticate github/git-lfs-test.git download
trace git-lfs: api: batch 1 files
trace git-lfs: HTTP: POST https://lfs.github.com/github/git-lfs-test/objects/batch
trace git-lfs: HTTP: 200
trace git-lfs: HTTP: LOTS OF JSON

And the cached output:

$ git config lfs.cachecredentials 1
$ rm -rf .git/lfs/objects && GIT_TRACE=1 git lfs fetch
...
trace git-lfs: tq: sending batch of size 1
trace git-lfs: fetch gif/atom-undo.gif [b9f86fab477109565871ced361ba69f2425a91fbe6057fa7a9629a8d536d7c71]
trace git-lfs: ssh: git@github.com git-lfs-authenticate github/git-lfs-test.git download
trace git-lfs: api: batch 1 files
trace git-lfs: HTTP: POST https://lfs.github.com/github/git-lfs-test/objects/batch
trace git-lfs: HTTP: 200
trace git-lfs: HTTP: LOTS OF JSON
trace git-lfs: tq: starting transfer adapter "basic"
Git LFS: (0 of 6 files) 0 B / 818.95 KB trace git-lfs: HTTP: GET https://github-cloud.s3.amazonaws.com/alambic/media/111976139/02/63/0263829989b6fd954f72baaf2fc64bc2e2f01d692d4de72986ea808f6e99813f
Git LFS: (0 of 6 files) 0 B / 818.95 KB trace git-lfs: HTTP: 200
trace git-lfs: tq: sending batch of size 1
trace git-lfs: fetch gif/droidtocat.gif [d1c8fab51418ef587fcf5dd8e73c60b9700704e2f8f5292ea12ec27c285b23a3]
trace git-lfs: ssh cache: git@github.com git-lfs-authenticate github/git-lfs-test.git download
trace git-lfs: api: batch 1 files
trace git-lfs: HTTP: POST https://lfs.github.com/github/git-lfs-test/objects/batch
trace git-lfs: HTTP: 200
trace git-lfs: HTTP: LOTS OF JSON

Notice the ssh cache: message ^

Copy link
Contributor

@ttaylorr ttaylorr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👏

return &fakeCreds{list: make([]Creds, 0)}
}

type erroringCreds struct {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

}
var sshResolver SSHResolver = &sshAuthClient{os: osEnv}

if gitEnv.Bool("lfs.cachecredentials", false) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: this is missing an entry in git-lfs-config(5).ronn.

lfsapi/ssh.go Outdated
}

key := strings.Join([]string{e.SshUserAndHost, e.SshPort, e.SshPath, method}, "//")
if res, ok := c.endpoints[key]; ok && (res.ExpiresAt.IsZero() || res.ExpiresAt.After(time.Now().Add(5*time.Second))) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: I think that:

resp.ExpiresAt.IsZero() || (time.Until(res.ExpiresAt) > 5*time.Second)

is clearer.

@ttaylorr ttaylorr merged commit 1274d62 into master Mar 28, 2017
@ttaylorr ttaylorr deleted the auth-caching branch March 28, 2017 22:15
ttaylorr added a commit that referenced this pull request Mar 28, 2017
ttaylorr added a commit that referenced this pull request Mar 28, 2017
Backport #2080 for v2.0.x: ssh auth and credential helper caching
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants