Skip to content

Retry-After header wait time when HTTP 429 is received is not considered. Does not wait. #6007

@max-wittig

Description

@max-wittig

Describe the bug
When using a git server with git-lfs support, and the rate limit when requesting files is it and HTTP 429 is emitted would expect git-lfs to wait the correct number of time, specified in Retry-After header.

What actually happens is that git-lfs retries HTTP 429 as often as possible, which puts additional load on the git server.

To Reproduce
Steps to reproduce the behavior:

  1. Set a low rate limit on your git server for git lfs (GitHub, Gitlab etc.)
  2. Try to clone a git repo with lots of git lfs tracked files by running
    GIT_CURL_VERBOSE=1 GIT_TRANSFER_TRACE=1 git clone https://your-git-server.com/your-repo.git > trace.txt 2>&1
  3. Observe the logs in trace.txt and notice that git-lfs is constantly retrying

In the trace.txt below you can see the Retry-After number counting down, clearly indicating that its not waiting for the rate limit to pass.

Retrying with git clone -c lfs.concurrenttransfers=1 or -c lfs.concurrenttransfers=1 -c lfs.transfer.batchSize=1000 did not improve the situation.

Expected behavior
When a HTTP 429 is received by a single worker and it contains the Retry-After response header with a number, git-lfs should wait for the specified amount of time and therefore respect Retry-After sent by the server.

System environment
Observed on Linux x86_64 (Fedora) and macOS 15.3.1 with git-lfs

Output of git lfs env

git-lfs/3.6.1 (GitHub; darwin arm64; go 1.23.4)
git version 2.48.1

Endpoint=https://githost.com/example-project.git/info/lfs (auth=none)
  SSH=git@githost.com:example-project.git
LocalWorkingDir=/Users/max/Documents/example-project
LocalGitDir=/Users/max/Documents/example-project/.git
LocalGitStorageDir=/Users/max/Documents/example-project/.git
LocalMediaDir=/Users/max/Documents/example-project/.git/lfs/objects
LocalReferenceDirs=
TempDir=/Users/max/Documents/example-project/.git/lfs/tmp
ConcurrentTransfers=8
TusTransfers=false
BasicTransfersOnly=false
SkipDownloadErrors=false
FetchRecentAlways=false
FetchRecentRefsDays=7
FetchRecentCommitsDays=0
FetchRecentRefsIncludeRemotes=true
PruneOffsetDays=3
PruneVerifyRemoteAlways=false
PruneVerifyUnreachableAlways=false
PruneRemoteName=origin
LfsStorageDir=/Users/max/Documents/example-project/.git/lfs
AccessDownload=none
AccessUpload=none
DownloadTransfers=basic,lfs-standalone-file,ssh
UploadTransfers=basic,lfs-standalone-file,ssh
GIT_EXEC_PATH=/opt/homebrew/opt/git/libexec/git-core
git config filter.lfs.process = "git-lfs filter-process"
git config filter.lfs.smudge = "git-lfs smudge -- %f"
git config filter.lfs.clean = "git-lfs clean -- %f"

Additional context

LFS Files downloaded (git lfs ls-files | wc -l): 652
Number of HTTP 429 received during git clone operation: 925

Related to #3122

trace.txt

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions