Help triaging git-lfs checkout failure #4974
Description
Describe the issue
In our CI environment, we run git lfs fetch followed by a git lfs checkout to properly pull down/populate large fixture objects in our repo from git LFS pointers, needed to run our tests. Randomly (anecdotally, seems to be about 5% of the time) the git lfs checkout step silently fails (see logs below), leaving the local repo checkout in a dirty state (unexpected/undesired behavior for CI environment for obvious reasons)
To break this apart, there's at least 2 issues/problems for us here:
git lfs checkoutis failing (see error message and stack trace below) and we cannot figure out why- when the above failure occurs, the
git lfs checkoutprocess still exits cleanly (exit code 0), despite failing and leaving our repo in a dirty state.
Any help in debugging the first problem would be very much appreciated, and any thoughts on if the second is an expected behavior (and if so, why) or if it's worth opening a separate bug report issue would also be helpful. Thanks!
System environment
CircleCI job, running with their Docker Executor, using a variety of images (observed on cimg/go:1.17, cimg/ruby:2.7.5-node, cimg/node:14.18.1-browsers, cimg/node:14.18.1, etc)
git lfs is not installed by default on any of these base cimg images, so it is installed each run using the following bash
# Skip installing git-lfs if already present
if ! command -v git-lfs &> /dev/null; then
curl -sL https://github.com/git-lfs/git-lfs/releases/download/v3.1.2/git-lfs-linux-amd64-v3.1.2.tar.gz -o git-lfs.tar.gz \
&& tar -xzf git-lfs.tar.gz \
&& sudo ./install.sh
fi
git-lfs --version # Print out version for debugging
cd ~/build && git lfs install # Finally, install it in the repoError logged to ~/build/.git/lfs/logs/20220428T065800.570779134.log
git-lfs/3.1.2 (GitHub; linux amd64; go 1.17.6)
git version 2.35.1
$ git-lfs checkout
Error updating the Git index:
panic: runtime error: index out of range [1] with length 1
goroutine 1 [running]:
github.com/git-lfs/git-lfs/v3/git.(*FilterProcessScanner).readRequest(0xc0000a3b48)
work/git-lfs/git-lfs/git/filter_process_scanner.go:188 +0x245
github.com/git-lfs/git-lfs/v3/git.(*FilterProcessScanner).Scan(0xc0000a3b48)
work/git-lfs/git-lfs/git/filter_process_scanner.go:152 +0x76
github.com/git-lfs/git-lfs/v3/commands.filterCommand(0xc000125900, {0x9be0d9, 0x0, 0x0})
work/git-lfs/git-lfs/commands/command_filter_process.go:74 +0x345
github.com/spf13/cobra.(*Command).execute(0xc000125900, {0xe710b8, 0x0, 0x0})
go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:766 +0x5f8
github.com/spf13/cobra.(*Command).ExecuteC(0xc000124000)
go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:852 +0x2dc
github.com/spf13/cobra.(*Command).Execute(...)
go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:800
github.com/git-lfs/git-lfs/v3/commands.Run()
work/git-lfs/git-lfs/commands/run.go:107 +0x2ed
main.main()
work/git-lfs/git-lfs/git-lfs.go:34 +0xea
error: packet write with format failed
error: external filter 'git-lfs filter-process' failed
fatal: title-vi/fixtures/orig_map_trips.json: clean filter 'lfs' failed
exit status 128
Current time in UTC:
2022-04-28 06:58:00
Environment:
LocalWorkingDir=/home/circleci/remix
LocalGitDir=/home/circleci/remix/.git
LocalGitStorageDir=/home/circleci/remix/.git
LocalMediaDir=/home/circleci/remix/.git/lfs/objects
LocalReferenceDirs=
TempDir=/home/circleci/remix/.git/lfs/tmp
ConcurrentTransfers=8
TusTransfers=false
BasicTransfersOnly=false
SkipDownloadErrors=false
FetchRecentAlways=false
FetchRecentRefsDays=7
FetchRecentCommitsDays=0
FetchRecentRefsIncludeRemotes=true
PruneOffsetDays=3
PruneVerifyRemoteAlways=false
PruneRemoteName=origin
LfsStorageDir=/home/circleci/remix/.git/lfs
AccessDownload=none
AccessUpload=none
DownloadTransfers=basic,lfs-standalone-file,ssh
UploadTransfers=basic,lfs-standalone-file,ssh
GIT_EXEC_PATH=/usr/lib/git-core
GIT_TRACE=1
Client IP addresses:
192.168.240.3
If relevant, include the output of
git config -las a code block.
url.ssh://git@github.com.insteadof=https://github.com
gc.auto=0
filter.lfs.clean=git-lfs clean -- %f
filter.lfs.smudge=git-lfs smudge -- %f
filter.lfs.process=git-lfs filter-process
filter.lfs.required=true
core.repositoryformatversion=0
core.filemode=true
core.bare=false
core.logallrefupdates=true
remote.origin.url=git@github.com:[redacted]/[redacted].git
remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*
branch.master.remote=origin
branch.master.merge=refs/heads/master
lfs.repositoryformatversion=0
Output of git lfs env
git-lfs/3.1.2 (GitHub; linux amd64; go 1.17.6)
git version 2.33.0
Endpoint=https://github.com/[redacted]/[redacted].git/info/lfs (auth=none)
SSH=git@github.com:[redacted]/[redacted].git
LocalWorkingDir=/home/circleci/build
LocalGitDir=/home/circleci/build/.git
LocalGitStorageDir=/home/circleci/build/.git
LocalMediaDir=/home/circleci/build/.git/lfs/objects
LocalReferenceDirs=
TempDir=/home/circleci/build/.git/lfs/tmp
ConcurrentTransfers=8
TusTransfers=false
BasicTransfersOnly=false
SkipDownloadErrors=false
FetchRecentAlways=false
FetchRecentRefsDays=7
FetchRecentCommitsDays=0
FetchRecentRefsIncludeRemotes=true
PruneOffsetDays=3
PruneVerifyRemoteAlways=false
PruneRemoteName=origin
LfsStorageDir=/home/circleci/build/.git/lfs
AccessDownload=none
AccessUpload=none
DownloadTransfers=basic,lfs-standalone-file,ssh
UploadTransfers=basic,lfs-standalone-file,ssh
GIT_EXEC_PATH=/usr/lib/git-core
git config filter.lfs.process = "git-lfs filter-process"
git config filter.lfs.smudge = "git-lfs smudge -- %f"
git config filter.lfs.clean = "git-lfs clean -- %f"
Additional context
I was able to capture a failure while running git lfs fetch && GIT_TRACE=1 git lfs checkout. You can view the trace output in this gist (omitting only the redundant listing of all files in our repo and other repeated lines)