Skip to content

Commit a84cd13

Browse files
chrisd8088chrisd8088
committed
t/t-batch-transfer.sh: test multiple SSH sessions
In commit 691de51 of PR #4446 we added the "batch transfers with ssh endpoint (git-lfs-transfer)" test to our t/t-batch-transfer.sh test script in order to validate that the new SSH object transfer protocol for Git LFS was operating as expected. This test only creates a single test object, and so does not cause the Git LFS client to establish multiple SSH sessions during the batch object transfer phase. While the sole SSH session started during this phase may create a control socket to multiplex its connection, it will not be shared by any other SSH sessions, since only one is started. In practice, this means that even if the "lfs.ssh.autoMultiplex" configuration option is set to "true" and the available SSH client is considered to be compatible with OpenSSH, the "batch transfers with ssh endpoint (git-lfs-transfer)" test will never try to start an SSH session with a value for the ControlMaster argument other than "yes". Our test suite sets the GIT_SSH environment variable to refer to our lfs-ssh-echo test utility program rather than an actual SSH client. As noted in #5903, at present the Git LFS client treats this program, which it does not recognize as matching the filename of any known SSH client, as compatible with OpenSSH, and so our tests invoke our lfs-ssh-echo utility with the ControlMaster and ControlPath arguments. As described in a prior commit in this PR, our lfs-ssh-echo test utility program had a bug which prevented it from starting SSH sessions with the ControlMaster argument set to "no", so the "batch transfers with ssh endpoint (git-lfs-transfer)" test would have failed if it tried to push more than a single object. Such a test would also fail if it tried to fetch more than a single object, due to the bug described in #5880. Specifically, we returned a nil from the Connection() method of the SSHTransfer structure in our "ssh" package when we had terminated all the SSH sessions, but then sometimes tried to reference that nil pointer in the batchInternal() method of the SSHBatchClient structure in the "tq" package, causing a Go panic condition. In prior commits in this PR we resolved these problems, first the bug in the lfs-ssh-echo test utility and then the bad return value from the Connection() method of the SSHTransfer structure in the Git LFS client. In the case of the latter issue, we adjusted the Connection() method to return a non-nil error when the requested session ID exceeds the maximum number of sessions permitted, which also covers the case where all the sessions have already been terminated. Therefore we can now introduce additional tests of the SSH object transfer protocol which push and fetch multiple objects. In these tests we use the new assert_remote_object() function that we defined in a prior commit in this PR to confirm that the pushed objects are all written to the remote repository. As well, we use the new assert_ssh_transfer_sessions() function we defined in another prior commit in this PR to check that the number of SSH sessions that create a control socket matches our expectations, as do the number of startup, success, and termination trace log messages. The first of our new tests leaves the maximum number of concurrent object transfers set to the default value of eight, so that all three of the objects we create in the test are pushed in a single batch, and may also be fetched in a single batch if the available version of Git is 2.11.0 or higher. To push or fetch these objects in a single batch requires that the Git LFS client establish as many as three separate SSH sessions per invocation, of which only the first should create a control socket. The second of our new tests sets the maximum number of concurrent object transfers to two, so we expect to see a maximum of only two SSH sessions per invocation of the Git LFS client, and only the first of these sessions should start an SSH connection with a control socket. Prior to version 2.11.0, Git did not support the "process" filter attribute, and so during a clone operation Git LFS would be invoked via the "smudge" filter instead, once for each object. When testing with such a version of Git, our assert_ssh_transfer_sessions() function adjusts its expectations to account for the fact that each invocation of Git LFS during a clone operation will establish its own SSH session with a control socket, and so the total number of trace log messages from these sessions should match the total number of objects being fetched.
1 parent eca6c8a commit a84cd13

1 file changed

Lines changed: 109 additions & 0 deletions

File tree

‎t/t-batch-transfer.sh‎

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -199,10 +199,32 @@ assert_ssh_transfer_sessions() {
199199
(( ++max_expected_start ))
200200
fi
201201

202+
# Versions of Git prior to 2.11.0 invoke Git LFS via the "smudge" filter
203+
# rather than the "process" filter, so a separate Git LFS process runs for
204+
# each downloaded object and spawns its own control socket SSH connection.
205+
if [ "download" = "$direction" ]; then
206+
gitversion="$(git version | cut -d" " -f3)"
207+
set +e
208+
compare_version "$gitversion" '2.11.0'
209+
result=$?
210+
set -e
211+
if [ "$result" -eq "$VERSION_LOWER" ]; then
212+
min_expected_start="$num_objs"
213+
max_expected_start="$num_objs"
214+
min_expected_end="$num_objs"
215+
max_expected_end="$num_objs"
216+
expected_ctrl="$num_objs"
217+
fi
218+
fi
219+
220+
local max_expected_nonctrl=$(( max_expected_start - expected_ctrl ))
221+
202222
local lines="$(grep "exec: lfs-ssh-echo.*git-lfs-transfer .*${reponame}.git $direction" "$log")"
203223
local ctrl_count="$(printf '%s' "$lines" | grep -c -- '-oControlMaster=yes')"
224+
local nonctrl_count="$(printf '%s' "$lines" | grep -c -- '-oControlMaster=no')"
204225

205226
[ "$expected_ctrl" -eq "$ctrl_count" ]
227+
[ "$max_expected_nonctrl" -ge "$nonctrl_count" ]
206228

207229
assert_ssh_transfer_session_counts "$log" 'spawning pure SSH connection' \
208230
"$min_expected_start" "$max_expected_start"
@@ -247,3 +269,90 @@ begin_test "batch transfers with ssh endpoint (git-lfs-transfer)"
247269
git lfs fsck
248270
)
249271
end_test
272+
273+
begin_test "batch transfers with ssh endpoint and multiple objects (git-lfs-transfer)"
274+
(
275+
set -e
276+
277+
setup_pure_ssh
278+
279+
reponame="batch-ssh-transfer-multiple"
280+
setup_remote_repo "$reponame"
281+
clone_repo "$reponame" "$reponame"
282+
283+
contents1="test1"
284+
contents2="test2"
285+
contents3="test3"
286+
git lfs track "*.dat"
287+
printf "%s" "$contents1" >test1.dat
288+
printf "%s" "$contents2" >test2.dat
289+
printf "%s" "$contents3" >test3.dat
290+
git add .gitattributes test*.dat
291+
git commit -m "initial commit"
292+
293+
sshurl=$(ssh_remote "$reponame")
294+
git config lfs.url "$sshurl"
295+
296+
# On Windows we do not multiplex SSH connections by default, so we
297+
# enforce their use in order to match other platforms' connection counts.
298+
git config --global lfs.ssh.autoMultiplex true
299+
300+
GIT_TRACE=1 git push origin main >push.log 2>&1
301+
assert_ssh_transfer_sessions 'push.log' 'upload' 3 8
302+
assert_remote_object "$reponame" "$(calc_oid "$contents1")" "${#contents1}"
303+
assert_remote_object "$reponame" "$(calc_oid "$contents2")" "${#contents2}"
304+
assert_remote_object "$reponame" "$(calc_oid "$contents3")" "${#contents3}"
305+
306+
cd ..
307+
GIT_TRACE=1 git clone "$sshurl" "$reponame-2" 2>&1 | tee trace.log
308+
assert_ssh_transfer_sessions 'trace.log' 'download' 3 8
309+
310+
cd "$reponame-2"
311+
git lfs fsck
312+
)
313+
end_test
314+
315+
begin_test "batch transfers with ssh endpoint and multiple objects and batches (git-lfs-transfer)"
316+
(
317+
set -e
318+
319+
setup_pure_ssh
320+
321+
reponame="batch-ssh-transfer-multiple-batch"
322+
setup_remote_repo "$reponame"
323+
clone_repo "$reponame" "$reponame"
324+
325+
contents1="test1"
326+
contents2="test2"
327+
contents3="test3"
328+
git lfs track "*.dat"
329+
printf "%s" "$contents1" >test1.dat
330+
printf "%s" "$contents2" >test2.dat
331+
printf "%s" "$contents3" >test3.dat
332+
git add .gitattributes test*.dat
333+
git commit -m "initial commit"
334+
335+
sshurl=$(ssh_remote "$reponame")
336+
git config lfs.url "$sshurl"
337+
338+
# On Windows we do not multiplex SSH connections by default, so we
339+
# enforce their use in order to match other platforms' connection counts.
340+
git config --global lfs.ssh.autoMultiplex true
341+
342+
# Allow no more than two objects to be transferred in each batch.
343+
git config --global lfs.concurrentTransfers 2
344+
345+
GIT_TRACE=1 git push origin main >push.log 2>&1
346+
assert_ssh_transfer_sessions 'push.log' 'upload' 3 2
347+
assert_remote_object "$reponame" "$(calc_oid "$contents1")" "${#contents1}"
348+
assert_remote_object "$reponame" "$(calc_oid "$contents2")" "${#contents2}"
349+
assert_remote_object "$reponame" "$(calc_oid "$contents3")" "${#contents3}"
350+
351+
cd ..
352+
GIT_TRACE=1 git clone "$sshurl" "$reponame-2" 2>&1 | tee trace.log
353+
assert_ssh_transfer_sessions 'trace.log' 'download' 3 2
354+
355+
cd "$reponame-2"
356+
git lfs fsck
357+
)
358+
end_test

0 commit comments

Comments
 (0)