perf(ssh,modal): bulk file sync via tar pipe and tar/base64 archive#7560
Closed
kshitijk4poor wants to merge 1 commit into
Closed
perf(ssh,modal): bulk file sync via tar pipe and tar/base64 archive#7560kshitijk4poor wants to merge 1 commit into
kshitijk4poor wants to merge 1 commit into
Conversation
This was referenced Apr 11, 2026
Wire bulk_upload_fn into SSH and Modal backends, matching the pattern established for Daytona in NousResearch#7447. SSH: stages files in a temp directory mirroring the remote path layout, then pipes tar cf | ssh tar xf in a single TCP stream. Eliminates per-file scp round-trips. Modal: builds an in-memory gzipped tar archive, base64-encodes it, and decodes+extracts in one exec call. Eliminates per-file base64|exec overhead. Both backends pre-create all unique parent directories in a single call before the bulk transfer. Closes NousResearch#7465 Closes NousResearch#7467
93aba67 to
e0b1e59
Compare
Collaborator
Author
|
Superseded by #7558 which has a more robust SSH implementation (symlink staging, timeout handling, Popen failure guards) plus the Modal bulk upload from this PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Wire
bulk_upload_fninto SSH and Modal backends, matching the pattern established for Daytona in #7447. Eliminates per-file transfer overhead duringFileSyncManager.sync().Changes
SSH (
tools/environments/ssh.py)_ssh_bulk_upload(): stages files in a temp directory mirroring the remote path layout, then pipestar cf - | ssh tar xf -in a single TCP stream_scp_upload)Modal (
tools/environments/modal.py)_modal_bulk_upload(): builds an in-memory gzipped tar archive, base64-encodes it, and decodes+extracts in oneexeccallexeccall total)_modal_upload)Tests
test_ssh_bulk_upload.py— 8 tests: empty noop, staging dir layout, mkdir ordering, tar pipe structure, extract error, mkdir error, FileSyncManager wiring, ControlMaster usagetest_modal_bulk_upload.py— 7 tests: empty noop, tar archive contents, mkdir coverage, single-exec verification, exit code error, FileSyncManager wiring, timeoutBackend status
upload_files()tar cf | ssh tar xftar+base64 | exec15 new tests, all passing. Existing file_sync, SSH, and Modal tests unaffected.
Closes #7465
Closes #7467