Skip to content

Initial sparse implementation#54

Merged
derrickstolee merged 6 commits intomicrosoft:masterfrom
derrickstolee:sparse
Aug 22, 2019
Merged

Initial sparse implementation#54
derrickstolee merged 6 commits intomicrosoft:masterfrom
derrickstolee:sparse

Conversation

@derrickstolee
Copy link
Contributor

@derrickstolee derrickstolee commented Aug 14, 2019 •
edited
Loading

Uses the code currently at microsoft/git#180. Covers the basics of #8.

  • The default scalar clone runs git sparse-checkout init so the working directory only has files at root.

  • Run git sparse-checkout add <folders.txt to pipe in a list of folders, and it will expand those files.

Example workflow:

$ scalar clone https://dev.azure.com/gvfs/ci/_git/ForTests
Clone parameters:
  Repo URL:     https://dev.azure.com/gvfs/ci/_git/ForTests
  Branch:       Default
  Cache Server: Default
  Local Cache:  C:\.scalarCache
  Destination:  C:\_git\test2\ForTests
Authenticating...Succeeded
Querying remote for config...Succeeded
Using cache server: None (https://dev.azure.com/gvfs/ci/_git/ForTests)

WARNING: Unable to validate your Scalar version
Server not configured to provide supported Scalar versions

Cloning...Succeeded
Fetching commits and trees from origin (no cache server)...Succeeded
Validating repo...Succeeded
Mounting...Succeeded

$ cd ForTests/src/
$ ls
AuthoringTests.md  GvFlt_EULA.md  GVFS.sln  License.md  nuget.config  Protocol.md  Readme.md  Settings.StyleCop

$ echo GVFS/GVFS.Common >>../folders.txt
$ echo GVFS/GVFS.UnitTests >>../folders.txt
$ echo GitHooksLoader >>../folders.txt
$ git sparse-checkout add <../folders.txt

$ ls
AuthoringTests.md  GitHooksLoader/  GvFlt_EULA.md  GVFS/  GVFS.sln  License.md  nuget.config  Protocol.md  Readme.md  Settings.StyleCop

$ echo GVFS/GVFS >>../folders2.txt
$ echo GVFS/GVFS.FunctionalTests >>../folders2.txt
$ git sparse-checkout add <../folders2.txt

$ ls
AuthoringTests.md  GitHooksLoader/  GvFlt_EULA.md  GVFS/  GVFS.sln  License.md  nuget.config  Protocol.md  Readme.md  Settings.StyleCop

$ ls GVFS
GVFS/  GVFS.Common/  GVFS.FunctionalTests/  GVFS.UnitTests/  LibGit2Sharp.NativeBinaries.props  ProjectedFSLib.NativeBinaries.props

$ git sparse-checkout list
/
/GVFS/
/GVFS/GVFS/*
/GVFS/GVFS.Common/*
/GVFS/GVFS.FunctionalTests/*
/GVFS/GVFS.UnitTests/*
/GitHooksLoader/*

I spun up a few remaining issues for follow-up work: #76, #77, #78.

derrickstolee
derrickstolee commented Aug 14, 2019
View reviewed changes
@derrickstolee derrickstolee mentioned this pull request Aug 16, 2019
Closed
@github-actions
Copy link

github-actions bot commented Aug 16, 2019

Labeling this pull request as stale. There has been no activity for 30 days. Remove stale label or comment or this PR will be closed in 7 days.

@github-actions github-actions bot added the no-activity Stale issue or PR that will be automatically closed soon. label Aug 16, 2019
@derrickstolee
Copy link
Contributor Author

derrickstolee commented Aug 16, 2019

Labeling this pull request as stale. There has been no activity for 30 days. Remove stale label or comment or this PR will be closed in 7 days.

@jrbriggs is this a bug in the YAML, or a bug in Actions after enabling the task?

@derrickstolee derrickstolee added performance and removed no-activity Stale issue or PR that will be automatically closed soon. labels Aug 16, 2019
@github-actions
Copy link

github-actions bot commented Aug 17, 2019

Labeling this pull request as stale. There has been no activity for 30 days. Remove stale label or comment or this PR will be closed in 7 days.

@github-actions github-actions bot added the no-activity Stale issue or PR that will be automatically closed soon. label Aug 17, 2019
@jrbriggs jrbriggs removed the no-activity Stale issue or PR that will be automatically closed soon. label Aug 17, 2019
@jrbriggs
Copy link
Member

jrbriggs commented Aug 17, 2019

| @jrbriggs is this a bug in the YAML, or a bug in Actions after enabling the task?

@derrickstolee Happened again today. About 30 hours (hmmm.....) after you cleaned it up last. It was actually a bug in the action implementation. Fix in actions/stale#8

@github-actions
Copy link

github-actions bot commented Aug 19, 2019

Labeling this pull request as stale. There has been no activity for 30 days. Remove stale label or comment or this PR will be closed in 7 days.

@github-actions github-actions bot added the no-activity Stale issue or PR that will be automatically closed soon. label Aug 19, 2019
@jrbriggs jrbriggs removed the no-activity Stale issue or PR that will be automatically closed soon. label Aug 19, 2019
@derrickstolee derrickstolee added this to the Demo milestone Aug 20, 2019
@derrickstolee derrickstolee changed the title [WIP] Initial sparse implementation [PR BUILD] Initial sparse implementation [PR BUILD] Aug 20, 2019
@derrickstolee derrickstolee changed the title Initial sparse implementation [PR BUILD] [WIP] Initial sparse implementation [PR BUILD] Aug 20, 2019
@derrickstolee derrickstolee changed the title [WIP] Initial sparse implementation [PR BUILD] Initial sparse implementation [PR BUILD] Aug 20, 2019
@derrickstolee derrickstolee marked this pull request as ready for review August 20, 2019 18:35
@derrickstolee derrickstolee requested review from jrbriggs and wilbaker and removed request for wilbaker August 20, 2019 18:36
@derrickstolee
Copy link
Contributor Author

derrickstolee commented Aug 21, 2019

After rebasing onto a recent master and cloning a big repo, I noticed that I need to prevent the "prefetch everything" code (which wasn't in the previous base). Will fix.

@derrickstolee derrickstolee mentioned this pull request Aug 21, 2019
Merged
derrickstolee added a commit that referenced this pull request Aug 21, 2019
@derrickstolee
Merge pull request #80: CloneVerb: remove --no-mount option
536ae42 
Long term, we will not have a mount, so this feature will
eventually be meaningless. In the meantime, it is only causing
possible problems with hydrating files.

Resolves #26 permanently. In a normal clone, we already resolved
it in #56 by prefetching all files on clone. However, we still
dynamically prefetch on a sparse clone (#54).
@derrickstolee
Initial sparse implementation
ecb213e 
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
@derrickstolee
fix parameter ordering
f27bf46 
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
@derrickstolee
Remove some tests that check prefetch --hydrate
3b8d549 
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
@derrickstolee
Fix the sparse option for 'scalar clone'
b35db24 
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
@derrickstolee derrickstolee changed the title Initial sparse implementation [PR BUILD] Initial sparse implementation Aug 22, 2019
@derrickstolee derrickstolee mentioned this pull request Aug 22, 2019
Merged
2 tasks
jrbriggs
jrbriggs approved these changes Aug 22, 2019
View reviewed changes
Copy link
Member

@jrbriggs jrbriggs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved with suggestions.

jrbriggs
jrbriggs reviewed Aug 22, 2019
View reviewed changes
wilbaker
wilbaker approved these changes Aug 22, 2019
View reviewed changes
@derrickstolee
Clone: convert --sparse=false to --full-clone
f418dec 
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
@derrickstolee
PrefetchVerbTests: update test numbers
8631c3c 
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
@derrickstolee derrickstolee mentioned this pull request Aug 22, 2019
Closed
@derrickstolee derrickstolee merged commit 19d745f into microsoft:master Aug 22, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

2 more reviewers

@jrbriggs jrbriggs jrbriggs approved these changes

@wilbaker wilbaker wilbaker approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

performance

Projects

None yet

Milestone

Demo

Development

Successfully merging this pull request may close these issues.

3 participants

@derrickstolee @jrbriggs @wilbaker