Improving performance of dependency git pulling #2641
Description
Description
We are seeing some performance cost due to our complex circular repo dependency graph. This can be speed up by using a cached version of the git repo that is being pulled. Logs below should make it clear.
Feature request 1 - Cache which git repos branches have been pulled, se solution in section below.
Feature request 2 - It would also be nice to skip downloading dependencies when the dependencies are filtered out by something like --dependency "-". Se below.
PS C:\Projects\NHN\HN-Configuration> devspace deploy --dependency "-" --debug
20:04:42 info Using namespace 'dev'
20:04:42 info Using kube context 'kind-hn-cluster'
20:04:43 debug Pulled https://nhnfelles@dev.azure.com/nhnfelles/Helsenorge/_git/HN-Oppsett
20:04:45 debug Pulled https://nhnfelles@dev.azure.com/nhnfelles/Helsenorge/_git/HN-Felleslogg
20:04:46 debug Pulled https://nhnfelles@dev.azure.com/nhnfelles/Helsenorge/_git/HN-Oppsett
20:04:47 debug Pulled https://nhnfelles@dev.azure.com/nhnfelles/Helsenorge/_git/HN-Oppsett
20:04:49 debug Pulled https://nhnfelles@dev.azure.com/nhnfelles/Helsenorge/_git/HN-Oppsett
20:04:52 debug Pulled https://nhnfelles@dev.azure.com/nhnfelles/Helsenorge/_git/HN-STS
20:04:53 debug Pulled https://nhnfelles@dev.azure.com/nhnfelles/Helsenorge/_git/HN-Oppsett
20:04:55 debug Pulled https://nhnfelles@dev.azure.com/nhnfelles/Helsenorge/_git/HN-Configuration
20:04:55 debug Cyclic dependency found:
hn-sts
hn-configuration
hn-sts
20:04:56 debug Pulled https://nhnfelles@dev.azure.com/nhnfelles/Helsenorge/_git/HN-Felleslogg
20:04:57 debug Pulled https://nhnfelles@dev.azure.com/nhnfelles/Helsenorge/_git/HN-Oppsett
20:05:01 debug Pulled https://nhnfelles@dev.azure.com/nhnfelles/Helsenorge/_git/HN-Personvern
20:05:02 debug Pulled https://nhnfelles@dev.azure.com/nhnfelles/Helsenorge/_git/HN-Oppsett
20:05:04 debug Pulled https://nhnfelles@dev.azure.com/nhnfelles/Helsenorge/_git/HN-Configuration
20:05:04 debug Cyclic dependency found:
hn-personvern
hn-configuration
hn-sts
hn-personvern
20:05:05 debug Pulled https://nhnfelles@dev.azure.com/nhnfelles/Helsenorge/_git/HN-Oppsett
20:05:06 debug Pulled https://nhnfelles@dev.azure.com/nhnfelles/Helsenorge/_git/HN-STS
20:05:06 debug Cyclic dependency found:
hn-personvern
hn-sts
hn-personvern
20:05:06 debug Run pipeline:
Is your feature request related to a problem?
No.
Which solution do you suggest?
Within a single devspace deploy command you should store in memory a dictionary for each unique pair of repo url + branch name, which points to a dependency cache folder. This can then be reused deeper down the dependency hierarchy.
Which alternative solutions exist?
None