Skip to content

Enable multiple revisions for project repos (first attempt 2024)#4659

Closed
marcodelapierre wants to merge 75 commits intomasterfrom
add/mult_revisions
Closed

Enable multiple revisions for project repos (first attempt 2024)#4659
marcodelapierre wants to merge 75 commits intomasterfrom
add/mult_revisions

Conversation

@marcodelapierre
Copy link
Contributor

@marcodelapierre marcodelapierre commented Jan 15, 2024 •
edited
Loading

This PR adds support for handling local copies of multiple revisions of the same pipeline.

Key points:

  • Under NXF_ASSETS, each pipeline is now pulled as <org>/<repo>[:<revision>] ; :<revision> is only appended if the corresponding flag was used on CLI, -r/--revision
  • The key implementation change is adding the revision attribute to the AssetManager class, as both pipeline name and revision are now required to fully identify a pipeline
  • Support has been added and tested for all relevant CLI commands: run, pull, clone, drop, list, view, config, info, inspect, kuberun
  • Unit tests for AssetManagerTest have been updated

Caveats:

  • Jgit does not implement git worktree, so the original idea within Allow the concurrent run of multiple pipeline revisions #2870 could not be applied
  • depth = 1 (shallow clones) was investigated to reduce disk usage, but could not be implemented as it would not allow to checkout branches/tags/commits
  • In the current implementation, pulling without --revision and with --revision <default branch> create two duplicate pulls; this is not optimal, but with very limited known negative impact; [update] this only happens if the default branch is not declared in the manifest and differs from master
  • As a by-product, if a repo has a default branch different from master, pulling and running it is now possible without specifying the branch name explicitly

Closes #2870 .
Also indirectly addresses #3593

@marcodelapierre
Multiple revisions 1st: AssetManager, Cmd Clone Pull Run
7d027ec 
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@netlify
Copy link

netlify bot commented Jan 15, 2024 •
edited
Loading

✅ Deploy Preview for nextflow-docs-staging canceled.

Name Link
🔨 Latest commit 61bd4bf
🔍 Latest deploy log https://app.netlify.com/sites/nextflow-docs-staging/deploys/666ff7b4650e42000878ae7a

@marcodelapierre marcodelapierre self-assigned this Jan 15, 2024
@marcodelapierre marcodelapierre changed the title Multiple revisions 1st: AssetManager, Cmd Clone Pull Run Enable multiple revisions for SCMs Jan 15, 2024
@marcodelapierre marcodelapierre marked this pull request as draft January 15, 2024 12:39
@marcodelapierre marcodelapierre changed the title Enable multiple revisions for SCMs Enable multiple revisions for project repos Jan 15, 2024
@bentsherman
Copy link
Member

bentsherman commented Jan 15, 2024

You could save disk space by cloning with depth = 0 when a revision is specified

@marcodelapierre
Copy link
Contributor Author

marcodelapierre commented Jan 16, 2024 •
edited
Loading

You could save disk space by cloning with depth = 0 when a revision is specified

Thanks @bentsherman , great idea, will do, for nextflow pull but not for clone. Why not for any case, including no revision specified? - I just have to check that a subsequent nextflow pull still works (for the purpose of updating the local copy).

@marcodelapierre
removed revision paramter from clone pull run
106e8df 
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@marcodelapierre
updated unit tests for AssetManager
0a9e6f8 
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@marcodelapierre
Merge branch 'master' into add/mult_revisions
37d332e
@marcodelapierre
AssetManager provider.revision assigned at AssetManager object creation
5a9b159 
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@marcodelapierre
Codespell typo in changelog
94d34f3 
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@marcodelapierre
AssetManager: removed checkout method
4744774 
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@bentsherman
Copy link
Member

bentsherman commented Jan 16, 2024

yes I suppose we could just clone with depth = 0 in all cases, since no revision just defaults to master. the clone, pull, and run commands all provide a -deep option for this, so users can change it if they want

@marcodelapierre
Merge branch 'master' into add/mult_revisions
d9e7b2b
@marcodelapierre
assetmanager: fixed build signature, and unit tests
08718ef 
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@marcodelapierre
assetmanager: one more build signature fix
242a600 
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@marcodelapierre
Copy link
Contributor Author

marcodelapierre commented Jan 17, 2024

Ah! unfortunately we cannot implement the deep functionality, as with a depth of 1 we cannot checkout to other branches/tags/commits.
This would be supported : git clone -b <branch/tag> --deep <deep> <repo> , but only works for branches and tags, not commits. Git still does not support a way to clone to a specific commit straight away.

@marcodelapierre
AssetManager: fix for multi revs in find() method
35ad1e9 
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@marcodelapierre
Multiple revisions: added for Cmds Drop, View, Config, Info
463b6f1 
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@marcodelapierre
minor fixes to log outputs in CmdRun
2d3273e 
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@marcodelapierre
AssetManager: documented new localPath schema
d2e5ffe 
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@marcodelapierre
K8sDriverLauncher : added revision support
60c3c5e 
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@marcodelapierre
updates to AssetManagerTest
4bf1e37 
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@marcodelapierre
edit to AssetManagerTest, git.pull for TAGS does not output result of…
7e32111 
… operation

Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@marcodelapierre
minor fix in GitlabRepositoryProvider: using DEFAULT_BRANCH instead o…
3b2d461 
…f "master"

Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@marcodelapierre marcodelapierre marked this pull request as ready for review January 18, 2024 08:11
@marcodelapierre
Copy link
Contributor Author

marcodelapierre commented May 27, 2024

On point 2. above, wondering whether we should leverage user-provided information more, when it comes to printing out information on repository in use. I.e.:

  • commit ID as de-referenced from user request
  • (if not null) revision as requested by the user
  • (eventually, also infer branch)

@marcodelapierre
fix to updateLocalBareRepo
3b44f9f 
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@marcodelapierre
Revert "fix to updateLocalBareRepo"
d69f542 
This reverts commit 3b44f9f.
@marcodelapierre
assetmanager comments
0cb331a 
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@marcodelapierre
Copy link
Contributor Author

marcodelapierre commented May 29, 2024

As dicussed in meeting, on point 1., let me re-implement with the original class signature for AssetManager - no revision attribute. Commit ID and hence local path will be set when the revision is set.

Note on available code so far:

  1. current branch for this PR is add/mult_revisions - impl with local bare repo, and commit id in local path
  2. branch add/mult_revisions_bkp15apr - first impl, uses revision in local path
  3. branch add/mult_revisions_bkp29may - backup of 1.

@marcodelapierre
Merge branch 'master' into add/mult_revisions
042e99f 
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@marcodelapierre
updated nextflow.scm.mmd
125959f
@marcodelapierre
updated docs/sharing
5bdb2b3 
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@marcodelapierre
Merge branch 'master' into add/mult_revisions
61bd4bf 
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@marcodelapierre marcodelapierre changed the title Enable multiple revisions for project repos Enable multiple revisions for project repos (first attempt 2024) Jun 24, 2024
@marcodelapierre marcodelapierre mentioned this pull request Jun 24, 2024
Closed
@marcodelapierre
Copy link
Contributor Author

marcodelapierre commented Jun 24, 2024 •
edited
Loading

Going forward, let's manage these through 2 separate new PRs:

  1. Enable multiple git revisions (Jan-Apr 2024) : temporary, using revision name in repo localPath #5087 : Jan-April implementation, using revision name in repo localPath
  2. Enable multiple git revisions (Jun 2024) : final, using commit ID in repo localPath #5089 : June implementation, using commit ID in repo localPath

The first one is for the record and for reference, unlikely to go forward.

The second one is that to be considered for revision and merging.

@marcodelapierre marcodelapierre mentioned this pull request Jun 25, 2024
Closed
@ewels ewels mentioned this pull request Sep 26, 2024
Closed
This was linked to issues Oct 2, 2024
Project currently is stuck on revision xxx when using -latest flag #1670
Closed
nextflow pull: unexpected behaviour after using -r #1334
Closed
@pditommaso pditommaso force-pushed the master branch 2 times, most recently from 5a93547 to 27345a6 Compare February 10, 2025 21:46
@pditommaso pditommaso force-pushed the master branch 3 times, most recently from b4b321e to 069653d Compare June 4, 2025 18:54
@bentsherman
Copy link
Member

bentsherman commented Dec 1, 2025

Closing in favor of #6620

@bentsherman bentsherman closed this Dec 1, 2025
@bentsherman bentsherman deleted the add/mult_revisions branch December 1, 2025 15:13
This was unlinked from issues Dec 3, 2025
Project currently is stuck on revision xxx when using -latest flag #1670
Closed
nextflow pull: unexpected behaviour after using -r #1334
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pditommaso pditommaso pditommaso left review comments

@bentsherman bentsherman Awaiting requested review from bentsherman

Assignees

No one assigned

Labels

software/git

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Main branch not recognised when executing an online workflow Allow the concurrent run of multiple pipeline revisions

3 participants

@marcodelapierre @bentsherman @pditommaso