Implementation of Git multiple revisions by jorgee · Pull Request #6620 · nextflow-io/nextflow

jorgee · 2025-12-01T10:24:16Z

Summary

This PR implements Git multi-revision asset management, enabling concurrent execution of different pipeline revisions (e.g., production on v1.0, testing on v2.0-dev) while maintaining efficient disk usage through object sharing.

Closes #2870. Alternative for #5089.

Solution

The implementation uses a bare repository + shared clones approach with the Strategy pattern for backward compatibility:

~/.nextflow/assets/.repos/<org>/<project>/
├── bare/                    # Bare repository (shared objects)
│   ├── objects/             # All Git objects stored here
│   ├── refs/
│   └── config
│
└── clones/                  # Revision-specific clones
    ├── <commit-sha-1>/      # Clone using JGit alternates
    └── <commit-sha-2>/      # (points to bare/objects)

Key mechanisms:

Bare repository as source of truth: Fetched/updated from remote, keeping refs current
Dynamic resolution: Revisions resolved to commit SHAs using bare repo refs
Object sharing: Clones use JGit alternates to reference bare repo objects, avoiding duplication
Atomic operations: Each clone is independent; downloading new revisions doesn't affect running pipelines
Lazy creation: Clones created on-demand when a specific revision is requested

Backward Compatibility

Uses the Strategy pattern to maintain compatibility with existing repositories:

RepositoryStrategy interface with LegacyRepositoryStrategy and MultiRevisionRepositoryStrategy implementations
AssetManager acts as facade with unchanged public API
Strategy selection: NXF_SCM_LEGACY=true forces legacy mode, otherwise auto-detected based on repository state
Existing legacy repositories continue to work without migration

Changes

Core Implementation

Added RepositoryStrategy interface and AbstractRepositoryStrategy base class
Implemented MultiRevisionRepositoryStrategy for bare repo + shared clones
Implemented LegacyRepositoryStrategy preserving original behavior
Refactored AssetManager to delegate to strategy implementations

CLI Updates

Changed -a, --all-revisions flag to -a, --all in CmdDrop
Updated nf-info command output to show clones with revision indicators

Concurrency & Reliability

Added FileMutex for safe concurrent clone operations
Improved error handling with cleanup of incomplete clones

Documentation

ADR: adr/20251205-git-multi-revision-asset-management.md

Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>

… operation Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>

…f "master" Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>

Signed-off-by: Ben Sherman <bentshermann@gmail.com>

Signed-off-by: Dr Marco De La Pierre <marco.delapierre@gmail.com>

Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>

docs/reference/cli.md

modules/nextflow/src/main/groovy/nextflow/cli/CmdConfig.groovy

modules/nextflow/src/main/groovy/nextflow/cli/CmdPull.groovy

modules/nextflow/src/main/groovy/nextflow/cli/CmdView.groovy

modules/nextflow/src/main/groovy/nextflow/scm/GithubRepositoryProvider.groovy

pditommaso · 2025-12-15T09:23:23Z

Little glitch, but likely there isn't much it can be done here

» nextflow list
nextflow-io/rnaseq-nf
nf-core/modules
pditommaso/hello
.repos/nextflow-io

jorgee · 2025-12-15T09:53:57Z

Little glitch, but likely there isn't much it can be done here
» nextflow list
nextflow-io/rnaseq-nf
nf-core/modules
pditommaso/hello
.repos/nextflow-io

Is this when runninglistwith the previous version?

pditommaso · 2025-12-15T09:55:28Z

Yes

Co-authored-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> Signed-off-by: Jorge Ejarque <jorgee@users.noreply.github.com>

jorgee · 2025-12-15T10:02:06Z

I can create another PR with a patch ignoring the path to backport to stable releases.

…) [ci skip] Update the multi-revision asset management ADR: - Fix directory structure to match implementation: - bare/ instead of .nextflow/bare_repo/ - commits/ instead of .nextflow/commits/ - Legacy repo at separate location ~/.nextflow/assets/<project>/ - Merge Option 4 (Strategy Pattern) into Option 3 since the Strategy Pattern is an implementation detail of the multi-revision approach, not a standalone option Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

Signed-off-by: jorgee <jorge.ejarque@seqera.io>

adr/20251205-multi-revision-asset-management-strategy-pattern.md

docs/cli.md

Co-authored-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> Signed-off-by: Jorge Ejarque <jorgee@users.noreply.github.com>

Signed-off-by: jorgee <jorge.ejarque@seqera.io>

jorgee · 2025-12-17T12:39:57Z

Addressed the last review comments.
I have also double check the other two issues linked to the PR, and they do not happen when using multi-revision. Still happening when using legacy strategy. Users must migrate to fix these issues.

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

pditommaso · 2025-12-17T13:18:59Z

Just replaced the • with > because i'm quite not all terminal can support it

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

Signed-off-by: jorgee <jorge.ejarque@seqera.io>

jorgee · 2025-12-17T14:12:27Z

Changing the symbol in docs

marcodelapierre and others added 30 commits January 15, 2024 17:29

Multiple revisions 1st: AssetManager, Cmd Clone Pull Run

7d027ec

Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>

removed revision paramter from clone pull run

106e8df

Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>

updated unit tests for AssetManager

0a9e6f8

Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>

Merge branch 'master' into add/mult_revisions

37d332e

AssetManager provider.revision assigned at AssetManager object creation

5a9b159

Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>

Codespell typo in changelog

94d34f3

Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>

AssetManager: removed checkout method

4744774

Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>

Merge branch 'master' into add/mult_revisions

d9e7b2b

assetmanager: fixed build signature, and unit tests

08718ef

Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>

assetmanager: one more build signature fix

242a600

Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>

AssetManager: fix for multi revs in find() method

35ad1e9

Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>

Multiple revisions: added for Cmds Drop, View, Config, Info

463b6f1

Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>

minor fixes to log outputs in CmdRun

2d3273e

Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>

AssetManager: documented new localPath schema

d2e5ffe

Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>

K8sDriverLauncher : added revision support

60c3c5e

Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>

updates to AssetManagerTest

4bf1e37

Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>

edit to AssetManagerTest, git.pull for TAGS does not output result of…

7e32111

… operation Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>

minor fix in GitlabRepositoryProvider: using DEFAULT_BRANCH instead o…

3b2d461

…f "master" Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>

Minor edits

2e6461e

Signed-off-by: Ben Sherman <bentshermann@gmail.com>

Merge branch 'master' into add/mult_revisions

a8d2dee

[ci fast] Merge branch 'master' into add/mult_revisions

eaf45d8

Merge branch 'master' into add/mult_revisions

ed17284

parametrised revision delimiter

b3f141d

Signed-off-by: Dr Marco De La Pierre <marco.delapierre@gmail.com>

nf pull: option to list or not revs for each project

6706bd9

Signed-off-by: Dr Marco De La Pierre <marco.delapierre@gmail.com>

nicer output for nf list -r

e62f543

Signed-off-by: Dr Marco De La Pierre <marco.delapierre@gmail.com>

minor edit to CmdList

bd19877

Signed-off-by: Dr Marco De La Pierre <marco.delapierre@gmail.com>

AssetManager: adding listRevisions method (work in progress)

c0268af

Signed-off-by: Dr Marco De La Pierre <marco.delapierre@gmail.com>

small tune to CmdList

aae2afc

Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>

minor tweak to CmdList

268ec79

Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>

AssetManager: listRevisions method now working

ac8d755

Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>

pditommaso requested changes Dec 15, 2025

View reviewed changes

Update docs/reference/cli.md

7fcf3c6

Co-authored-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> Signed-off-by: Jorge Ejarque <jorgee@users.noreply.github.com>

pditommaso and others added 2 commits December 15, 2025 11:27

address review comments

f30022e

Signed-off-by: jorgee <jorge.ejarque@seqera.io>

pditommaso reviewed Dec 15, 2025

View reviewed changes

adr/20251205-multi-revision-asset-management-strategy-pattern.md Outdated Show resolved Hide resolved

pditommaso changed the title ~~Implementation of multiple revisions without revisions map~~ Implementation of Git multiple revisions Dec 15, 2025

pditommaso reviewed Dec 15, 2025

View reviewed changes

jorgee and others added 8 commits December 15, 2025 17:09

Update adr/20251205-multi-revision-asset-management-strategy-pattern.md

27a6acf

Co-authored-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> Signed-off-by: Jorge Ejarque <jorgee@users.noreply.github.com>

Apply suggestions from code review [ci fast]

575dd40

Co-authored-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> Signed-off-by: Jorge Ejarque <jorgee@users.noreply.github.com>

Remove unnecesary RepositoryStatus

d48a798

Signed-off-by: jorgee <jorge.ejarque@seqera.io>

fix test

509e70e

Signed-off-by: jorgee <jorge.ejarque@seqera.io>

Merge branch 'master' into mult_revisions_no_map

f37b17c

change commits dir by revs and '*' by '·' for checkouts

5e81412

Signed-off-by: jorgee <jorge.ejarque@seqera.io>

fix info command in docs

3fe441a

Signed-off-by: jorgee <jorge.ejarque@seqera.io>

fix test and change revs to clones

b517c10

Signed-off-by: jorgee <jorge.ejarque@seqera.io>

jorgee requested a review from pditommaso December 17, 2025 12:40

Minor change [ci fast]

2883bc7

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

pditommaso approved these changes Dec 17, 2025

View reviewed changes

pditommaso and others added 2 commits December 17, 2025 14:26

Rename adr filename [ci skip]

df8731c

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

Change '·' by '>' in docs [ci fast]

9adaa63

Signed-off-by: jorgee <jorge.ejarque@seqera.io>

bentsherman merged commit ce9d7b5 into master Dec 17, 2025
12 checks passed

bentsherman deleted the mult_revisions_no_map branch December 17, 2025 18:00

bentsherman mentioned this pull request Dec 17, 2025

Set a default revision #4427

Open

Conversation

jorgee commented Dec 1, 2025 • edited by pditommaso Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Solution

Backward Compatibility

Changes

Core Implementation

CLI Updates

Concurrency & Reliability

Documentation

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pditommaso commented Dec 15, 2025

Uh oh!

jorgee commented Dec 15, 2025

Uh oh!

pditommaso commented Dec 15, 2025

Uh oh!

jorgee commented Dec 15, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jorgee commented Dec 17, 2025

Uh oh!

pditommaso commented Dec 17, 2025

Uh oh!

jorgee commented Dec 17, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jorgee commented Dec 1, 2025 •

edited by pditommaso

Loading