-
Notifications
You must be signed in to change notification settings - Fork 776
Description
Bug report
Expected behavior and actual behavior
Using the storeDir directive in combination with the rsync stageOutMode is unable to create a valid storage directory which causes the pipeline to fail. Everything works fine when using the directive with a copy stageOutMode
Expected behaviour (copy with storeDir 'example'):
$ tree example/
example/
├── 1.txt
├── 2.txt
├── 3.txt
├── 4.txt
└── 5.txtActual behaviour (rsync with storeDir 'example'):
$ ll
...
-rw-rw-r-- 1 nvnieuwk nvnieuwk 42 Jun 25 10:44 example
...A file has been created with the name of the storeDir instead of a directory containing the correct files
Steps to reproduce the problem
main.nf:
workflow {
def numbers = Channel.of(1, 2, 3, 4, 5)
EXAMPLE(numbers)
}
process EXAMPLE {
storeDir 'example'
tag "$number"
input:
val number
output:
path('*.txt')
script:
"""
echo "This is an example process with number: ${number}" > ${number}.txt
"""
}nextflow.config:
params.mode = "rsync"
process.stageOutMode = params.modeThe issue can be replicated by running the following command in a directory with the files presented above:
rm -rf example && nextflow run main.nf --mode rsyncNote that this method does work when the storeDir is already filled with the correct files, that's why you should remove it every time you run this command
The expected behaviour can be replicated by running the following command
rm -rf example && nextflow run main.nf --mode copyProgram output
ERROR ~ Error executing process > 'EXAMPLE (3)'
Caused by:
Missing output file(s) `*.txt` expected by process `EXAMPLE (3)`
Command executed:
echo "This is an example process with number: 3" > 3.txt
Command exit status:
0
Command output:
(empty)
Work dir:
/home/nvnieuwk/Documents/nextflow/issues/rsync-storeDir/work/93/2feb18d4f824a9d9c247c773b31e15
Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line
-- Check '.nextflow.log' file for details
Environment
- Nextflow version: 25.04.2 (but also noticed in older versions)
- Java version: 17.0.3
- Operating system: Linux
- Bash version: 5.0.17