Dynamic resources with 1 or more input files #4207
Description
First, thanks for the awesome tool and all the hard work.
Bug report
Specifying dynamic resources over multiple inputs fails when a single input is passed. Note the issue occurs when inputs are passed in the same method (i.e. collected before being passed).
The goal is to set dynamic ram on a process that takes multiple inputs together (collected). Sometimes, only a single input is passed.
Expected behavior and actual behavior
It should succeed.
Steps to reproduce the problem
Nextflow config:
process {
container = 'cimg/aws:2023.08'
executor = 'awsbatch'
queue = '<my-queue>'
}
aws {
region = 'ca-central-1'
}
Script:
#!/usr/bin/env nextflow
nextflow.enable.dsl=2
process genFile {
cpus 1
memory '1 GB'
input:
val x
output:
path 'out.txt'
script:
"""
echo '$x' > out.txt
"""
}
process catFiles {
cpus 1
memory { inputs*.size().sum() < 1.GB ? 1.GB : 2.GB }
input:
path inputs, stageAs: 'in*.txt'
output:
path 'catted.txt'
script:
"""
cat $inputs > catted.txt
"""
}
workflow {
Channel.of(0) | genFile | collect | catFiles | view
}
Error:
bash-4.2# nextflow run main.nf -bucket-dir s3://<bucket>/nextflow
N E X T F L O W ~ version 23.04.1
Launching `main.nf` [golden_bohr] DSL2 - revision: 36677f67a9
executor > awsbatch (1)
[c0/673e46] process > genFile (1) [100%] 1 of 1 ✔
[- ] process > catFiles [ 0%] 0 of -1
ERROR ~ No such file or directory: in.txt
-- Check script 'main.nf' at line: 19 or see '.nextflow.log' file for more details
Note:
If we change the workflow to:
workflow {
Channel.of(0..1) | genFile | collect | catFiles | view
}
Then it actually works.
Commenting out memory { inputs*.size().sum() < 1.GB ? 1.GB : 2.GB } also causes it to work.
How can we dynamically compute ram when the input could be 1 or 2 files?
Program output
Environment
nextflow/nextflow:23.04.3 docker image
In an attempt to cover all bases, I also tried with 23.08.1-edge, same issue.
Additional context
Open to workarounds!