Skip to content

Solve Job arrays exceeding queuesize#6314

Merged
bentsherman merged 8 commits intomasterfrom
5920-job-array-with-queuesize-exceeds-queuesize
Jul 28, 2025
Merged

Solve Job arrays exceeding queuesize#6314
bentsherman merged 8 commits intomasterfrom
5920-job-array-with-queuesize-exceeds-queuesize

Conversation

@jorgee
Copy link
Contributor

@jorgee jorgee commented Jul 28, 2025

close #5920

Modifying canSubmit to avoid to avoid exceed executor.queueSize parameter.
Printing warning when array size exceeds the queueSize. This can make the task array 'unsubmitable' (Not sure if we should abort the execution in this situation)
Add unit tests
Tested with this pipeline

process test {
    array 10 
    input:
    val x

    """
    echo $x
    """
}

workflow {
   Channel.of(1..20) | test
}

With a config with awsbatch executor and different executor.queueSize values:

size 15: Only one job submitted at a time in AWS Batch.
size 5: Prints a warning but run continues without the task array jobs submitted.

jorgee and others added 7 commits May 7, 2025 12:32
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
@netlify
Copy link

netlify bot commented Jul 28, 2025

Deploy Preview for nextflow-docs-staging canceled.

Name Link
🔨 Latest commit a02aa25
🔍 Latest deploy log https://app.netlify.com/projects/nextflow-docs-staging/deploys/6887895fd6c20400082bd38d

@bentsherman bentsherman self-requested a review July 28, 2025 14:31
@bentsherman bentsherman merged commit fcf5b87 into master Jul 28, 2025
22 checks passed
@bentsherman bentsherman deleted the 5920-job-array-with-queuesize-exceeds-queuesize branch July 28, 2025 14:38
pditommaso added a commit that referenced this pull request Aug 14, 2025
@pditommaso
Copy link
Member

I went for a simpler solution for this problem #6345

@bentsherman
Copy link
Member

Your PR does not update the forks count correctly. This PR should be restored

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Job Array with queueSize exceeds queueSize

3 participants