Tested with:

nextflow run -profile test,uppmax main.nf --outdir '/proj/snic2018-8-34/rnaseq-test' --project 'snic2018-8-34'

Directory tree is exactly the same as nextflow run -profile test,uppmax nf-core/rnaseq --outdir '/proj/snic2018-8-34/rnaseq-nf-core-test' --project 'snic2018-8-34'

Major changes to remember/check for rebasing:

• Move inclusion of modules until after institute configs are loaded.
• Update test.config with new way of passing tools args.
• module.config updated to use process selectors and not pass using params.
• Remove all configuration setting and passing in rnaseq.nf workflow and subworkflows too (addParams not needed in most cases)
• Remove options and publishDir from all module files.
• All modules should check if task.ext.args is initialized.
• Will need to change saveAs: case to check for versions.yml instead of *versions.txt.

Additional suggestion.

• Can move functions.nf to libs folder and all modules then load file from "$projectDir/libs/functions.nf". Only getProcessName and getSoftwareName functions needed. Technically getSoftwareName only really needed for generic case.

Ready for review.

Updates from 3.4 incorporated.
nextflow run -profile test,uppmax rnaseq/main.nf --outdir '/proj/snic2018-8-34/rnaseq-nf-core-test' --project 'snic2018-8-34' on branch use_publishDir produces the same output as
nextflow run -profile test,uppmax nf-core/rnaseq --outdir '/proj/snic2018-8-34/rnaseq-nf-core-test' --project 'snic2018-8-34'

nf-core lint overall result: Passed ✅ ⚠️

Posted for pipeline commit 0ad0307

+| ✅ 129 tests passed       |+
#| ❔   8 tests were ignored |#
!| ❗   1 tests had warnings |!

WARN: There's no process matching config selector: NFCORE_RNASEQ:RNASEQ:INPUT_CHECK:SAMPLESHEET_CHECK
WARN: There's no process matching config selector: .*:PREPARE_GENOME:BBMAP_BBSPLIT
WARN: There's no process matching config selector: .*:QUANTIFY_RSEM:RSEM_CALCULATEEXPRESSION
WARN: There's no process matching config selector: .*:QUANTIFY_RSEM:RSEM_MERGE_COUNTS
WARN: There's no process matching config selector: NFCORE_RNASEQ:RNASEQ:.*:BAM_SORT_SAMTOOLS:SAMTOOLS_SORT
<TRUNCATED>

Mannn all of these WARN messages have started to prop up now with the new syntax 🤦🏽

Related to nextflow-io/nextflow#2072

Lemme know what you think. I am feeling optimistic that we may even be able to start the process of converting modules/pipelines to this format during the Hackathon.

The only thing left we need to decide is whether to use the getProcessName and getSoftwareName in the modules code now I think

@mahesh-panchal are we able to dump the more "controversial" aspects of this implementation that have been raised here somewhere so we can maybe have more focussed discussions around those? e.g. this, this, this etc.

A TODO list of things we need to get sorted to implement it would be useful too (e.g. updating nf-core/modules, module template, pipeline template, lib/, linter etc). We can update this collectively as we go along. I am trying to assess whether it is something we can get done before the Hackathon (but in any case I think it would be useful either way I think).

I got an idea that we can keep passing opts using params too if we like:

main.nf:

nextflow.enable.dsl = 2

workflow {

    FOO(params.name)
}

process FOO {

    input:
    val name

    script:
    def message = task.ext.message ?: 'Hello'
    """
    echo $message $name
    """
}

nextflow.config:

params.name = 'J.A.R.V.I.S'
params.process = [:]

process.echo = true
process.'withName:FOO'.ext.message = 'Welcome'

if ( params.process ){
    params.process.each { selector ->
        process."withName:${selector.key}".ext.message = selector.value.message
    }
}

Example:

nextflow run main.nf --process.FOO.message 'Good day'

output:

N E X T F L O W  ~  version 21.04.0
Launching `main.nf` [big_poisson] - revision: ecbd9202f8
executor >  local (1)
[c0/ad0122] process > FOO [100%] 1 of 1 ✔
Good day J.A.R.V.I.S

Can someone check the json schema please for the params.process. Is the type correct ?

New bug. Not sure if it's my fault in configuration.

Error executing process > 'NFCORE_RNASEQ:RNASEQ:BBMAP_BBSPLIT (1)'

Caused by:
  Process `NFCORE_RNASEQ:RNASEQ:BBMAP_BBSPLIT (1)` terminated with an error exit status (1)

Command executed:

  bbsplit.sh \
      -Xmx6g \
      path=bbsplit \
      threads=2 \
      in=RAP1_IAA_30M_REP1_1_val_1.fq.gz in2=RAP1_IAA_30M_REP1_2_val_2.fq.gz \
      basename=RAP1_IAA_30M_REP1_%_#.fastq.gz \
      refstats=RAP1_IAA_30M_REP1.stats.txt \
      build=1 ambiguous2=all maxindel=150000
  
  cat <<-END_VERSIONS > versions.yml
  BBMAP_BBSPLIT:
      bbmap: $(bbversion.sh 2>&1)
  END_VERSIONS

Command exit status:
  1
Command output:
  (empty)

Command error:
  java -ea -Xmx6g -Xms6g -cp /usr/local/opt/bbmap-38.93-0/current/ align2.BBSplitter ow=t fastareadlen=500 minhits=1 minratio=0.56 maxindel=20 qtrim=rl untrim=t trimq=6 -Xmx6g path=bbsplit threads=2 in=RAP1_IAA_30M_REP1_1_val_1.fq.gz in2=RAP1_IAA_30M_REP1_2_val_2.fq.gz basename=RAP1_IAA_30M_REP1_%_#.fastq.gz refstats=RAP1_IAA_30M_REP1.stats.txt build=1 ambiguous2=all maxindel=150000
  Executing align2.BBSplitter [ow=t, fastareadlen=500, minhits=1, minratio=0.56, maxindel=20, qtrim=rl, untrim=t, trimq=6, -Xmx6g, path=bbsplit, threads=2, in=RAP1_IAA_30M_REP1_1_val_1.fq.gz, in2=RAP1_IAA_30M_REP1_2_val_2.fq.gz, basename=RAP1_IAA_30M_REP1_%_#.fastq.gz, refstats=RAP1_IAA_30M_REP1.stats.txt, build=1, ambiguous2=all, maxindel=150000]
  
  Merged reference file bbsplit/ref/genome/1/merged_ref_8374379829187813017.fa.gz already exists; skipping merge.
  Ref merge time:       0.191 seconds.
  Executing align2.BBMap [ow=t, fastareadlen=500, minhits=1, minratio=0.56, maxindel=20, qtrim=rl, untrim=t, trimq=6, threads=2, in=RAP1_IAA_30M_REP1_1_val_1.fq.gz, in2=RAP1_IAA_30M_REP1_2_val_2.fq.gz, refstats=RAP1_IAA_30M_REP1.stats.txt, build=1, ambiguous2=all, maxindel=150000, ref=bbsplit/ref/genome/1/merged_ref_8374379829187813017.fa.gz, out_primary=RAP1_IAA_30M_REP1_primary_#.fastq.gz, out_sarscov2=RAP1_IAA_30M_REP1_sarscov2_#.fastq.gz, out_human=RAP1_IAA_30M_REP1_human_#.fastq.gz]
  Version 38.93
  
  Set MINIMUM_ALIGNMENT_SCORE_RATIO to 0.560
  Set threads to 2
  Reference set statistics will be written to RAP1_IAA_30M_REP1.stats.txt
  Retaining first best site only for ambiguous mappings.
  NOTE: Ignoring reference file because it already appears to have been processed.
  NOTE: If you wish to regenerate the index, please manually delete bbsplit/ref/genome/1/summary.txt
  Set genome to 1
  
  Loaded Reference:     0.126 seconds.
  Loading index for chunk 1-1, build 1
  Exception in thread "Thread-0" java.lang.RuntimeException: java.util.zip.ZipException: too many length or distance symbols
        at fileIO.ReadWrite.readObject(ReadWrite.java:823)
        at fileIO.ReadWrite.read(ReadWrite.java:1282)
        at align2.Block.read(Block.java:132)
        at align2.IndexMaker4$BlockMaker.makeArrays(IndexMaker4.java:140)
        at align2.IndexMaker4$BlockMaker.run(IndexMaker4.java:128)
  Caused by: java.util.zip.ZipException: too many length or distance symbols
        at java.base/java.util.zip.InflaterInputStream.read(InflaterInputStream.java:165)
        at java.base/java.util.zip.GZIPInputStream.read(GZIPInputStream.java:118)
        at java.base/java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2854)
        at java.base/java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2870)
        at java.base/java.io.ObjectInputStream$BlockDataInputStream.readInts(ObjectInputStream.java:3523)
        at java.base/java.io.ObjectInputStream.readArray(ObjectInputStream.java:2081)
        at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
        at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:493)
        at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:451)
        at fileIO.ReadWrite.readObject(ReadWrite.java:819)
        ... 4 more
  Generated Index:      0.642 seconds.
  Exception in thread "main" java.lang.NullPointerException
        at align2.BBIndex.analyzeIndex(BBIndex.java:115)
        at align2.BBMap.loadIndex(BBMap.java:420)
        at align2.BBMap.main(BBMap.java:32)
        at align2.BBSplitter.main(BBSplitter.java:48)

Run with:

#! /usr/bin/env nextflow

NXF_VER=21.10.0-SNAPSHOT \
nextflow run rnaseq/main.nf \
    --outdir "$PWD/pubdir_results" \
    --project 'snic2018-8-34' \
    -profile test,uppmax \
    -resume \
    -w "$PWD/work"

Test profile still throws these warnings with the fix in 21.10.0-SNAPSHOT

WARN: There's no process matching config selector: .*:QUANTIFY_RSEM:RSEM_CALCULATEEXPRESSION
WARN: There's no process matching config selector: .*:QUANTIFY_RSEM:RSEM_MERGE_COUNTS
WARN: There's no process matching config selector: .*:QUANTIFY_STAR_SALMON:SALMON_MERGE_COUNTS
WARN: There's no process matching config selector: .*:QUANTIFY_SALMON:SALMON_MERGE_COUNTS
WARN: There's no process matching config selector: NFCORE_RNASEQ:RNASEQ:SAMTOOLS_SORT
WARN: There's no process matching config selector: .*:FASTQC_UMITOOLS_TRIMGALORE:UMITOOLS_EXTRACT
WARN: There's no process matching config selector: .*:ALIGN_HISAT2:HISAT2_ALIGN
WARN: There's no process matching config selector: NFCORE_RNASEQ:RNASEQ:BAM_SORT_SAMTOOLS:SAMTOOLS_SORT
WARN: There's no process matching config selector: NFCORE_RNASEQ:RNASEQ:BAM_SORT_SAMTOOLS:SAMTOOLS_INDEX
WARN: There's no process matching config selector: NFCORE_RNASEQ:RNASEQ:BAM_SORT_SAMTOOLS:BAM_STATS_SAMTOOLS:.*
WARN: There's no process matching config selector: .*:DEDUP_UMI_UMITOOLS_GENOME:UMITOOLS_DEDUP
WARN: There's no process matching config selector: .*:DEDUP_UMI_UMITOOLS_GENOME:SAMTOOLS_INDEX
WARN: There's no process matching config selector: .*:DEDUP_UMI_UMITOOLS_GENOME:BAM_STATS_SAMTOOLS:.*
WARN: There's no process matching config selector: .*:DEDUP_UMI_UMITOOLS_TRANSCRIPTOME:UMITOOLS_DEDUP
WARN: There's no process matching config selector: .*:DEDUP_UMI_UMITOOLS_TRANSCRIPTOME:SAMTOOLS_INDEX
WARN: There's no process matching config selector: .*:DEDUP_UMI_UMITOOLS_TRANSCRIPTOME:BAM_STATS_SAMTOOLS:.*

These process are not run in the test profile, but there is configuration for them.

What if we use conditional process selectors? 🤔 Is that even possible in a config file? So for example, withName for certain processes are only used if a particular params is set. I realise the code below won't work but something along those lines?

if (params.aligner == 'star_rsem') {
    withName: ......

}

Yes. That's possible. That's effectively how we're allowing args from the command line:

if ( params.process ){
    params.process.each { selector ->
        selector.value.each { extension ->
            process."withName:${selector.key}".ext."${extension.key}" = selector.value."${extension.key}"
        }
    }
}

I need to take a closer look at the code to see what conditionals are needed though.

Absolutely fantastic effort @mahesh-panchal ! 👏🏽

New bug. Not sure if it's my fault in configuration.

Error executing process > 'NFCORE_RNASEQ:RNASEQ:BBMAP_BBSPLIT (1)'

Caused by:
  Process `NFCORE_RNASEQ:RNASEQ:BBMAP_BBSPLIT (1)` terminated with an error exit status (1)

Command executed:

  bbsplit.sh \
      -Xmx6g \
      path=bbsplit \
      threads=2 \
      in=RAP1_IAA_30M_REP1_1_val_1.fq.gz in2=RAP1_IAA_30M_REP1_2_val_2.fq.gz \
      basename=RAP1_IAA_30M_REP1_%_#.fastq.gz \
      refstats=RAP1_IAA_30M_REP1.stats.txt \
      build=1 ambiguous2=all maxindel=150000
  
  cat <<-END_VERSIONS > versions.yml
  BBMAP_BBSPLIT:
      bbmap: $(bbversion.sh 2>&1)
  END_VERSIONS

Command exit status:
  1
Command output:
  (empty)

Command error:
  java -ea -Xmx6g -Xms6g -cp /usr/local/opt/bbmap-38.93-0/current/ align2.BBSplitter ow=t fastareadlen=500 minhits=1 minratio=0.56 maxindel=20 qtrim=rl untrim=t trimq=6 -Xmx6g path=bbsplit threads=2 in=RAP1_IAA_30M_REP1_1_val_1.fq.gz in2=RAP1_IAA_30M_REP1_2_val_2.fq.gz basename=RAP1_IAA_30M_REP1_%_#.fastq.gz refstats=RAP1_IAA_30M_REP1.stats.txt build=1 ambiguous2=all maxindel=150000
  Executing align2.BBSplitter [ow=t, fastareadlen=500, minhits=1, minratio=0.56, maxindel=20, qtrim=rl, untrim=t, trimq=6, -Xmx6g, path=bbsplit, threads=2, in=RAP1_IAA_30M_REP1_1_val_1.fq.gz, in2=RAP1_IAA_30M_REP1_2_val_2.fq.gz, basename=RAP1_IAA_30M_REP1_%_#.fastq.gz, refstats=RAP1_IAA_30M_REP1.stats.txt, build=1, ambiguous2=all, maxindel=150000]
  
  Merged reference file bbsplit/ref/genome/1/merged_ref_8374379829187813017.fa.gz already exists; skipping merge.
  Ref merge time:       0.191 seconds.
  Executing align2.BBMap [ow=t, fastareadlen=500, minhits=1, minratio=0.56, maxindel=20, qtrim=rl, untrim=t, trimq=6, threads=2, in=RAP1_IAA_30M_REP1_1_val_1.fq.gz, in2=RAP1_IAA_30M_REP1_2_val_2.fq.gz, refstats=RAP1_IAA_30M_REP1.stats.txt, build=1, ambiguous2=all, maxindel=150000, ref=bbsplit/ref/genome/1/merged_ref_8374379829187813017.fa.gz, out_primary=RAP1_IAA_30M_REP1_primary_#.fastq.gz, out_sarscov2=RAP1_IAA_30M_REP1_sarscov2_#.fastq.gz, out_human=RAP1_IAA_30M_REP1_human_#.fastq.gz]
  Version 38.93
  
  Set MINIMUM_ALIGNMENT_SCORE_RATIO to 0.560
  Set threads to 2
  Reference set statistics will be written to RAP1_IAA_30M_REP1.stats.txt
  Retaining first best site only for ambiguous mappings.
  NOTE: Ignoring reference file because it already appears to have been processed.
  NOTE: If you wish to regenerate the index, please manually delete bbsplit/ref/genome/1/summary.txt
  Set genome to 1
  
  Loaded Reference:     0.126 seconds.
  Loading index for chunk 1-1, build 1
  Exception in thread "Thread-0" java.lang.RuntimeException: java.util.zip.ZipException: too many length or distance symbols
        at fileIO.ReadWrite.readObject(ReadWrite.java:823)
        at fileIO.ReadWrite.read(ReadWrite.java:1282)
        at align2.Block.read(Block.java:132)
        at align2.IndexMaker4$BlockMaker.makeArrays(IndexMaker4.java:140)
        at align2.IndexMaker4$BlockMaker.run(IndexMaker4.java:128)
  Caused by: java.util.zip.ZipException: too many length or distance symbols
        at java.base/java.util.zip.InflaterInputStream.read(InflaterInputStream.java:165)
        at java.base/java.util.zip.GZIPInputStream.read(GZIPInputStream.java:118)
        at java.base/java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2854)
        at java.base/java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2870)
        at java.base/java.io.ObjectInputStream$BlockDataInputStream.readInts(ObjectInputStream.java:3523)
        at java.base/java.io.ObjectInputStream.readArray(ObjectInputStream.java:2081)
        at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
        at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:493)
        at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:451)
        at fileIO.ReadWrite.readObject(ReadWrite.java:819)
        ... 4 more
  Generated Index:      0.642 seconds.
  Exception in thread "main" java.lang.NullPointerException
        at align2.BBIndex.analyzeIndex(BBIndex.java:115)
        at align2.BBMap.loadIndex(BBMap.java:420)
        at align2.BBMap.main(BBMap.java:32)
        at align2.BBSplitter.main(BBSplitter.java:48)

Run with:

#! /usr/bin/env nextflow

NXF_VER=21.10.0-SNAPSHOT \
nextflow run rnaseq/main.nf \
    --outdir "$PWD/pubdir_results" \
    --project 'snic2018-8-34' \
    -profile test,uppmax \
    -resume \
    -w "$PWD/work"

Looks like this may have been a one-off fluke. I haven't been able to reproduce.