Skip to content

micromamba-exported explicit conda lock files don't work over http, or named as .lock #5453

@pinin4fjords

Description

@pinin4fjords

Bug report

Expected behavior and actual behavior

Expected:

  • If a Conda lock file works locally, we should be able to supply it by http.
  • We should also have clear documentation around required naming/ file extensions for lock files.

Observed

Here is a conda lockfile produced with micromamba like micromamba env export --explicit > env.txt:

env.txt

This works:

#!/usr/bin/env nextflow

process FOO {

    debug true
    conda 'env.txt'

    script:
    """
    R --version
    """

}

workflow {
    FOO()
}

We would expect the same file hosted remotely to work:

...
conda 'https://gist.githubusercontent.com/pinin4fjords/e8cdca9521586f667ed1c5bcd33598d9/raw/550f0f4ed17f5d2741552a2b9f1d625239e499ec/env.txt'
...

... but it does not:

Nextflow 24.09.2-edge is available - Please consider updating your version to it

 N E X T F L O W   ~  version 24.02.0-edge

 ┃ Launching `main.nf` [cheesy_marconi] DSL2 - revision: 510b3a1397

[-        ] FOO -
[-        ] FOO -
Creating env using conda: https://gist.githubusercontent.com/pinin4fjords/e8cdca9521586f667ed1c5bcd33598d9/raw/550f0f4ed17f5d2741552a2b9f1d625239e499ec/env.txt [cache /workspace/gitpod/hello-nextflow/foo/work/conda/env-eeddd0960df503db901cc9c3644e607a]
ERROR ~ Error executing process > 'FOO'

Caused by:
[-        ] FOO -
Creating env using conda: https://gist.githubusercontent.com/pinin4fjords/e8cdca9521586f667ed1c5bcd33598d9/raw/550f0f4ed17f5d2741552a2b9f1d625239e499ec/env.txt [cache /workspace/gitpod/hello-nextflow/foo/work/conda/env-eeddd0960df503db901cc9c3644e607a]
ERROR ~ Error executing process > 'FOO'

Caused by:
  Failed to create Conda environment
  command: conda create --mkdir --yes --quiet --prefix /workspace/gitpod/hello-nextflow/foo/work/conda/env-eeddd0960df503db901cc9c3644e607a --file /workspace/gitpod/hello-nextflow/foo/https\:/gist.githubusercontent.com/pinin4fjords/e8cdca9521586f667ed1c5bcd33598d9/raw/550f0f4ed17f5d2741552a2b9f1d625239e499ec/env.txt
  status : 1
  message:
    /opt/conda/lib/python3.12/site-packages/conda/cli/conda_argparse.py:248: FutureWarning: `--mkdir` is deprecated and will be removed in 25.3. Redundant argument.
      super().__call__(parser, namespace, values, option_string)
    
    CondaFileIOError: '/workspace/gitpod/hello-nextflow/foo/https:/gist.githubusercontent.com/pinin4fjords/e8cdca9521586f667ed1c5bcd33598d9/raw/550f0f4ed17f5d2741552a2b9f1d625239e499ec/env.txt'. [Errno 2] No such file or directory: '/workspace/gitpod/hello-nextflow/foo/https:/gist.githubusercontent.com/pinin4fjords/e8cdca9521586f667ed1c5bcd33598d9/raw/550f0f4ed17f5d2741552a2b9f1d625239e499ec/env.txt'



 -- Check '.nextflow.log' file for details

Naming the file with .lock, even locally does not work:

...
    conda 'env.lock'
...

(i.e renaming env.txt to env.lock):

Nextflow 24.09.2-edge is available - Please consider updating your version to it

 N E X T F L O W   ~  version 24.02.0-edge

 ┃ Launching `main.nf` [awesome_dalembert] DSL2 - revision: 021eb262d1

[-        ] FOO -
Creating env using conda: env.lock [cache /workspace/gitpod/hello-nextflow/foo/work/conda/env-f187ecbc64371e10748465ebfbff2242]
ERROR ~ Error executing process > 'FOO'

Caused by:
  Failed to create Conda environment
  command: conda create --mkdir --yes --quiet --prefix /workspace/gitpod/hello-nextflow/foo/work/conda/env-f187ecbc64371e10748465ebfbff2242 env.lock
  status : 1
  message:
    /opt/conda/lib/python3.12/site-packages/conda/cli/conda_argparse.py:248: FutureWarning: `--mkdir` is deprecated and will be removed in 25.3. Redundant argument.
      super().__call__(parser, namespace, values, option_string)
    Channels:
     - conda-forge
     - bioconda
     - defaults
    Platform: linux-64
[-        ] FOO -
Creating env using conda: env.lock [cache /workspace/gitpod/hello-nextflow/foo/work/conda/env-f187ecbc64371e10748465ebfbff2242]
ERROR ~ Error executing process > 'FOO'

Caused by:
  Failed to create Conda environment
  command: conda create --mkdir --yes --quiet --prefix /workspace/gitpod/hello-nextflow/foo/work/conda/env-f187ecbc64371e10748465ebfbff2242 env.lock
  status : 1
  message:
    /opt/conda/lib/python3.12/site-packages/conda/cli/conda_argparse.py:248: FutureWarning: `--mkdir` is deprecated and will be removed in 25.3. Redundant argument.
      super().__call__(parser, namespace, values, option_string)
    Channels:
     - conda-forge
     - bioconda
     - defaults
    Platform: linux-64
    Collecting package metadata (repodata.json): ...working... done
    Solving environment: ...working... failed
    
    PackagesNotFoundError: The following packages are not available from current channels:
    
      - env.lock
    
    Current channels:
    
      - https://conda.anaconda.org/conda-forge
      - https://conda.anaconda.org/bioconda
      - defaults
      - https://repo.anaconda.com/pkgs/main
      - https://repo.anaconda.com/pkgs/r
    
    To search for alternate channels that may provide the conda package you're
    looking for, navigate to
    
        https://anaconda.org
    
    and use the search bar at the top of the page.



 -- Check '.nextflow.log' file for details

Neither does supplying the .lock file by http:

...
    conda 'https://gist.githubusercontent.com/pinin4fjords/c5a32673f084cdb772b03a63b9b1d118/raw/c6493bad0819bf8dff0c342af06eaeeee9c03af0/env.lock'
...

Error:

 N E X T F L O W   ~  version 24.02.0-edge

 ┃ Launching `main.nf` [nasty_torvalds] DSL2 - revision: b01a400c6a

[-        ] FOO -
[-        ] FOO -
Creating env using conda: https://gist.githubusercontent.com/pinin4fjords/c5a32673f084cdb772b03a63b9b1d118/raw/c6493bad0819bf8dff0c342af06eaeeee9c03af0/env.lock [cache /workspace/gitpod/hello-nextflow/foo/work/conda/env-ea0ebf23bfe548bbed42e257c0d75fa4]
ERROR ~ Error executing process > 'FOO'

Caused by:
  Failed to create Conda environment
  command: conda create --mkdir --yes --quiet --prefix /workspace/gitpod/hello-nextflow/foo/work/conda/env-ea0ebf23bfe548bbed42e257c0d75fa4 https://gist.githubusercontent.com/pinin4fjords/c5a32673f084cdb772b03a63b9b1d118/raw/c6493bad0819bf8dff0c342af06eaeeee9c03af0/env.lock
  status : 1
  message:
    /opt/conda/lib/python3.12/site-packages/conda/cli/conda_argparse.py:248: FutureWarning: `--mkdir` is deprecated and will be removed in 25.3. Redundant argument.
      super().__call__(parser, namespace, values, option_string)
    Channels:
     - conda-forge
     - bioconda
     - defaults
    Platform: linux-64
[-        ] FOO -
Creating env using conda: https://gist.githubusercontent.com/pinin4fjords/c5a32673f084cdb772b03a63b9b1d118/raw/c6493bad0819bf8dff0c342af06eaeeee9c03af0/env.lock [cache /workspace/gitpod/hello-nextflow/foo/work/conda/env-ea0ebf23bfe548bbed42e257c0d75fa4]
ERROR ~ Error executing process > 'FOO'

Caused by:
  Failed to create Conda environment
  command: conda create --mkdir --yes --quiet --prefix /workspace/gitpod/hello-nextflow/foo/work/conda/env-ea0ebf23bfe548bbed42e257c0d75fa4 https://gist.githubusercontent.com/pinin4fjords/c5a32673f084cdb772b03a63b9b1d118/raw/c6493bad0819bf8dff0c342af06eaeeee9c03af0/env.lock
  status : 1
  message:
    /opt/conda/lib/python3.12/site-packages/conda/cli/conda_argparse.py:248: FutureWarning: `--mkdir` is deprecated and will be removed in 25.3. Redundant argument.
      super().__call__(parser, namespace, values, option_string)
    Channels:
     - conda-forge
     - bioconda
     - defaults
    Platform: linux-64
    Collecting package metadata (repodata.json): ...working... done
    Solving environment: ...working... failed
    
    PackagesNotFoundError: The following packages are not available from current channels:
    
      - //gist.githubusercontent.com/pinin4fjords/c5a32673f084cdb772b03a63b9b1d118/raw/c6493bad0819bf8dff0c342af06eaeeee9c03af0/env.lock
    
    Current channels:
    
      - https://conda.anaconda.org/conda-forge
      - https://conda.anaconda.org/bioconda
      - defaults
      - https://repo.anaconda.com/pkgs/main
      - https://repo.anaconda.com/pkgs/r
    
    To search for alternate channels that may provide the conda package you're
    looking for, navigate to
    
        https://anaconda.org
    
    and use the search bar at the top of the page.



 -- Check '.nextflow.log' file for details

Environment

  • Nextflow version: 24.09.2-edge
  • Java version: [?]
  • Operating system: Linux
  • Bash version: (use the command $SHELL --version)

Additional context

(Add any other context about the problem here)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions