Skip to content

Update upper bound for galaxy packages to < 23.2#1388

Merged
mvdbeek merged 4 commits intogalaxyproject:masterfrom
bernt-matthias:23.1
Dec 6, 2023
Merged

Update upper bound for galaxy packages to < 23.2#1388
mvdbeek merged 4 commits intogalaxyproject:masterfrom
bernt-matthias:23.1

Conversation

@bernt-matthias
Copy link
Collaborator

@bernt-matthias bernt-matthias commented Sep 14, 2023 •
edited
Loading

... waiting for the 23.1 release :)

nsoranzo
nsoranzo reviewed Sep 14, 2023
View reviewed changes
@bernt-matthias bernt-matthias mentioned this pull request Oct 27, 2023
Closed
5 tasks
@mvdbeek mvdbeek closed this Oct 29, 2023
@mvdbeek mvdbeek reopened this Oct 29, 2023
@bernt-matthias
Copy link
Collaborator Author

bernt-matthias commented Oct 29, 2023

Test is failing reproducibly. But not sure if related.

@bernt-matthias bernt-matthias mentioned this pull request Nov 6, 2023
Merged
5 tasks
@bgruening bgruening closed this Nov 10, 2023
@bgruening bgruening reopened this Nov 10, 2023
@bgruening
Copy link
Member

bgruening commented Nov 12, 2023

The error I can see is:

E           PackagesNotFoundError: The following packages are not available from current channels:
E           
E             - fleeqtk
E           
E           Current channels:
E           
E             - https://conda.anaconda.org/conda-forge/linux-64
E             - https://conda.anaconda.org/conda-forge/noarch
E             - https://conda.anaconda.org/bioconda/linux-64
E             - https://conda.anaconda.org/bioconda/noarch
E             - https://repo.anaconda.com/pkgs/main/linux-64
E             - https://repo.anaconda.com/pkgs/main/noarch
E             - https://repo.anaconda.com/pkgs/r/linux-64
E             - https://repo.anaconda.com/pkgs/r/noarch

Which is true. This packages does not exist, its a fake package that is build locally during a test.

I think --conda_use_local is not working in this PR.

@bernt-matthias
Copy link
Collaborator Author

bernt-matthias commented Nov 12, 2023 •
edited
Loading

Tried to execute the commands manually and got

/tmp/tmpn84st44m/miniconda3/bin/conda build /home/berntm/projects/planemo/project_templates/conda_answers/exercise_2/fleeqtk   

# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<

    Traceback (most recent call last):
      File "/tmp/tmpn84st44m/miniconda3/lib/python3.10/site-packages/conda/exceptions.py", line 1132, in __call__
        return func(*args, **kwargs)
      File "/tmp/tmpn84st44m/miniconda3/lib/python3.10/site-packages/conda/cli/main.py", line 69, in main_subshell
        exit_code = do_call(args, p)
      File "/tmp/tmpn84st44m/miniconda3/lib/python3.10/site-packages/conda/cli/conda_argparse.py", line 115, in do_call
        return args.plugin_subcommand.action(sys.argv[2:])
      File "/tmp/tmpn84st44m/miniconda3/lib/python3.10/site-packages/conda_build/plugin.py", line 10, in build
        execute(*args, **kwargs)
      File "/tmp/tmpn84st44m/miniconda3/lib/python3.10/site-packages/conda_build/cli/main_build.py", line 516, in execute
        _parser, args = parse_args(args)
      File "/tmp/tmpn84st44m/miniconda3/lib/python3.10/site-packages/conda_build/cli/main_build.py", line 470, in parse_args
        check_recipe(args.recipe)
    AttributeError: 'Namespace' object has no attribute 'recipe'

`$ /tmp/tmpn84st44m/miniconda3/bin/conda build /home/berntm/projects/planemo/project_templates/conda_answers/exercise_2/fleeqtk`

  environment variables:
                 CIO_TEST=<not set>
    CONDA_ALLOW_SOFTLINKS=false
                CONDA_EXE=/home/berntm/miniconda3/bin/conda
         CONDA_PYTHON_EXE=/home/berntm/miniconda3/bin/python
               CONDA_ROOT=/tmp/tmpn84st44m/miniconda3
              CONDA_SHLVL=0
           CURL_CA_BUNDLE=<not set>
            DEFAULTS_PATH=/usr/share/gconf/gnome-xorg.default.path
               LD_PRELOAD=<not set>
           MANDATORY_PATH=/usr/share/gconf/gnome-xorg.mandatory.path
                     PATH=/tmp/venv-planemo/bin:/usr/local/go/bin:/home/berntm/perl5/bin:no:/hom
                          e/berntm/bin:/home/berntm/.local/bin:/home/berntm/miniconda3/condabin:
                          /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/game
                          s:/usr/local/games:/snap/bin:/snap/bin
       REQUESTS_CA_BUNDLE=<not set>
            SSL_CERT_FILE=<not set>
               WINDOWPATH=2

     active environment : None
            shell level : 0
       user config file : /home/berntm/.condarc
 populated config files : /tmp/tmpn84st44m/miniconda3/.condarc
                          /home/berntm/.condarc
          conda version : 23.3.1
    conda-build version : 3.27.0
         python version : 3.10.12.final.0
       virtual packages : __archspec=1=x86_64
                          __glibc=2.35=0
                          __linux=5.15.0=0
                          __unix=0=0
       base environment : /tmp/tmpn84st44m/miniconda3  (writable)
      conda av data dir : /tmp/tmpn84st44m/miniconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/anaconda/linux-64
                          https://conda.anaconda.org/anaconda/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /tmp/tmpn84st44m/miniconda3/pkgs
                          /home/berntm/.conda/pkgs
       envs directories : /tmp/tmpn84st44m/miniconda3/envs
                          /home/berntm/.conda/envs
               platform : linux-64
             user-agent : conda/23.3.1 requests/2.31.0 CPython/3.10.12 Linux/5.15.0-83-generic ubuntu/22.04.3 glibc/2.35
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False


An unexpected error has occurred. Conda has prepared the above report.

If submitted, this report will be used by core maintainers to improve
future releases of conda.
Would you like conda to send this report to the core maintainers? [y/N]: ^C
Timeout reached. No report sent.

Might be solved by an update to conda conda/conda-build#4984 (comment)

bernt-matthias added a commit to bernt-matthias/galaxy that referenced this pull request Nov 12, 2023
@bernt-matthias
bump required conda version
450b71d 
potential solution for the build prblems in galaxyproject/planemo#1388
@bernt-matthias bernt-matthias mentioned this pull request Nov 12, 2023
Merged
4 tasks
@bernt-matthias
Copy link
Collaborator Author

bernt-matthias commented Nov 12, 2023

Bumping conda helps with the failing test. Let's see if there are side effects galaxyproject/galaxy#17013

@nsoranzo
Copy link
Member

nsoranzo commented Nov 12, 2023

I am not sure how galaxyproject/galaxy#17013 can help, it bumps the minimum conda version but in tests it's most likely that the latest conda version is installed either way.

@bernt-matthias
Copy link
Collaborator Author

bernt-matthias commented Nov 12, 2023

All I know is that in the failing test on my local computer not the latest was installed, but:

           conda version : 23.3.1
    conda-build version : 3.27.0

When I bumped the minimum requirement the tests were successful. No idea how to test this here, without bumping the Galaxy packages first. Can we somehow patch tool-util during the test?

@nsoranzo
Copy link
Member

nsoranzo commented Nov 12, 2023

All I know is that in the failing test on my local computer not the latest was installed, but:

           conda version : 23.3.1
    conda-build version : 3.27.0

Is that what you have in ~/miniconda3 ?

When I bumped the minimum requirement the tests were successful. No idea how to test this here, without bumping the Galaxy packages first. Can we somehow patch tool-util during the test?

Did you update the conda package manually? Changing CONDA_PACKAGE_SPECS only affects new conda installations.

@nsoranzo
Copy link
Member

nsoranzo commented Nov 12, 2023

What's weird when looking at the CI logs is that test_local_conda_dependencies_version failed at the planemo test stage, when the __fleeqtk@1.3 conda environment should have been created already by planemo conda_install right before.

@bernt-matthias
Copy link
Collaborator Author

bernt-matthias commented Nov 12, 2023

What's weird when looking at the CI logs is..

Yes. They had exit code 0, despite the failure.

@mvdbeek
Copy link
Member

mvdbeek commented Nov 13, 2023

Can you check in the github workflow if there is any interference from the conda or .condarc already installed on the VM ?

@bernt-matthias
Copy link
Collaborator Author

bernt-matthias commented Nov 13, 2023

So, now we see the error from the build step. Seems that this installs 23.3.1 .. my hypothesis is as follows:

  • the mambaforge installer comes with preinstalled conda 23.3.1 (just verified)
  • when installing conda use the two-step approach
    1. try to fulfill requirements such that installed packages don't need to be updated
    2. if this is not possible fulfill requirements and allow updating installed packages

so with our current conda pins, we are stuck at 23.3.1 (maybe also because we install in the base env?) and not the newest packages are installed.

Does this sound reasonable?

Note that also planemo conda_search seems not to use the exit code of the exec_command function. If desired I could try to fix it here.

@nsoranzo
Copy link
Member

nsoranzo commented Nov 13, 2023

Thanks @bernt-matthias , the latest commit to check the exit code really helped clarifying the issue, and your explanation makes sense. I didn't expect the latest Mambaforge to install conda 23.3.1, but any way it looks like that galaxyproject/galaxy#17013 is the correct solution here.

Note that also planemo conda_search seems not to use the exit code of the exec_command function. If desired I could try to fix it here.

Yes, that would be good, thanks!

@nsoranzo
Copy link
Member

nsoranzo commented Nov 13, 2023

planemo/commands/cmd_conda_install.py also has the same issue.

bernt-matthias added a commit to bernt-matthias/galaxy that referenced this pull request Nov 13, 2023
@bernt-matthias
bump required conda version
0aafa22 
potential solution for the build prblems in galaxyproject/planemo#1388
@bernt-matthias
Copy link
Collaborator Author

bernt-matthias commented Dec 4, 2023

Does anyone have an idea what's going on with the 3.7 tests?

@mvdbeek
Copy link
Member

mvdbeek commented Dec 5, 2023

dev doesn't work on 3.7, that's why I rebased your PR

@mvdbeek mvdbeek changed the title update upper bound for galaxy packages Update upper bound for galaxy packages to < 23.2 Dec 6, 2023
mvdbeek
mvdbeek approved these changes Dec 6, 2023
View reviewed changes
Copy link
Member

@mvdbeek mvdbeek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very cool, thanks @bernt-matthias

@mvdbeek mvdbeek merged commit 5951e90 into galaxyproject:master Dec 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@nsoranzo nsoranzo nsoranzo left review comments

@mvdbeek mvdbeek mvdbeek approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@bernt-matthias @bgruening @nsoranzo @mvdbeek