CI: Use mamba-org/provision-with-micromamba to setup micromamba#2435
CI: Use mamba-org/provision-with-micromamba to setup micromamba#2435
Conversation
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| python-version: ['3.11'] |
There was a problem hiding this comment.
I don't think we will build docs on multiple Python versions, so no need to define python-version in matrix.
weiji14
left a comment
There was a problem hiding this comment.
Thanks @seisman, I've been thinking about using provision-with-micromamba for a while to speed the CI tests up. But am a little unsure about maintaining more environment.yml files in the repo because it'll require more jumping back and forth. Trying to think if there's a cleaner way to handle our matrix builds somehow...
| cache-downloads: true | ||
| cache-env: true |
There was a problem hiding this comment.
According to https://github.com/mamba-org/provision-with-micromamba/tree/v15#cache-env-key:
With the default environment cache key, separate caches will be created for each operating system (eg., Linux) and platform (eg., x64) and day (eg., 2022-01-31), and the cache will be invalidated whenever the contents of environment-file or extra-specs change.
And at https://github.com/mamba-org/provision-with-micromamba/tree/v15#notes-on-caching / mamba-org/provision-with-micromamba#38 (comment)
Env caching should always work OK given how Conda works, and assuming that tar + untar doesn't lose file system permissions etc. One caveat is that you might have older versions of dependencies but that is limited to the cache TTL (currently hardcoded to 1 day). If you make any changes to your Conda env folder for whatever horrible reason, the changes are included in the cached version and your build may break.
Download caching is generally slower because it only eliminates the download time. Actually it doesn't eliminate it, only (presumably) reduce it because it's loaded from a "faster" network (Azure Blob storage?!) Potentially it's even slower than an uncached install because download + extraction are possibly done with less parallelism. In practice it always seems to be faster though. Advantage of download caching is that you are always up to date.
So we want to use both env and download caching? We could try both for now, and can modify things later if it becomes problematic.
There are two different options to specify the dependencies:
Option 2 doesn't work well for the For the |
The problem can be solved if we use multiline-string syntax for optional packages. See https://github.com/GenericMappingTools/pygmt/pull/2507/files for a proof of concept. It means we can list all packages in the workflow file. |
| uses: mamba-org/provision-with-micromamba@v15 | ||
| with: | ||
| environment-name: pygmt | ||
| environment-file: false |
There was a problem hiding this comment.
Must set environment-file to false, otherwise it will read the environment.yml file plus the packages listed in extra-specs.
| # Show installed pkg information for postmortem diagnostic | ||
| - name: List installed packages | ||
| run: mamba list |
There was a problem hiding this comment.
The mamba list step is removed because all packages are already listed at the "Environment Info" section of the Setup Micromamba step:
Co-authored-by: Wei Ji <23487320+weiji14@users.noreply.github.com>
Description of proposed changes
Use the provision-with-micromamba action to setup micromamba, which is faster than the current conda-incubator/setup-miniconda action which setups mamba.
Comparisons between the old (using mamba) and the new (using micromamba) runs
Here are the times to setup the virtual enviroment using micromamb and mamba:
After enabling caching of the downloads and the environment, the workflow takes 2-3 minutes to make the tarball of cache and upload them, so the whole workflow may be even slower. However, for the next run with the downloads and env cache (e.g., https://github.com/GenericMappingTools/pygmt/actions/runs/4459966220/usage), setting up the environment only takes 25s (Linux), 72s (macOS) and 260s (Windows).
Reference: https://github.com/mamba-org/provision-with-micromamba
Addresses #584.