MRG: Use caching in Github Actions and Azure Pipelines#8695
Merged
larsoner merged 47 commits intomne-tools:masterfrom Jan 6, 2021
Merged
MRG: Use caching in Github Actions and Azure Pipelines#8695larsoner merged 47 commits intomne-tools:masterfrom
larsoner merged 47 commits intomne-tools:masterfrom
Conversation
larsoner
reviewed
Jan 4, 2021
.github/workflows/compat_minimal.yml
Outdated
| id: cache-dataset | ||
| with: | ||
| path: /home/runner/mne_data | ||
| key: ${{ runner.os }}-dataset |
Member
There was a problem hiding this comment.
I don't think we need to separate by OS, the data files should all be the same
Contributor
Author
|
Among the first batch of runners, no one detected the cache: and Failing skips the I'll push an empty commit to confirm that they at least restore the cache successfully. |
larsoner
reviewed
Jan 4, 2021
Contributor
Author
|
|
Contributor
Author
|
I see no problem at restoration 👌 :
|
Member
|
Do you want to try dealing with testing dataset |
Contributor
Author
|
You'll probably be more efficient than me 😅 . Go ahead if you have the time. |
Member
|
Will do now, could really use something in my wheelhouse to jump-start things over here... |
Member
|
@GuillaumeFavelier if you want to look into it, we should do the same for Azure |
Member
|
@GuillaumeFavelier you can use something like the following in a one-liner When Dan's #8679 is in progress (probably after my PR then yours) then this line can probably be adjusted fairly easily |
Contributor
Author
|
Checking the key based on the testing version is a success!
|
Contributor
Author
|
The Perl extension is not available on MacOS: |
Contributor
Author
|
Also, the cache is different depending on the OS: On MacOS: For the others (on Linux): |
Contributor
Author
|
Because there is a And no way to make their tip work since it's not a fixed string but it comes from a variable:
I'll |
Contributor
Author
|
I don't know what to say 🤦 |
Contributor
Author
|
The jobs on Windows detected the key: And restored the cache correctly But the data are downloaded anyway because the path is different on Windows. |
Member
|
Hah! 44 sec to download the data on And 1m 11 sec to get it from cache: So we actually get a slowdown from this on Azure at least (!) but at least it should be more reliable. |
Contributor
Author
|
Hm... But for some reason it downloads more no? Restoring the cache: Compared to |
Member
That's probably the uncompressed size, vs the download which is compressed. |
Contributor
Author
|
Unrelated failure on DetailsMaybe one last empty commit for testing but everything is ok for me. Is there something missing @larsoner ? |
Member
I don't think we need the empty commit, I'll just restart the segfaulted GitHub actions build, and it looks like everything else should be okay. Will merge once CIs are happy |
Member
|
Thanks @GuillaumeFavelier ! |
Contributor
Author
|
Something interesting about caching:
|
larsoner
added a commit
to larsoner/mne-python
that referenced
this pull request
Jan 7, 2021
* upstream/master: MAINT: Actually use all caches (mne-tools#8702) MRG: Prototype of notebook viz (ipyvtk) (mne-tools#8503) MRG: Use caching in Github Actions and Azure Pipelines (mne-tools#8695)
cbrnr
pushed a commit
to cbrnr/mne-python
that referenced
this pull request
Jan 15, 2021
* First test [skip azp][skip circle] * Download anyway [skip azp][skip circle] * Unify cache key [skip azp][skip circle] * Test new key * Test linux_pip [skip azp][skip circle] * Try again [skip azp][skip circle] * Deploy to all jobs * Try another syntax * Test new key format [skip azp][skip circle] * Use home env variable * Fix key [skip azp][skip circle] * Try naive version on Azure * Try again [skip github][skip circle] * Try again [skip github][skip circle] * Try another syntax [skip github][skip circle] * Try again [skip github] * Try again [skip github] * Try again [skip github] * Try again [skip github] * Try again [skip github] * Debug variable [skip github] * Debug variable [skip github] * Debug variable [skip github] * Debug variable [skip github] * Debug variable [skip github] * Debug variable [skip github] * Debug variable [skip github] * Debug variable [skip github] * Debug variable [skip github] * Debug variable [skip github] * Test again? [skip github] * Debug variable [skip github] * Debug variable [skip github] * Debug variable [skip github] * Debug variable [skip github] * Test new key format [skip github] * Test new key format [skip github] * Test new key format [skip github] * Try again [skip github] * Try again [skip github] * Try again [skip github] * Deploy to all jobs * Try another syntax [skip github][skip circle] * Try another syntax [skip github][skip circle] * Try another syntax [skip github][skip circle] * Test again
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR follows #8685 (comment) and attempts to solve the recent failures related to testing data download.
ToDo:
$HOMEforpathvariable