Update dask backend for compatibility with return_as=generator#1520
Update dask backend for compatibility with return_as=generator#1520ogrisel merged 7 commits intojoblib:masterfrom
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #1520 +/- ##
==========================================
+ Coverage 94.99% 95.06% +0.07%
==========================================
Files 45 44 -1
Lines 7551 7564 +13
==========================================
+ Hits 7173 7191 +18
+ Misses 378 373 -5
☔ View full report in Codecov by Sentry. |
d91ae62 to
1999ea6
Compare
1acf7d4 to
104e6e3
Compare
d350c34 to
72f1b68
Compare
72f1b68 to
42e5731
Compare
b4957b2 to
25a6768
Compare
|
@ogrisel the PR now proposes a fix to the coverage pipeline. The issue seems to have been that codecov and/or github integration do not support updating the github annotations dynamically with successive upload from parallel pipelines, but we had each pipeline individually upload their report. The first pipeline to reach codecov would trigger codecov annotations and those would not be cleared by subsequent updates, despite annotated code being actually covered by later pipelines. (I also suspect consecutive uploads do not merge correctly for python to begin with so this might have affected the final reports too ?) I changed the script so that every pipeline creates a local artifact with its coverage report, and a single pipeline is triggered after all other are done that combines all coverage reports before issuing a single codecov upload. That will also prevent receiving early e-mails from codecov because the codecov pipeline is temporarily failing because incomplete reports, despite later turning green with subsequent updates. It also fixes some unelegant codecov error messages in the pipelines. While debugging I also found that the 3.8 distributed pipeline was not working because the pinned versions were uninstallable and caused |
83954fb to
4aead14
Compare
ee7376e to
cbc7604
Compare
cbc7604 to
20b4418
Compare
|
There were also issues when combining coverage reports from different pipelines because coverage reports stored absolute paths, and those diverged accross pipelines (macos / windows / linux absolute paths are different ?). Fixed now. |
ogrisel
left a comment
There was a problem hiding this comment.
Thanks very much for the CI fix and the new feature.
Just a minor comment on naming, but otherwise LGTM!
Reasonably simple changes, it seems to be just working as intended. Let's see if the CI agrees.
Are there particular tests to add on top ? the code paths for return_as=generator are the same than for return_as=list so test_dask already tests most of it.