MNT add CI jobs to run integration tests for downstream projects: loky, joblib, distributed #236

pierreglaser · 2019-01-25T11:38:46Z

This PR configures cron-jobs that build and launch the test suite of downtream projects that rely on cloudpickle. The main purpose is to find out quickly if we introduce breaking changes.

Ideally we would rather test the stable (PyPI) rather than the latest (master branch) version, but each time, a project-specific issue was making it at least clumsy, at worst impossible. (I can detail upon request).

codecov-io · 2019-01-25T13:30:20Z

Codecov Report

Merging #236 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #236   +/-   ##
=======================================
  Coverage   85.12%   85.12%           
=======================================
  Files           1        1           
  Lines         585      585           
  Branches      117      117           
=======================================
  Hits          498      498           
  Misses         63       63           
  Partials       24       24

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ca8be49...5b468cd. Read the comment docs.

ogrisel

Thanks for tackling this. Here are some comments.

.travis.yml

pierreglaser · 2019-01-25T17:25:32Z

Mmh. So the good news is that there does not seem to be any syntax error. Also, loky and joblib tests passed. However, 3 distributed tests failed, but travis is still green...

.travis.yml

this seems to fix test_duplicate_clients

pierreglaser · 2019-01-28T11:55:19Z

OK so apparentlty openssl is using using SSLv3, whereas distributed would like us to use TLSv1.2 at least. @ogrisel is this worth the hassle? Should we switch to an anaconda distribution or skip the tests? @lesteve are you familiar with any of this? If yes, any help would be much appreciated :)

lesteve · 2019-01-28T16:31:13Z

@lesteve are you familiar with any of this? If yes, any help would be much appreciated :)

No suggestion off the top of my head, sorry ...

ogrisel

Instead of running those integration tests as cron jobs, I am thinking we could run them conditionally whenever the commit message has a special flag. For instance, for the joblib entry:

commit_message =~ /(ci-downstream|ci-joblib)/

https://docs.travis-ci.com/user/conditions-v1

WDYT?

pierreglaser · 2019-01-29T09:19:47Z

This sounds great to tackle issues early. But it will be harder for us to spot a breaking change if we miss it initially. We would at least have to push empty commits with ci-downstream flags before each new PyPI release.

ogrisel · 2019-01-29T09:21:25Z

We would at least have to push empty commits with ci-downstream flags before each new PyPI release.

Yes but we can do that in a PR that we do not merge to avoid polluting master with empty commits.

ogrisel · 2019-01-29T09:23:36Z

But it will be harder for us to spot a breaking change if we miss it initially.

So be it. It's a tradeoff between long CI jobs, unwanted noisy notifications (in case the downstream master branches are failing randomly for causes unrelated to cloudpickle) and thinking about doing manual triggers prior to making a cloudpickle release.

pierreglaser · 2019-01-29T09:26:08Z

OK. I'll change the trigger and make a few test commits inside the PR with the appropriate flags.

ogrisel

Actually, changing my mind on the commit message tagging syntax:

.travis.yml

ogrisel · 2019-01-29T13:52:13Z

It seems to work. Can you just try with [ci downstream] to make sure that it will trigger all of them at once?

ogrisel · 2019-01-29T13:53:24Z

Did you cancel the [ci distributed] build on purpose?

pierreglaser · 2019-01-29T13:55:44Z

Did you cancel the [ci distributed] build on purpose?

Nope, actually I thought you did. Might be because of auto cancelling builds when more recent commits are pushed?

pierreglaser · 2019-01-29T13:56:40Z

Besides that the filter seems to work indeed.

ogrisel · 2019-01-29T13:57:05Z

Good, let's wait for the final build to complete.

ogrisel · 2019-01-29T14:28:56Z

Merged. Thanks @pierreglaser!

pierreglaser added 3 commits January 25, 2019 12:35

MNT add cron builds for loky, joblib, distributed

12b337a

FIX add semicolons

8e9bb45

last semicolon

43ad1d6

ogrisel reviewed Jan 25, 2019

View reviewed changes

.travis.yml Outdated Show resolved Hide resolved

.travis.yml Outdated Show resolved Hide resolved

.travis.yml Outdated Show resolved Hide resolved

.travis.yml Outdated Show resolved Hide resolved

.travis.yml Outdated Show resolved Hide resolved

pierreglaser added 3 commits January 25, 2019 17:57

DOC add comment about travis's integration tests

61e410f

CLN check on PROJECT rather than TRAVIS_EVENT_TYPE

9e457ae

TST temporary remove the cron condition

bb9e327

ogrisel reviewed Jan 25, 2019

View reviewed changes

.travis.yml Outdated Show resolved Hide resolved

pierreglaser added 7 commits January 28, 2019 10:10

manually catch pytest exit code

2be1fcd

run distributed tests on a full virtual machine

04dd384

re-create distributed exact test environment

392227c

use python 3.6 for distributed

500de3e

add bokeh to distributed dependencies

33c4bbe

this seems to fix test_duplicate_clients

upgrade ubuntu because of openssl version

57b1d66

debug using my local fork

1a1def6

pierreglaser added 10 commits January 28, 2019 13:09

remove python3.6 dependency, upgrade to xenial

84236a3

test only the failing tests

98640fb

python 3.7

4efe205

skip the two failing tests

bcdd785

use pytest_args

8f23253

escape quotes properly

eeb82f0

another quote try

a783d64

yet another try

fb68703

debug

718f594

WIP

e9b3644

WIP

84ec4cf

pierreglaser added 2 commits January 29, 2019 09:56

last simplification attempt

1b41594

make sure exit codes are retrieved properly

a6551a3

ogrisel reviewed Jan 29, 2019

View reviewed changes

ogrisel changed the title ~~MNT add cron builds for loky, joblib, distributed~~ MNT add CI jobs to run integration tests for downstream projects: loky, joblib, distributed Jan 29, 2019

pierreglaser added 2 commits January 29, 2019 10:28

go back to manually catch exit code

0564dd1

ci-downstream use flags in commit_messages instead of cron jobs

a3c86bb

pierreglaser force-pushed the add-cron-jobs-for-downstream-projects branch from 64a122b to a3c86bb Compare January 29, 2019 09:43

ogrisel reviewed Jan 29, 2019

View reviewed changes

.travis.yml Outdated Show resolved Hide resolved

pierreglaser force-pushed the add-cron-jobs-for-downstream-projects branch from 9afceb6 to f1c2196 Compare January 29, 2019 13:35

[ci skip] use travis conventions to trigger ci

cacd7a0

pierreglaser force-pushed the add-cron-jobs-for-downstream-projects branch from f1c2196 to cacd7a0 Compare January 29, 2019 13:37

pierreglaser added 4 commits January 29, 2019 14:38

[ci loky] trigger loky's ci

95c7429

ci loky make sure this does not trigger ci

f86a03c

[ci distributed] trigger distributed test suite

1ae09c8

ci distributed make sure ci is not triggered

59bba18

[ci downstream] trigger all integration builds

5b468cd

ogrisel merged commit d5f37db into cloudpipe:master Jan 29, 2019

This was referenced Jan 29, 2019

CI configuration to run the tests of PySpark against development version of cloudpickle #237

Open

CI configuration to run the tests of Ray against development version of cloudpickle #238

Closed

williamma12 mentioned this pull request Mar 11, 2019

Add ray tests to ci configuration against the development version of cloudpickle #256

Merged

MNT add CI jobs to run integration tests for downstream projects: loky, joblib, distributed #236

MNT add CI jobs to run integration tests for downstream projects: loky, joblib, distributed #236

Uh oh!

Conversation

pierreglaser commented Jan 25, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-io commented Jan 25, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

ogrisel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pierreglaser commented Jan 25, 2019

Uh oh!

Uh oh!

pierreglaser commented Jan 28, 2019

Uh oh!

lesteve commented Jan 28, 2019

Uh oh!

ogrisel left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pierreglaser commented Jan 29, 2019

Uh oh!

ogrisel commented Jan 29, 2019

Uh oh!

ogrisel commented Jan 29, 2019

Uh oh!

pierreglaser commented Jan 29, 2019

Uh oh!

ogrisel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ogrisel commented Jan 29, 2019

Uh oh!

ogrisel commented Jan 29, 2019

Uh oh!

pierreglaser commented Jan 29, 2019

Uh oh!

pierreglaser commented Jan 29, 2019

Uh oh!

ogrisel commented Jan 29, 2019

Uh oh!

ogrisel commented Jan 29, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pierreglaser commented Jan 25, 2019 •

edited

Loading

codecov-io commented Jan 25, 2019 •

edited

Loading

ogrisel left a comment •

edited

Loading

ogrisel commented Jan 29, 2019 •

edited

Loading