Skip to content

ENH update loky 2.4.0#802

Merged
ogrisel merged 11 commits intojoblib:masterfrom
tomMoral:ENH_update_loky_2.4.0
Nov 2, 2018
Merged

ENH update loky 2.4.0#802
ogrisel merged 11 commits intojoblib:masterfrom
tomMoral:ENH_update_loky_2.4.0

Conversation

@tomMoral
Copy link
Copy Markdown
Contributor

@tomMoral tomMoral commented Nov 1, 2018

Bump the version of loky to 2.4.0.

This update changes the default behavior of joblib: cloudpickle is used by default to serialize the tasks. Note that this might cause some performance drop in particular for large python objects.

@codecov
Copy link
Copy Markdown

codecov bot commented Nov 1, 2018 •
edited
Loading

Codecov Report

Merging #802 into master will decrease coverage by 0.09%.
The diff coverage is 100%.

Impacted file tree graph

@@           Coverage Diff            @@
##           master    #802     +/-   ##
========================================
- Coverage   95.39%   95.3%   -0.1%     
========================================
  Files          44      44             
  Lines        6346    6308     -38     
========================================
- Hits         6054    6012     -42     
- Misses        292     296      +4
Impacted Files Coverage Δ
joblib/parallel.py 97.25% <ø> (-0.32%) ⬇️
joblib/test/test_parallel.py 96.85% <100%> (+0.01%) ⬆️
joblib/__init__.py 100% <100%> (ø) ⬆️
joblib/pool.py 91.37% <0%> (-3.45%) ⬇️
joblib/_parallel_backends.py 94.8% <0%> (ø) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 122f5f1...8d06b80. Read the comment docs.

@tomMoral tomMoral changed the title Enh update loky 2.4.0 ENH update loky 2.4.0 Nov 1, 2018
@ogrisel
Copy link
Copy Markdown
Contributor

ogrisel commented Nov 2, 2018

@tomMoral The failing test is expected on windows: it's no longer possible to naively use any interactively defined functions with the multiprocessing backend with the spawn start method.

I think we should just skip this test under windows with the multiprocessing backend.

@ogrisel
Copy link
Copy Markdown
Contributor

ogrisel commented Nov 2, 2018

Let me resolve the conflict with master.

@ogrisel
Copy link
Copy Markdown
Contributor

ogrisel commented Nov 2, 2018

Appveyor is confused by the merge commit done by the github web UI.

@ogrisel
Copy link
Copy Markdown
Contributor

ogrisel commented Nov 2, 2018

I rand the scikit-learn tests from scikit-learn master with this branch of joblib and all tests pass.

@tomMoral
Copy link
Copy Markdown
Contributor Author

tomMoral commented Nov 2, 2018

Note that before merging this PR, we need to do a bug-fix release of loky2.4.1 as it is now in 2.4.1dev0. I can do it once all the test are green and we are happy with the PR state.

Another missing part in this PR is to document the set_loky_pickler and the wrapper in the joblib documentation.

@ogrisel
Copy link
Copy Markdown
Contributor

ogrisel commented Nov 2, 2018

Ooops I deleted this comment by mistake:

This fixes: scikit-learn/scikit-learn#12250.

@ogrisel
Copy link
Copy Markdown
Contributor

ogrisel commented Nov 2, 2018

@tomMoral alright let's release loky 2.4.1 from the current master and re-update this joblib PR with it.

@ogrisel
Copy link
Copy Markdown
Contributor

ogrisel commented Nov 2, 2018

We got an error on appveyor with:

IOError: [Errno 13] Permission denied: 'c:\\users\\appveyor\\appdata\\local\\temp\\1\\pytest-of-appveyor\\pytest-0\\test_concurrency_safe_write_mu0\\test.pkl'

in test_concurrency_safe_write from joblib\test\test_store_backends.py.

This is annoying but I don't think it's related to loky.

@ogrisel ogrisel merged commit 4f1ca52 into joblib:master Nov 2, 2018
@tomMoral tomMoral deleted the ENH_update_loky_2.4.0 branch November 3, 2018 07:48
yarikoptic added a commit to yarikoptic/joblib that referenced this pull request Apr 12, 2019
@yarikoptic
Merge tag '0.13.0' into releases
03f656e 
* tag '0.13.0':
  Release 0.13.0
  Use https (joblib#805)
  MTN bump loky to 2.4.2 (joblib#804)
  DOC: provide some useful variables for Makefile (joblib#794)
  DOC serialization and processes (joblib#803)
  ENH update loky 2.4.0 (joblib#802)
  FIX backward compatibility for nested backend (joblib#789)
  enable python 3.7 (joblib#795)
  memory: add test for call_and_shelve performance (joblib#791)
  FIX nested backend not changed by SequentialBackend (joblib#792)
  cloudpickle 0.6.0 (joblib#788)
  FIX nested backend setting n_jobs=-1 (joblib#785)
  Raises a more explicit exception when a corrupted MemorizedResult is loaded (joblib#768)
  Back to dev mode
yarikoptic added a commit to yarikoptic/joblib that referenced this pull request Apr 12, 2019
@yarikoptic
Merge branch 'releases' into debian
11b4c79 
* releases:
  Release 0.13.0
  Use https (joblib#805)
  MTN bump loky to 2.4.2 (joblib#804)
  DOC: provide some useful variables for Makefile (joblib#794)
  DOC serialization and processes (joblib#803)
  ENH update loky 2.4.0 (joblib#802)
  FIX backward compatibility for nested backend (joblib#789)
  enable python 3.7 (joblib#795)
  memory: add test for call_and_shelve performance (joblib#791)
  FIX nested backend not changed by SequentialBackend (joblib#792)
  cloudpickle 0.6.0 (joblib#788)
  FIX nested backend setting n_jobs=-1 (joblib#785)
  Raises a more explicit exception when a corrupted MemorizedResult is loaded (joblib#768)
  Back to dev mode
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@tomMoral @ogrisel