Skip to content

FEA parallel_config context manager to allow more fine-grained control#1392

Merged
tomMoral merged 19 commits intojoblib:masterfrom
jeremiedbb:parallel-config
Apr 26, 2023
Merged

FEA parallel_config context manager to allow more fine-grained control#1392
tomMoral merged 19 commits intojoblib:masterfrom
jeremiedbb:parallel-config

Conversation

@jeremiedbb
Copy link
Copy Markdown
Contributor

@jeremiedbb jeremiedbb commented Feb 16, 2023 •
edited
Loading

Fixes #912
Fixes #979
related to #1376

adds a new context_manager parallel_config that should be used in place of parallel_backend. It exposes more parameters and does not enforce setting a backend.

It allows to do something like:

from joblib.parallel import parallel_config
from joblib.memory import Memory
from joblib import Parallel, delayed
 
with parallel_config(prefer="threads", n_jobs=3):
    res = Parallel(verbose=100)(delayed(lambda :1)() for _ in range(10))
 
# [Parallel(n_jobs=3)]: Using backend ThreadingBackend with 3 concurrent workers.
# [Parallel(n_jobs=3)]: Done   1 tasks      | elapsed:    0.0s
# [Parallel(n_jobs=3)]: Done   2 tasks      | elapsed:    0.0s
# [Parallel(n_jobs=3)]: Done   3 tasks      | elapsed:    0.0s
# [Parallel(n_jobs=3)]: Done   4 tasks      | elapsed:    0.0s
# [Parallel(n_jobs=3)]: Done   5 tasks      | elapsed:    0.0s
# [Parallel(n_jobs=3)]: Done   6 out of  10 | elapsed:    0.0s remaining:    0.0s
# [Parallel(n_jobs=3)]: Done   7 out of  10 | elapsed:    0.0s remaining:    0.0s
# [Parallel(n_jobs=3)]: Done   8 out of  10 | elapsed:    0.0s remaining:    0.0s
# [Parallel(n_jobs=3)]: Done  10 out of  10 | elapsed:    0.0s remaining:    0.0s
# [Parallel(n_jobs=3)]: Done  10 out of  10 | elapsed:    0.0s finished

It's using the threading backend instead of the default loky, with 3 jobs instead of the default 1.

The config parameters of the context are used only if they are not explicitely set in Parallel. It makes us forced to use sentinels for the default values, adding a little bit of complexity.

TODO

  • raise an error if backend constructor params are set but backend is left to None
  • add tests

@codecov
Copy link
Copy Markdown

codecov bot commented Feb 16, 2023 •
edited
Loading

Codecov Report

Patch coverage: 97.12% and project coverage change: -0.03 ⚠️

Comparison is base (41b70ff) 94.78% compared to head (4aa3509) 94.75%.

Additional details and impacted files 
@@            Coverage Diff             @@
##           master    #1392      +/-   ##
==========================================
- Coverage   94.78%   94.75%   -0.03%     
==========================================
  Files          44       44              
  Lines        7207     7306      +99     
==========================================
+ Hits         6831     6923      +92     
- Misses        376      383       +7
Impacted Files Coverage Δ
joblib/_utils.py 95.45% <83.33%> (-4.55%) ⬇️
joblib/test/test_parallel.py 96.20% <97.34%> (+0.12%) ⬆️
joblib/parallel.py 97.24% <97.75%> (+0.17%) ⬆️
joblib/__init__.py 100.00% <100.00%> (ø)

... and 1 file with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

tomMoral
tomMoral reviewed Feb 16, 2023
View reviewed changes
Copy link
Copy Markdown
Contributor

@tomMoral tomMoral left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From a quick scan, the setup looks nice :)
Just a few first comments.

jjerphan
jjerphan reviewed Feb 17, 2023
View reviewed changes
Copy link
Copy Markdown
Contributor

@jjerphan jjerphan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To me, this is a valuable contribution for users. Thank you, @jeremiedbb.

Here is a review from an outsider (I am starting getting around joblib).

@jeremiedbb jeremiedbb changed the title [WIP] FEA parallel_config context manager to allow more fine-grained control FEA parallel_config context manager to allow more fine-grained control Feb 17, 2023
glemaitre
glemaitre reviewed Feb 17, 2023
View reviewed changes
glemaitre
glemaitre reviewed Feb 17, 2023
View reviewed changes
@ogrisel ogrisel mentioned this pull request Feb 18, 2023
Open
ogrisel
ogrisel reviewed Feb 18, 2023
View reviewed changes
@glemaitre glemaitre mentioned this pull request Mar 23, 2023
Closed
@ogrisel
Copy link
Copy Markdown
Contributor

ogrisel commented Apr 19, 2023

@jeremiedbb There are a few conflicts to resolve after the merge of the return_generator branch but it does not look too complicated.

tomMoral
tomMoral approved these changes Apr 21, 2023
View reviewed changes
Copy link
Copy Markdown
Contributor

@tomMoral tomMoral left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did a pass but the code looks super nice!! Good job @jeremiedbb :)

tomMoral
tomMoral approved these changes Apr 26, 2023
View reviewed changes
Copy link
Copy Markdown
Contributor

@tomMoral tomMoral left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! thanks @jeremiedbb for making this :)

@tomMoral tomMoral merged commit ebacd8c into joblib:master Apr 26, 2023
This was referenced Apr 26, 2023
Open
Open
@trendelkampschroer
Copy link
Copy Markdown

trendelkampschroer commented Apr 26, 2023

Thanks a lot for this feature.

@larsoner larsoner mentioned this pull request Jun 30, 2023
Closed
@man-shu man-shu mentioned this pull request Feb 13, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ogrisel ogrisel ogrisel left review comments

@tomMoral tomMoral tomMoral approved these changes

@glemaitre glemaitre glemaitre left review comments

+1 more reviewer

@jjerphan jjerphan jjerphan left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Pass verbose through parallel_backend Pass max_nbytes through parallel_backend

6 participants

@jeremiedbb @ogrisel @trendelkampschroer @tomMoral @glemaitre @jjerphan