Parallel Joblib Process Entries#3933
Merged
mkhorton merged 9 commits intomaterialsproject:masterfrom Aug 2, 2024
Merged
Conversation
Member
|
Thanks @CompRhys for the PR. Wondering if we should add |
Contributor
Author
|
Regarding default for n_workers i'm not opposed but I would always want to have it be controllable at the function call and it would be best to make that update in a separate PR imo. Re test failures I have been using this method and works very great for me on 3.12. I don't see these pickle errors when running local tests so not sure which object is responsible. I will try take a look at the weekend. |
…into joblib-process-entries
Contributor
Author
|
Cannot replicate the error locally even building a py39 venv. Traceback (most recent call last):
File "/home/runner/micromamba/envs/pmg/lib/python3.12/site-packages/joblib/externals/loky/process_executor.py", line 426, in _process_worker
call_item = call_queue.get(block=True, timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/micromamba/envs/pmg/lib/python3.12/multiprocessing/queues.py", line 122, in get
return _ForkingPickler.loads(res)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/micromamba/envs/pmg/lib/python3.12/site-packages/monty/design_patterns.py", line 84, in __new__
inst = klass(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^
TypeError: GasCorrection.__init__() missing 1 required positional argument: 'config_file' |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the option to use joblib to apply corrections in parallel, this depended on materialyzeai/monty#691 which was linked to #3898. Opening this PR incase maintainers think it's a useful addition to the code.
Checklist
ruff.mypy.duecredit@due.dcitedecorators to reference relevant papers by DOI (example)