naïve way to return a generator instead of a list (not asynchronously)#585
Closed
fcharras wants to merge 1 commit intojoblib:masterfrom
Closed
naïve way to return a generator instead of a list (not asynchronously)#585fcharras wants to merge 1 commit intojoblib:masterfrom
fcharras wants to merge 1 commit intojoblib:masterfrom
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR implements #217 (* but does does implement #79 *)
This is a naïve way to return the results from Parallel.call: simply yield outputs instead of storing it inside a list.
The default behavior is not changed (the parameter
as_generatoris set by default tofalse), but the__call__now always use a generator on the result, it only transforms it to a list ifas_generator = False.All the unitest are still green so it proves that the generator behaves as expected.
Here the generator return the results in the same order as usual, and not in asynchronously (= the first results returned are the fastest to be processed).
BUT since the main thread can still be used for other computations (rather than reading the generator), it could be combined with #582 to offer an API to the user to read the list of asynchronous results. If we change
self.last_async_outputin #582 to a list (let's sayself.async_output_list) that record all the tasks that have already finished, then at any given time the user could inspect everything.