Skip to content

naïve way to return a generator instead of a list (not asynchronously)#585

Closed
fcharras wants to merge 1 commit intojoblib:masterfrom
fcharras:gen_results
Closed

naïve way to return a generator instead of a list (not asynchronously)#585
fcharras wants to merge 1 commit intojoblib:masterfrom
fcharras:gen_results

Conversation

@fcharras
Copy link
Copy Markdown
Contributor

This PR implements #217 (* but does does implement #79 *)

This is a naïve way to return the results from Parallel.call: simply yield outputs instead of storing it inside a list.

The default behavior is not changed (the parameter as_generator is set by default to false), but the __call__ now always use a generator on the result, it only transforms it to a list if as_generator = False.

All the unitest are still green so it proves that the generator behaves as expected.

Here the generator return the results in the same order as usual, and not in asynchronously (= the first results returned are the fastest to be processed).

BUT since the main thread can still be used for other computations (rather than reading the generator), it could be combined with #582 to offer an API to the user to read the list of asynchronous results. If we change self.last_async_output in #582 to a list (let's say self.async_output_list) that record all the tasks that have already finished, then at any given time the user could inspect everything.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant