Codecov Report

Merging #594 into master will increase coverage by 0.16%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #594      +/-   ##
==========================================
+ Coverage   94.78%   94.94%   +0.16%     
==========================================
  Files          38       38              
  Lines        4969     4969              
==========================================
+ Hits         4710     4718       +8     
+ Misses        259      251       -8

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 65d0cb2...49afa8d. Read the comment docs.

I think that we should merge this for the moment. It allows others to add their own examples from now.
I will do another PR with all possible examples.

What's the vision in terms of deployment? Does this work on read the docs?

Let's merge this infrastructure and open new PR to add more examples and fix readthedocs if it does not work.

@GaelVaroquaux We can make an independent Circle CI build to check the artefacts.
However, I don't see any issue of keeping readthedocs to build the master documentation.

I am in favor of adding a CircleCI build. Do you think that you can do that, @glemaitre ?

@glemaitre Let's do this: new PR for circleci in PRs, new PR for new gotcha examples.

I am on Circle CI now

I had a look at the docs on readthedocs:
https://joblib.readthedocs.io/en/latest/auto_examples/parallel_memmap.html

They look fine. I don't think that we need to worry about the deployment with CircleCI, just the testing.

Also, as a comment on the example added: it should be simpler and more didactic. We should give up on purity, and not use mkdtemp but a hard coded directory, not use try/finally, and also break up the example in small easy to understand blocks.

The examples should be as simple as possible, because they are how people learn.

Also, as a comment on the example added: it should be simpler and more didactic. We should give up on purity, and not use mkdtemp but a hard coded directory, not use try/finally, and also break up the example in small easy to understand blocks.

The examples should be as simple as possible, because they are how people learn.

Ok I will put on the TODO list. I just converted this example just that it run for the moment.

Other comment (sorry to be spaming this): "python setup.py build_sphinx" crashes on my computer with the following crash:

running build_sphinx
Running Sphinx v1.5.6
loading pickled environment... failed: build environment version not current
[autosummary] generating autosummary for: CHANGES.rst, README.rst, developing.rst, generated/joblib.Memory.rst, generated/joblib.Parallel.rst, generated/joblib.dump.rst, generated/joblib.hash.rst, generated/joblib.load.rst, index.rst, installing.rst, memory.rst, parallel.rst, parallel_numpy.rst, persistence.rst, why.rst
loading intersphinx inventory from https://docs.python.org/3/objects.inv...
loading intersphinx inventory from https://docs.scipy.org/doc/numpy/objects.inv...
loading intersphinx inventory from https://docs.scipy.org/doc/scipy/reference/objects.inv...
generating gallery...
/usr/bin/python3: Error while finding module specification for 'joblib.externals.loky.backend.popen_loky_posix' (ModuleNotFoundError: No module named 'joblib.externals')
/usr/bin/python3: Error while finding module specification for 'joblib.externals.loky.backend.popen_loky_posix' (ModuleNotFoundError: No module named 'joblib.externals')
/usr/bin/python3: Error while finding module specification for 'joblib.externals.loky.backend.popen_loky_posix' (ModuleNotFoundError: No module named 'joblib.externals')
/usr/bin/python3: Error while finding module specification for 'joblib.externals.loky.backend.popen_loky_posix' (ModuleNotFoundError: No module named 'joblib.externals')
exception calling callback for 
Traceback (most recent call last):
  File "/home/varoquau/dev/joblib/joblib/externals/loky/_base.py", line 322, in _invoke_callbacks
    callback(self)
  File "/home/varoquau/dev/joblib/joblib/parallel.py", line 224, in __call__
    self.parallel.dispatch_next()
  File "/home/varoquau/dev/joblib/joblib/parallel.py", line 610, in dispatch_next
    if not self.dispatch_one_batch(self._original_iterator):
  File "/home/varoquau/dev/joblib/joblib/parallel.py", line 636, in dispatch_one_batch
    self._dispatch(tasks)
  File "/home/varoquau/dev/joblib/joblib/parallel.py", line 595, in _dispatch
    job = self._backend.apply_async(batch, callback=cb)
  File "/home/varoquau/dev/joblib/joblib/_parallel_backends.py", line 407, in apply_async
    future = self._workers.submit(SafeFunction(func))
  File "/home/varoquau/dev/joblib/joblib/externals/loky/process_executor.py", line 973, in submit
    raise BrokenProcessPool('A child process terminated abruptly, '
joblib.externals.loky.process_executor.BrokenProcessPool: A child process terminated abruptly, the process pool is not usable anymore

Warning, treated as error:
WARNING: /home/varoquau/dev/joblib/examples/parallel_memmap.py failed to execute correctly: Traceback (most recent call last):
  File "/home/varoquau/dev/joblib/examples/parallel_memmap.py", line 50, in 
    for i in range(samples.shape[0]))
  File "/home/varoquau/dev/joblib/joblib/parallel.py", line 808, in __call__
    self.retrieve()
  File "/home/varoquau/dev/joblib/joblib/parallel.py", line 710, in retrieve
    self._output.extend(job.get(timeout=self.timeout))
  File "/home/varoquau/dev/joblib/joblib/_parallel_backends.py", line 418, in wrap_future_result
    return future.result(timeout=timeout)
  File "/home/varoquau/dev/joblib/joblib/externals/loky/_base.py", line 431, in result
    return self.__get_result()
  File "/home/varoquau/dev/joblib/joblib/externals/loky/_base.py", line 382, in __get_result
    raise self._exception
joblib.externals.loky.process_executor.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

I get the same problem in Python 2 and Python 3. However, simply running the example file works. :$. Let's see if CircleCI reproduces this.

It looks like if joblib is not installed?

I got the same error 20 minutes ago, setting up travis build :)
I would say that the traceback is scary at first :)