[MRG] numpy_pickle: several enhancements by aabadie · Pull Request #626 · joblib/joblib

aabadie · 2018-02-01T10:30:28Z

This PR fixes #583 by extending and improving the compression/decompression mechanism used by the dump/load functions. Here is what has changed:

move compressor file object and related things in a dedicated compressor.py module
add feature to extend the available compressors: use register_compressor_file() function
add LZ4 ~~and snappy~~ compressor by default if it's installed
update the dump function signature: now it accepts compress parameter passed as string

Some words about the register_compress_file: the idea is to register a compressor based on a name, the file prefix (magic number of the compression format) and a file-like object. After that, dump can be used with compress='<compressor name>' as parameter. Passing a tuple parameter (<compressor name>, <level>) can also be used but the compression level will not be taken into account. This is a limitation of the current implementation: it's not possible to finely tweak registered compressor when they are used.

How to use it with a CompressorFile object:

from external_module import CompressorFile
from joblib import register_compressor_file, dump, load, Memory

file_prefix = b'FilePrefix'

# Register the new compressor
register_compressor_file('new_compressor', file_prefix, CompressorFile)

# Then use it with joblib dump/load/Memory:
data = 'test data'
filename = '/tmp/test.pkl'
dump(data, filename, compress='new_compressor')
load(filename)

# Use the new compressor to cache computationally intensive results with Memory:
mem = Memory('/tmp/joblib-cache', compress='new_compressor')
func = mem.cache(func)

Otherwise, if one has lz4 installed, this compressor can be used directly without the need to register it:

data = 'test data'
filename = '/tmp/test.pkl'

# Use LZ4 compression
dump(data, filename, compress='lz4')
load(filename)

# Use LZ4 to cache computationally intensive results with Memory:
mem = Memory('/tmp/joblib-cache', compress='lz4')
func = mem.cache(func)

Later potential improvements:

update the benchmark scripts with the new compressors
~~add an example about compressors in the gallery, maybe this gist could be adapted with real world data~~
~~improve reloading of lz4 compressed pickle when it's not available~~
~~allow fine tuning of registered compressors~~

cc @ogrisel @lsorber @GaelVaroquaux @lesteve if you want to have a look.

aabadie · 2018-02-01T10:33:14Z

Ok, the CI needs some C libraries to be installed, marking this PR as WIP...

ogrisel · 2018-02-01T10:39:18Z

Real-world benchmark data could be the adult census data parsed as a pandas dataframe:

import pandas as pd
import os
from urllib.request import urlretrieve

url = ("https://archive.ics.uci.edu/ml/machine-learning-databases"
       "/adult/adult.data")
local_filename = os.path.basename(url)
if not os.path.exists(local_filename):
    print("Downloading Adult Census datasets from UCI")
    urlretrieve(url, local_filename)
names = ("age, workclass, fnlwgt, education, education-num, "
         "marital-status, occupation, relationship, race, sex, "
         "capital-gain, capital-loss, hours-per-week, "
         "native-country, income").split(', ')    
data = pd.read_csv(local_filename, names=names)

This is a mixed types datastructure with both categorical and numerical values.

aabadie · 2018-02-01T11:42:44Z

I updated the gist bench with the Adult Census datasets and KDDCUP99 (which is rather big, 2GB uncompressed). The benches were run on a laptop with an i7 and a SSD.
Here are the results:

Data: np.ones((10000, 10000)

strategy	dump	dump ratio	load	load ratio	size	size ratio
No compression	1.94s	1.00	0.43s	1.00	762.94MB	1.00
Snappy	2.79s	0.69	3.03s	0.14	36.00MB	21.20
LZ4	0.18s	11.04	0.83s	0.52	3.19MB	239.17
GZip	2.48s	0.78	1.75s	0.25	4.07MB	187.51
Zlib	2.12s	0.91	1.40s	0.31	4.07MB	187.51

Data: np.arange(100000000)

strategy	dump	dump ratio	load	load ratio	size	size ratio
No compression	1.58s	1.00	0.43s	1.00	762.94MB	1.00
Snappy	4.14s	0.38	4.07s	0.11	381.80MB	2.00
LZ4	2.79s	0.57	2.03s	0.21	381.75MB	2.00
GZip	12.83s	0.12	2.74s	0.16	143.82MB	5.30
Zlib	12.45s	0.13	2.37s	0.18	143.82MB	5.30

Data: np.random((10000, 10000))

strategy	dump	dump ratio	load	load ratio	size	size ratio
No compression	1.75s	1.00	0.42s	1.00	762.94MB	1.00
Snappy	3.77s	0.46	3.61s	0.12	763.03MB	1.00
LZ4	2.36s	0.74	1.59s	0.27	762.99MB	1.00
GZip	29.98s	0.06	5.83s	0.07	723.54MB	1.05
Zlib	29.59s	0.06	5.46s	0.08	723.54MB	1.05

Data: Pandas dataframe (Adult Census from UCI)

strategy	dump	dump ratio	load	load ratio	size	size ratio
No compression	0.10s	1.00	0.02s	1.00	4.11MB	1.00
Snappy	0.04s	2.62	0.03s	0.47	1.10MB	3.72
LZ4	0.02s	4.12	0.03s	0.59	1.22MB	3.38
GZip	0.08s	1.24	0.03s	0.50	0.65MB	6.34
Zlib	0.08s	1.26	0.03s	0.57	0.65MB	6.34

Data: Pandas dataframe (KDDCUP99)

strategy	dump	dump ratio	load	load ratio	size	size ratio
No compression	4.87s	1.00	1.62s	1.00	2270.92MB	1.00
Snappy	8.06s	0.60	9.68s	0.17	130.96MB	17.34
LZ4	1.09s	4.46	4.15s	0.39	49.25MB	46.11
GZip	9.14s	0.53	5.71s	0.28	26.46MB	85.84
Zlib	7.60s	0.64	4.64s	0.35	26.46MB	85.84

aabadie · 2018-02-01T12:07:57Z

Updated the bench results. The results on KDDCUP99 show that LZ4 is by far the most efficient at dump time.

ogrisel · 2018-02-01T12:10:31Z

Very nice!

ogrisel · 2018-02-01T12:13:47Z

In your benchmark code you should also report additional columns for ratios:

dump speedup (time uncompressed / time compressed)
load speedup (same)
size reduction ratio (size uncompressed / size compressed)

aabadie · 2018-02-01T12:18:49Z

you should also report additional columns for ratios

ok

aabadie · 2018-02-01T12:33:52Z

Edited with the ratios

codecov · 2018-02-01T12:47:20Z

Codecov Report

Merging #626 into master will decrease coverage by 0.02%.
The diff coverage is 95.31%.

@@            Coverage Diff            @@
##           master    #626      +/-   ##
=========================================
- Coverage   95.13%   95.1%   -0.03%     
=========================================
  Files          39      40       +1     
  Lines        5484    5663     +179     
=========================================
+ Hits         5217    5386     +169     
- Misses        267     277      +10

Impacted Files	Coverage Δ
joblib/numpy_pickle.py	`98.52% <100%> (+0.04%)`	⬆️
joblib/numpy_pickle_utils.py	`93.54% <100%> (+0.25%)`	⬆️
joblib/test/common.py	`88.13% <100%> (+1.59%)`	⬆️
joblib/test/test_numpy_pickle.py	`97.82% <100%> (-1.22%)`	⬇️
joblib/__init__.py	`100% <100%> (ø)`	⬆️
joblib/test/test_numpy_pickle_utils.py	`100% <100%> (ø)`	⬆️
joblib/compressor.py	`93.2% <93.2%> (ø)`
joblib/test/test_parallel.py	`96.09% <0%> (ø)`	⬆️
... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0bc239b...241b34a. Read the comment docs.

lsorber · 2018-02-01T13:37:15Z

Would this support Blosc too?

aabadie · 2018-02-01T13:52:44Z

Would this support Blosc too?

If one can provide an external file-like buffered object that uses blosc for compression, then yes. You'll just have to register the new compressor.
The problem is that I don't know such existing implementation.

aabadie · 2018-02-01T14:08:48Z

Update on this PR:

CIs now pass, even on Windows
snappy removed: lz4 is always better and it requires non standard system libraries being installed (require apt-get for example). lz4 installation is straight forward using pip.

This PR ready for review.

ogrisel

Here is a short review. I think the documentation should be updated and you should also add or update an example to show the tradeoffs of common compressors (no compression, lz4, zlib, lzma): dump / load speed, disk usage, ability to mmap, standard library vs additional dependencies...

ogrisel · 2018-02-06T09:57:36Z

joblib/compressor.py

+
+
+def register_compressor_file(compressor_name, file_prefix, file_object,
+                             force=False):


Please rename this function to register_compressor_file as we do not want to register a "file" but a compression algorithm to be made available to dump. The fact that it implements the file object API is a detail.

Please rename this function to register_compressor_file

Do you mean register_compressor ?

ogrisel · 2018-02-06T09:58:14Z

joblib/compressor.py

+
+def register_compressor_file(compressor_name, file_prefix, file_object,
+                             force=False):
+    """Register a compressor file object."""


"""Register a compressor implementing the traditional Python file object API."""

ogrisel · 2018-02-06T09:58:51Z

joblib/compressor.py

+    wbits = 31  # zlib compressor/decompressor wbits value for gzip format.
+
+
+def register_compressor_file(compressor_name, file_prefix, file_object,


Please rename file_object to compressor.

ogrisel · 2018-02-06T10:00:53Z

joblib/compressor.py

+
+    if file_object is None or not issubclass(file_object, io.BufferedIOBase):
+        raise ValueError("File object should inherit io.BufferedIOBase, "
+                         "'{}' given.".format(file_object))


Instead of checking inheritance explicitly, I think we should do a duck type check: make sure that the object has the methods we need (write, read, ...) and leave the implementers the freedom to no inherit from io.BufferedIOBase if they want.

ogrisel · 2018-02-06T10:10:14Z

joblib/numpy_pickle.py

+    if compress_method == 'lz4' and lz4 is None:
+        raise ValueError('LZ4 in not installed. Consider installing it '
+                         'from PyPI: https://pypi.python.org/pypi/lz4 '
+                         'using: "pip install lz4"')


Let's stay neutral w.r.t. the installation method (pip, conda, debian...), it's up to the user to decide.

raise ValueError('LZ4 in not installed. Consider installing it: ' 'http://python-lz4.readthedocs.io/')

ogrisel · 2018-02-06T10:18:32Z

Please also add an entry to CHANGES.

aabadie · 2018-02-06T15:38:54Z

@ogrisel, comments addressed. I added an example in the gallery, adapted from the gist mentioned above. Comments welcome !

ogrisel

I think the example should be simplified to be more readable and demonstrate code that the reader is likely to write. In particular, our users will not write a compression benchmark framework.

So to simplify the example you should select a single dataset and demonstrate the dump and load times of uncompressed, gzip and lz4 sequentially in the main flow of the example (not in a benchmark loop) and comment the results (gzip is reasonably fast and compress well, lz4 is much faster but compress less than gzip)...

ogrisel · 2018-02-06T17:54:50Z

doc/persistence.rst

+  >>> joblib.load(filename + '.lz4')
+  [('a', [1, 2, 3]), ('b', array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]))]
+
+The LZ4 compression in Joblib relies on the ``lz4.frame.LZ4File`` file object.


I think the users don't care about this implementation detail. Those advanced users who really want to know about this will naturally read the source code. What the median users care about is when is why would they want to use LZ4 instead of gzip. You can say that while LZ4 does not compress as much as gzip, LZ4 is a very fast both when dumping and loading data and therefore often results in shorter overall load and dump times than with uncompressed data even on SSD drives.

GaelVaroquaux · 2018-02-07T09:21:45Z

I think the example should be simplified to be more readable and demonstrate code that the reader is likely to write. In particular, our users will not write a compression benchmark framework.

One option is to write two example: one that demonstrates simple usage, and another that does a benchmark.

aabadie · 2018-03-07T17:02:20Z

@ogrisel, I tried to simplify the example as we discussed IRL. I think the plots could be improved, any help/good advice will be appreciated :)

aabadie · 2018-03-08T08:36:42Z

@ogrisel, I finally went for the bar chart for comparing the results, and the resulting plots are nicer IMHO.

The generated example can be seen here

GaelVaroquaux · 2018-03-09T06:43:15Z

examples/compressors_comparison.py

+#   The compression format is identified by the standard magic number present
+#   at the beginning of the file. Joblib uses this information to determine
+#   the compression method used.
+#   This is the case for all compression methods supported by Joblib.


This does not seem like valid rst: the indentation after "note" is strange. But maybe I am wrong. Did you check that it renders right?

I pushed an update with the valid rst: much better rendering indeed ! Thanks.

GaelVaroquaux · 2018-03-09T06:48:51Z

I haven't looked at the details of this PR, but the general direction is awesome! Thanks a lot!

ogrisel

This is really going in the right direction. A few more things to improve though :)

ogrisel · 2018-03-12T08:39:24Z

examples/compressors_comparison.py

+
+###############################################################################
+# The compression level is using the default value, 3, which is in general the
+# best compromise between good compression and speed.


which is, in general, a good compromise between compression and speed.

ogrisel · 2018-03-12T08:47:58Z

examples/compressors_comparison.py

+
+###############################################################################
+# Then measure the size of the raw dumped data on disk:
+raw_file_size = os.stat(pickle_file).st_size / 2**20


1 MB is a 1e6 bytes. Mega is standardized to always be a prefix for 1 million in the international unit system. It makes things simpler.

Multiples of 1024 are for KiB (kibibyte), MiB (mebibyte) and so on. We don't care about those. Leave powers of 1024 to the hard drive manufacturers from the 90's and just use decimal powers to keep things simpler.

Also please add

print("Raw dump file size: %0.3f MB" % raw_file_size)

ogrisel · 2018-03-12T08:49:51Z

examples/compressors_comparison.py

+start = time.time()
+with open(pickle_file, 'wb') as f:
+    dump(data, f)
+raw_dump_duration = time.time() - start


Please print the measurements along the way to make linear reading of this example more interesting.

print("Raw dump duration: %0.3fs" % raw_dump_duration)

ogrisel · 2018-03-12T08:51:47Z

examples/compressors_comparison.py

+local_filename = os.path.basename(url)
+urlretrieve(url, local_filename)
+
+data = pd.read_csv(local_filename, names=names)


Please do:

data = pd.read_csv(url, name=names)

to make the code simpler and having to delete a temporary file at the end of the notebook.

ogrisel · 2018-03-12T08:57:00Z

joblib/compressor.py

+
+    If mode is 'wb', compresslevel can be a number between 1
+    and 9 specifying the level of compression: 1 produces the least
+    compression, and 9 (default) produces the most compression.


Why use 9 as the default? I think we tend to use 3 elsewhere. Better stay consistent and use 3 everywhere in joblib unless there is a good reason not to do so.

Since I just copy pasted this from the numpy_pickle_utils module, this was here from the beginning I think. I changed that.

ogrisel · 2018-03-12T09:41:43Z

doc/persistence.rst

+If the ``lz4`` package is installed, this compression method is automatically
+available with the dump function.
+
+  >>> joblib.dump(to_persist, filename + '.lz4', compress='lz4')  # doctest: +ELLIPSIS


There is no need to add compress='lz4' right? I think using the ".lz4" extension should be explicit enough to automatically select the lz4 compressor.

Actually, I just tried and lz4 is not automatically used when the filename ends with the ".lz4" extensions. I think this should be the case by default.

ogrisel · 2018-03-12T10:19:08Z

joblib/test/test_numpy_pickle.py

+    # Check that lz4 cannot be used when dependency is not available.
+    fname = tmpdir.join('test.no.lz4').strpath
+    data = 'test data'
+    compressor = 'lz4'


The test would be more readable by not defining this local variable and using the literatl compressor='lz4' directly in the code.

ogrisel · 2018-03-12T10:20:17Z

joblib/test/test_numpy_pickle.py

+    with raises(ValueError) as excinfo:
+        numpy_pickle.dump(data, fname, compress=compressor)
+    excinfo.match('LZ4 in not installed. Consider installing it: '
+                  'http://python-lz4.readthedocs.io/')


Please add another check without setting the compress kwarg:

with raises(ValueError) as excinfo: # fname ends with .lz4 numpy_pickle.dump(data, fname) excinfo.match('LZ4 in not installed. Consider installing it: ' 'http://python-lz4.readthedocs.io/')

ogrisel · 2018-03-12T10:21:27Z

joblib/test/test_numpy_pickle.py

+    data = 'test data'
+    numpy_pickle.dump(data, fname, compress=compressor)
+
+    assert open(fname, 'rb').read(len(_LZ4_PREFIX)) == _LZ4_PREFIX


Please with the with statement to close the file explicitly:

with open(fname, 'rb') as f: assert f.read(len(_LZ4_PREFIX)) == _LZ4_PREFIX

lesteve · 2018-03-12T10:49:52Z

.readthedocs-requirements.txt

 matplotlib
 pillow
 sphinx-gallery
+pandas


Before I forget it would be to mention in the README.rst that pandas is needed to run some examples.

The notes about building the docs already mention that some dependencies are required and gives the command to install them: https://github.com/joblib/joblib#building-the-docs.

I am sure you are aware that joblib users may want to run the examples without building the doc. They may even be completely unaware that both are related. Add a sentence after the optional numpy dependency and/or look at how scikit-learn does it.

I have seen this has been addressed, sorry for the noise.

aabadie · 2018-03-13T08:07:12Z

@ogrisel @lesteve I think I addressed your last comments. I'm not sure how to mention the dependency to pandas/lz4 in a nice way in the Readme though.

aabadie · 2018-06-08T21:02:43Z

comments addressed @ogrisel

ogrisel · 2018-06-13T06:55:55Z

Merged! Thanks @aabadie.

Now that the infrastructure is in place, you might want to add support for zstd:

https://facebook.github.io/zstd/
pip install zstandard

It's uniformly better than most other compression algorithms in the compression ratio vs speed tradeoff (except for LZ4 that is the fastest of all at the cost of worse compression ratio).

aabadie · 2018-06-13T07:16:03Z

Thanks for merging @ogrisel !
Indeed zstd looks promising, just needs to provide a file-like object on top of it but this looks rather simple since its compressor object supports streaming.

GaelVaroquaux · 2018-06-13T08:09:59Z

Exciting! The next joblib release is going to be a killer!

tomMoral · 2018-06-14T09:24:58Z

The test are failing when lz4 is not installed (e.g. here ). The test test_joblib_compression_formats is not properly skiped when lz4 is not installed. I opened #695 to fix this.

jakirkham · 2018-06-22T21:41:32Z

Yeah, zstd is an interesting one as it supports learning a dictionary for compression, which can make it viable for compressing even small pieces of data well (assuming that data is relatively similar).

ref: https://github.com/facebook/zstd#the-case-for-small-data-compression

* tag '0.12': (116 commits) Release 0.12 typo typo typo ENH add initializer limiting n_threads for C-libs (joblib#701) DOC better parallel docstring (joblib#704) [MRG] Nested parallel call thread bomb mitigation (joblib#700) MTN vendor loky2.1.3 (joblib#699) Make it possible to configure the reusable executor workers timeout (joblib#698) MAINT increase timeouts to make test more robust on travis DOC: use the .joblib extension instead of .pkl (joblib#697) [MRG] Fix exception handling in nested parallel calls (joblib#696) Fix skip test lz4 not installed (joblib#695) [MRG] numpy_pickle: several enhancements (joblib#626) Introduce Parallel.__call__ backend callbacks (joblib#689) Add distributed on readthedocs (joblib#686) Support registration of external backends (joblib#655) [MRG] Add a dask.distributed example (joblib#613) ENH use cloudpickle to pickle interactively defined callable (joblib#677) CI freeze the version of sklearn0.19.1 and scipy1.0.1 (joblib#685) ...

* releases: (121 commits) Release 0.12.1 fix kwonlydefaults key error in filter_args (joblib#715) MNT fix some "undefined name" flake8 warnings (joblib#713). from importlib import reload for Python 3 (joblib#675) MTN vendor loky2.1.4 (joblib#708) Release 0.12 typo typo typo ENH add initializer limiting n_threads for C-libs (joblib#701) DOC better parallel docstring (joblib#704) [MRG] Nested parallel call thread bomb mitigation (joblib#700) MTN vendor loky2.1.3 (joblib#699) Make it possible to configure the reusable executor workers timeout (joblib#698) MAINT increase timeouts to make test more robust on travis DOC: use the .joblib extension instead of .pkl (joblib#697) [MRG] Fix exception handling in nested parallel calls (joblib#696) Fix skip test lz4 not installed (joblib#695) [MRG] numpy_pickle: several enhancements (joblib#626) Introduce Parallel.__call__ backend callbacks (joblib#689) ...

aabadie changed the title ~~[MRG] numpy_pickle: several enhancements~~ [WIP] numpy_pickle: several enhancements Feb 1, 2018

aabadie force-pushed the snappy_compress branch from 7083482 to bdbffe4 Compare February 1, 2018 12:47

aabadie changed the title ~~[WIP] numpy_pickle: several enhancements~~ [MRG] numpy_pickle: several enhancements Feb 1, 2018

ogrisel requested changes Feb 6, 2018

View reviewed changes

aabadie force-pushed the snappy_compress branch from a9e704d to ef5fdd2 Compare February 6, 2018 13:55

ogrisel requested changes Feb 6, 2018

View reviewed changes

aabadie force-pushed the snappy_compress branch from 6ecbaaf to 7d93a10 Compare February 7, 2018 19:38

aabadie force-pushed the snappy_compress branch from 7d93a10 to 4dca3ea Compare March 7, 2018 16:59

GaelVaroquaux reviewed Mar 9, 2018

View reviewed changes

ogrisel requested changes Mar 12, 2018

View reviewed changes

lesteve reviewed Mar 12, 2018

View reviewed changes

aabadie added 19 commits June 8, 2018 22:50

trying to use the default compresslevel

307b241

fix CI

4396c58

remove useless test data

d14bd42

cleanup with default compress level

87c3779

reorganizing code

78b1f43

cleanup code reorganization

edf4be9

remove useless line

4ddddd2

a bit more cleanup

ff89cad

cleanup and documenting

de922e0

comments

1ed1bdd

move compressors registration to numpy_pickle module

cde01d2

other comments

d0514db

improve test

32b622f

fix test

2dddf93

flake8

42965ba

flake8

3a514c2

fix test

bbcf743

typo

03ec743

addressing (last?) nitpicks

241b34a

aabadie force-pushed the snappy_compress branch from d5771e2 to 241b34a Compare June 8, 2018 21:02

ogrisel merged commit da01924 into joblib:master Jun 13, 2018

aabadie deleted the snappy_compress branch August 8, 2018 21:08



		def register_compressor_file(compressor_name, file_prefix, file_object,
		force=False):

		wbits = 31 # zlib compressor/decompressor wbits value for gzip format.


		def register_compressor_file(compressor_name, file_prefix, file_object,

Conversation

aabadie commented Feb 1, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aabadie commented Feb 1, 2018

Uh oh!

ogrisel commented Feb 1, 2018

Uh oh!

aabadie commented Feb 1, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aabadie commented Feb 1, 2018

Uh oh!

ogrisel commented Feb 1, 2018

Uh oh!

ogrisel commented Feb 1, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aabadie commented Feb 1, 2018

Uh oh!

aabadie commented Feb 1, 2018

Uh oh!

codecov bot commented Feb 1, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

lsorber commented Feb 1, 2018

Uh oh!

aabadie commented Feb 1, 2018

Uh oh!

aabadie commented Feb 1, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ogrisel left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ogrisel commented Feb 6, 2018

Uh oh!

aabadie commented Feb 6, 2018

Uh oh!

ogrisel left a comment

Choose a reason for hiding this comment

Uh oh!

ogrisel Feb 6, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

GaelVaroquaux commented Feb 7, 2018 via email

Uh oh!

aabadie commented Mar 7, 2018

Uh oh!

aabadie commented Mar 8, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

GaelVaroquaux commented Mar 9, 2018

Uh oh!

ogrisel left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ogrisel Mar 12, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

aabadie commented Feb 1, 2018 •

edited

Loading

aabadie commented Feb 1, 2018 •

edited

Loading

ogrisel commented Feb 1, 2018 •

edited

Loading

codecov bot commented Feb 1, 2018 •

edited

Loading

aabadie commented Feb 1, 2018 •

edited

Loading

ogrisel Feb 6, 2018 •

edited

Loading

aabadie commented Mar 8, 2018 •

edited

Loading

ogrisel Mar 12, 2018 •

edited

Loading

lesteve Mar 12, 2018 •

edited

Loading

aabadie commented Mar 13, 2018 •

edited

Loading