Also, it looks like joblib.hash is much slower for DataFrames and sparse arrays than dask.base.tokenize which is also using md5 as far as I understand. (I'm wondering if dask tokenization is valid for sparse arrays though).

In the benchmarks below it is 30x slower for hashing pandas DataFrames and 100x times slower for sparse numpy arrays (though I'm not sure if running benchmarks with multiple iterations of timeit could have an effect here),

import pandas as pd
import numpy as np
import scipy.sparse

from dask.base import tokenize
from sklearn.externals import joblib

df = pd.read_csv('http://www.gagolewski.com/resources/data/titanic3.csv', comment='#')

rng = np.random.RandomState(42)
X = rng.rand(1000, 100)
X_csr = scipy.sparse.rand(1000, 10000, random_state=rng)

for data in [df, X, X_csr]:

    print('# {}, shape={}'.format(type(data).__name__, data.shape))
    print('   joblib.hash          ', end='')
    %timeit joblib.hash(data)
    print('   dask.base.tokenize   ', end='')
    %timeit tokenize(data)

which produces,

# DataFrame, shape=(1309, 14)
   joblib.hash          33.5 ms ± 3.71 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
   dask.base.tokenize   3.67 ms ± 95.9 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
# ndarray, shape=(1000, 100)
   joblib.hash          1.63 ms ± 12.4 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
   dask.base.tokenize   1.45 ms ± 254 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
# csr_matrix, shape=(1000, 10000)
   joblib.hash          3.58 ms ± 781 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
   dask.base.tokenize   26.6 µs ± 2.43 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

Note: these benchmarks were run on Linux with Python 3.6 and joblib 0.11

A problem is that we try to reduce joblib dependencies as much as possible, in particular dependencies that require compilation (outside of numpy) is a big no no.

We could maybe support alternative hashing algorithms if the user accepts to install optional dependencies by themselves

Accepting custom hashing function would quite useful. In particular in the context of scikit-learn, mumurhash is already there...

Faster hashing #343

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions