MNT More replacements of numpy aliases with built-in types by alfaro96 · Pull Request #17707 · scikit-learn/scikit-learn

alfaro96 · 2020-06-24T16:24:13Z

Reference Issues/PRs

Coming from #17687.

What does this implement/fix? Explain your changes.

This PR continues the deprecations of numpy aliases by built-in types.

Any other comments?

This PR was generated by:

$ find . -name "*.py" -print0 | xargs -0 sed -i "s/\bnp.object\b/object/g"

Two manual changes were applied in sklearn/ensemble/_gradient_boosting.pyx and sklearn/neural_network/tests/test_rbm.py.

CC @thomasjpfan @glemaitre @adrinjalali

rth

Thanks @alfaro96 !

glemaitre · 2020-06-24T16:51:01Z

Do we try to correct the one mentioned by Travis in this PR or it is preferable to merge as it is?

alfaro96 · 2020-06-24T17:18:24Z

Do we try to correct the one mentioned by Travis in this PR or it is preferable to merge as it is?

I think it will be better to correct the errors shown by Travis before merging this PR to ensure that everything is properly working.

…kit-learn into replace_deprecated_np_items

alfaro96 · 2020-06-24T20:05:35Z

I have applied the corresponding changes. Let us wait for the green in travis.

alfaro96 · 2020-06-24T20:42:26Z

Travis is failing because of these deprecations coming from pandas.

I think we should ignore these warnings until pandas decides to solve them (similarly to #17652).

glemaitre · 2020-06-24T21:25:25Z

True but it seems that we have some PicklingError :S

alfaro96 · 2020-06-25T08:18:08Z

True but it seems that we have some PicklingError :S

This is what I am worried about 😕.

…kit-learn into replace_deprecated_np_items

alfaro96 · 2020-06-25T10:15:24Z

Pandas has already fixed these warnings (see pandas-dev/pandas#34835). I do not think that it is necessary to silent these warnings.

I think that this PR is ready to merge (we should handle the PicklingError in another PR).

WDYT @glemaitre?

adrinjalali · 2020-06-25T10:27:43Z

Could you maybe explain where the pickling error comes from?

alfaro96 · 2020-06-25T10:46:00Z

Could you maybe explain where the pickling error comes from?

The PicklingError comes from sklearn.cluster.AgglomerativeClustering while trying to cache the output of the computation of the tree.

In particular, the exception raises in lines:

scikit-learn/sklearn/cluster/_agglomerative.py

Lines 883 to 886 in 41488fc

    
           out = memory.cache(tree_builder)(X, connectivity=connectivity, 
        
                                            n_clusters=n_clusters, 
        
                                            return_distance=return_distance, 
        
                                            **kwargs)

while sklearn/cluster/tests/test_hierarchical.py::test_agglomerative_clustering test is executed.

The exact error is:

 _pickle.PicklingError: ("Can't pickle <class 'numpy.dtype[float64]'>: it's not found as numpy.dtype[float64]", 'PicklingError while hashing {\'**\': {\'connectivity\': <100x100 sparse matrix of type \'<class \'numpy.int64\'>\'\n\twith 460 stored elements in COOrdinate format>, \'n_clusters\': None, \'return_distance\': False}, \'*\': [array([[ 1.76405235,  0.40015721,  0.97873798, ...,  0.77749036,\n        -1.61389785, -0.21274028],\n       [-0.89546656,  0.3869025 , -0.51080514, ...,  1.78587049,\n         0.12691209,  0.40198936],\n       [ 1.8831507 , -1.34775906, -1.270485  , ...,  1.11701629,\n        -1.31590741, -0.4615846 ],\n       ...,\n       [ 0.94853292,  1.23127609, -0.28818653, ...,  1.59242571,\n        -0.04144268,  1.37594158],\n       [ 1.35560413,  0.62532518,  1.67550176, ...,  0.52308533,\n         1.73676704,  0.50865349],\n       [ 1.16503885,  0.71312032,  1.3199769 , ...,  0.92918181,\n         0.22941801,  0.41440588]])]}: PicklingError("Can\'t pickle <class \'numpy.dtype[float64]\'>: it\'s not found as numpy.dtype[float64]")')

adrinjalali · 2020-06-25T11:30:58Z

Does the

PicklingError("Can\'t pickle <class \'numpy.dtype[float64]\'>: it\'s not found as numpy.dtype[float64]")')

error make sense to you? Besides, It doesn't seem to be relevant to this PR since you're changing the object ones, right?

alfaro96 · 2020-06-25T11:38:05Z

Does the
PicklingError("Can\'t pickle <class \'numpy.dtype[float64]\'>: it\'s not found as numpy.dtype[float64]")')
error make sense to you? Besides, It doesn't seem to be relevant to this PR since you're changing the object ones, right?

IMHO, this error does not make sense (why <class numpy.dtype[float64]> is not pickable as numpy.dtype[float64] 😕?).

I have applied a few of manual changes apart from np.object to object (specified in the description) but neither related with the sklearn.cluster module.

Nevertheless, I would like to investigate this issue and a few more related with the Linux_Nightly pylatest_pip_scipy_dev job to keep the CRON jobs properly working.

adrinjalali · 2020-06-25T11:47:37Z

right now the best thing to do is to isolate the issue and figure out what exactly the issue is. And if it's not related to your changes (which I suspect is the case) then we can safely merge this PR. The issue right now is that we're in the dark.

Alternatively, if you confirm that the error exists on master, then we could also merge this and tackle it in a separate PR.

alfaro96 · 2020-06-25T11:56:33Z

right now the best thing to do is to isolate the issue and figure out what exactly the issue is. And if it's not related to your changes (which I suspect is the case) then we can safely merge this PR. The issue right now is that we're in the dark.

Alternatively, if you confirm that the error exists on master, then we could also merge this and tackle it in a separate PR.

You are absolutely right, the issue seems quite strange and we need to check if the error comes from this PR or also exists on master.

I will open a PR with the version of the master and check if this issue happens in travis and Linux_Nightly pylatest_pip_scipy_dev.

adrinjalali · 2020-06-25T15:19:47Z

Ok this is becoming tricky (see #17719). I'm happy to merge this one and have the pickling issue investigated after.

adrinjalali · 2020-06-25T15:20:22Z

WDYT @glemaitre

alfaro96 · 2020-06-26T08:04:25Z

Bad news. PicklingError is issued on master for the Linux_Nightly pylatest_pip_scipy_dev job (lines 16924-17076 in https://dev.azure.com/scikit-learn/d9c6f05f-7f58-4a17-b143-2a4e7ff015af/_build/results?buildId=18821&view=logs&jobId=dfe99b15-50db-5d7b-b1e9-4105c42527cf).

This error is not related with this PR, but needs further investigation. I will try to have a look and determine why this is happening.

adrinjalali

Good then, let's merge this one. It'd be really nice if you could look into the pickling error.

glemaitre · 2020-06-26T11:17:18Z

pickling make me think that something is happening with cloudpickle maybe. ping @ogrisel :)

lesteve · 2020-06-26T14:36:22Z

pickling make me think that something is happening with cloudpickle maybe. ping @ogrisel :)

From what I can see, the error seems to be related to joblib.Memory argument hashing and its special dtype treatment:
https://github.com/joblib/joblib/blob/ac0f1528ffec9cd53cabbafaca887ac880697862/joblib/hashing.py#L222-L235

This comes from the fact that np.dtype('float64').__class__ is not picklable in numpy dev:

import pickle

import numpy as np


print('numpy version:', np.__version__)
dt = np.dtype('float64')
print('class: ', dt.__class__)
try:
    pickle.dumps(dt.__class__)
    print('Can pickle dt.__class__')
except Exception as exc:
    print('Exception while pickling dt.__class__')
    print(exc)

Output numpy-dev:

numpy version: 1.20.0.dev0+be8ab91
class:  <class 'numpy.dtype[float64]'>
Exception while pickling dt.__class__
Can't pickle <class 'numpy.dtype[float64]'>: attribute lookup dtype[float64] on numpy failed

Ouptut numpy 1.19.0:

numpy version: 1.19.0
class:  <class 'numpy.dtype'>
Can pickle dt.__class__

alfaro96 · 2020-06-26T15:08:35Z

pickling make me think that something is happening with cloudpickle maybe. ping @ogrisel :)

From what I can see, the error seems to be related to joblib.Memory argument hashing and its special dtype treatment:
https://github.com/joblib/joblib/blob/ac0f1528ffec9cd53cabbafaca887ac880697862/joblib/hashing.py#L222-L235

This comes from the fact that np.dtype('float64').__class__ is not picklable in numpy dev:
import pickle

import numpy as np


print('numpy version:', np.__version__)
dt = np.dtype('float64')
print('class: ', dt.__class__)
try:
    pickle.dumps(dt.__class__)
    print('Can pickle dt.__class__')
except Exception as exc:
    print('Exception while pickling dt.__class__')
    print(exc)
Output numpy-dev:
numpy version: 1.20.0.dev0+be8ab91
class:  <class 'numpy.dtype[float64]'>
Exception while pickling dt.__class__
Can't pickle <class 'numpy.dtype[float64]'>: attribute lookup dtype[float64] on numpy failed
Ouptut numpy 1.19.0:
numpy version: 1.19.0
class:  <class 'numpy.dtype'>
Can pickle dt.__class__

Thanks @lesteve for investigating this!

I will submit a PR and trigger the nightly jobs to check if these errors appear with the master version (using numpy==1.19.0).

If so, I would say that there is no much left we can do to solve this issue, just to wait until numpy makes np.float64 picklable.

lesteve · 2020-06-26T15:10:36Z

FYI I have opened numpy/numpy#16692 to see whether that was an intended change from numpy.

lesteve · 2020-06-26T15:13:53Z

If so, I would say that there is no much left we can do to solve this issue, just to wait until numpy makes np.float64 picklable.

Just for completeness the thing that is not picklable in numpy development version is numpy.dtype('float64').__class__ (and we were relying on it being picklable in joblib for working around some quirks of numpy.dtype pickling)

alfaro96 · 2020-06-26T15:17:40Z

If so, I would say that there is no much left we can do to solve this issue, just to wait until numpy makes np.float64 picklable.

Just for completeness the thing that is not picklable in numpy development version is numpy.dtype('float64').__class__ (and we were relying on it being picklable in joblib for working around some quirks of numpy.dtype pickling)

You are absolutely right. Thanks for the correction!

…arn#17707) * MNT More replacements of numpy aliases with built-in types [scipy-dev] * MNT More (manual) replacements of numpy aliases * MNT More (manual) replacements of numpy aliases * FIX Minor change * MNT Trigger build [scipy-dev] * FIX Minor change * MNT Trigger build [scipy-dev] * FIX More deprecations of numpy aliases [scipy-dev] * MNT Silent numpy aliases warnings [scipy-dev] * FIX Fix silent deprecations * Trigger build [scipy-dev] * FIX Add scape characters [scipy-dev] * FIX Remove package specification from silent * Trigger build [scipy-dev] * Revert

MNT More replacements of numpy aliases with built-in types [scipy-dev]

f07d757

rth approved these changes Jun 24, 2020

View reviewed changes

alfaro96 added 7 commits June 24, 2020 21:47

MNT More (manual) replacements of numpy aliases

e3911a2

MNT More (manual) replacements of numpy aliases

78dffa1

FIX Minor change

98f2f51

MNT Trigger build [scipy-dev]

06c2b3f

FIX Minor change

8a95641

Merge branch 'replace_deprecated_np_items' of github.com:alfaro96/sci…

9dad6b5

…kit-learn into replace_deprecated_np_items

MNT Trigger build [scipy-dev]

6ce127b

FIX More deprecations of numpy aliases [scipy-dev]

f90a858

MNT Silent numpy aliases warnings [scipy-dev]

7e8f4e2

FIX Fix silent deprecations

ac393a4

alfaro96 added 8 commits June 25, 2020 10:20

Trigger build [scipy-dev]

d96636f

FIX Add scape characters [scipy-dev]

69b8d8f

FIX Remove package specification from silent

f604420

Trigger build [scipy-dev]

10a6c87

Merge branch 'master' into replace_deprecated_np_items

c75fdc7

Revert

6db10d9

Solve merge conflicts

16bd270

Merge branch 'replace_deprecated_np_items' of github.com:alfaro96/sci…

ec9e540

…kit-learn into replace_deprecated_np_items

alfaro96 mentioned this pull request Jun 25, 2020

[NOMRG] Check if pickle error issues on master #17719

Closed

adrinjalali approved these changes Jun 26, 2020

View reviewed changes

adrinjalali merged commit 547ead6 into scikit-learn:master Jun 26, 2020

alfaro96 deleted the replace_deprecated_np_items branch June 26, 2020 12:26

lesteve mentioned this pull request Jun 26, 2020

np.dtype('float64').__class__ not picklable in numpy master numpy/numpy#16692

Closed

lesteve mentioned this pull request Jul 1, 2020

Hashing : possible work-around for avoiding to rely on np.dtype('float64').__class__ being picklable joblib/joblib#1080

Closed

alfaro96 mentioned this pull request Oct 30, 2020

[MRG] CI fix some nightly job failures #18711

Merged

alfaro96 mentioned this pull request Jan 25, 2021

MNT silence warning dev [scipy-dev] #19267

Merged

Uh oh!

Conversation

alfaro96 commented Jun 24, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reference Issues/PRs

What does this implement/fix? Explain your changes.

Any other comments?

Uh oh!

rth left a comment

Choose a reason for hiding this comment

Uh oh!

glemaitre commented Jun 24, 2020

Uh oh!

alfaro96 commented Jun 24, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alfaro96 commented Jun 24, 2020

Uh oh!

alfaro96 commented Jun 24, 2020

Uh oh!

glemaitre commented Jun 24, 2020

Uh oh!

alfaro96 commented Jun 25, 2020

Uh oh!

alfaro96 commented Jun 25, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adrinjalali commented Jun 25, 2020

Uh oh!

alfaro96 commented Jun 25, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adrinjalali commented Jun 25, 2020

Uh oh!

alfaro96 commented Jun 25, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adrinjalali commented Jun 25, 2020

Uh oh!

alfaro96 commented Jun 25, 2020

Uh oh!

adrinjalali commented Jun 25, 2020

Uh oh!

adrinjalali commented Jun 25, 2020

Uh oh!

alfaro96 commented Jun 26, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adrinjalali left a comment

Choose a reason for hiding this comment

Uh oh!

glemaitre commented Jun 26, 2020

Uh oh!

lesteve commented Jun 26, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alfaro96 commented Jun 26, 2020

Uh oh!

lesteve commented Jun 26, 2020

Uh oh!

lesteve commented Jun 26, 2020

Uh oh!

alfaro96 commented Jun 26, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

alfaro96 commented Jun 24, 2020 •

edited

Loading

alfaro96 commented Jun 24, 2020 •

edited

Loading

alfaro96 commented Jun 25, 2020 •

edited

Loading

alfaro96 commented Jun 25, 2020 •

edited

Loading

alfaro96 commented Jun 25, 2020 •

edited

Loading

alfaro96 commented Jun 26, 2020 •

edited

Loading

lesteve commented Jun 26, 2020 •

edited

Loading