Fix for OvR partial_fit in various edge cases by kaichogami · Pull Request #6239 · scikit-learn/scikit-learn

kaichogami · 2016-01-27T18:04:20Z

There are many corner cases when not all classes are present in partial_fit call that needs to be addressed.

partial_fit works perfectly fine in cases where target classes are present in each mini-batch iteration. Although when some classes are missing in some mini batch, ovr.predict(X) gives unpredictable results.

X = np.random.randn(14, 2)
y = [0, 0, 1, 1, 2, 3, 3, 0, 1, 2, 3, 0, 2, 3]
ovr = OneVsRestClassifier(SGDClassifier())
ovr.partial_fit(X[:7], y[:7], np.unique(y))
ovr.partial_fit(X[7:], y[7:])
pred = ovr.predict(X)
ovr1 = OneVsRestClassifier(SGDClassifier())
pred1 = ovr1.fit(X, y).predict(X)

In [46]: np.mean(pred==y)
Out[46]: 0.35714285714285715

In [47]: np.mean(pred1==y)
Out[47]: 0.5

I think they should give the same result, like they do when all target classes are present in each iteration of mini-batch.

When mini-batch contain binary classes but all classes are not binary, ovr.predict(X) will give an error as self.label_binarizer_.y_type_ is not multiclass.

ovr = OneVsRestClassifier(SGDClassifier())
X, y = iris.data, iris.target
ovr.partial_fit(X[:60], y[:60], np.unique(y))
ovr.partial_fit(X[60:], y[60:])
In [54]: pred = ovr.predict(X)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-54-48d087b7f7f3> in <module>()
----> 1 pred = ovr.predict(X)

/home/kaichogami/codes/development_scikit-learn/scikit-learn/sklearn/multiclass.py in predict(self, X)
    307             indicator = sp.csc_matrix((data, indices, indptr),
    308                                       shape=(n_samples, len(self.estimators_)))
--> 309             return self.label_binarizer_.inverse_transform(indicator)
    310 
    311     def predict_proba(self, X):

/home/kaichogami/codes/development_scikit-learn/scikit-learn/sklearn/preprocessing/label.pyc in inverse_transform(self, Y, threshold)
    369         else:
    370             y_inv = _inverse_binarize_thresholding(Y, self.y_type_,
--> 371                                                    self.classes_, threshold)
    372 
    373         if self.sparse_input_:

/home/kaichogami/codes/development_scikit-learn/scikit-learn/sklearn/preprocessing/label.pyc in _inverse_binarize_thresholding(y, output_type, classes, threshold)
    582     if output_type == "binary" and y.ndim == 2 and y.shape[1] > 2:
    583         raise ValueError("output_type='binary', but y.shape = {0}".
--> 584                          format(y.shape))
    585 
    586     if output_type != "binary" and y.shape[1] != len(classes):

ValueError: output_type='binary', but y.shape = (150, 3)

When mini-batch contain only one class.

ovr.partial_fit(X[:20], y[:20], np.unique(y))
/home/kaichogami/codes/development_scikit-learn/scikit-learn/sklearn/multiclass.py in partial_fit(self, X, y, classes)
    253         self.label_binarizer_ = LabelBinarizer(sparse_output=True)
    254         Y = self.label_binarizer_.fit_transform(y)
--> 255         Y = Y.tocsc()
    256         Y = Y.T.toarray()
    257         if len(self.label_binarizer_.classes_) == 2:

AttributeError: 'numpy.ndarray' object has no attribute 'toc

This will be resolved once #6202 is solved.

Solving the first issue here takes priority, and I am unable to figure out where it goes wrong.
@MechCoder

MechCoder · 2016-01-27T23:51:04Z

The first issue is because of the SGDClassifier. Note that there are no missing classes in the second batch (at least in your example, np.unique(y[7:]) == np.unique(y))

If you want to make sure in the SGDClassifer a fit is equal to doing partial_fit twice, you need to set n_iter=1, shuffle=False and random_state=0 in the constructor of SGD

kaichogami · 2016-01-28T08:12:40Z

hmm.. Okay. Looks like the issue was with SGDClassifier and my in adequate knowledge about the parameters. It works perfectly fine with MultinomialNB

X = np.random.rand(14, 2)
y = [0, 0, 0, 1, 2, 2, 2, 1, 1, 2, 1, 3, 3, 3]
ovr = OneVsRestClassifier(MultinomialNB())
ovr.partial_fit(X[:7], y[:7], np.unique(y))
ovr.partial_fit(X[7:], y[7:])
pred = ovr.predict(X)

ovr1 = OneVsRestClassifier(MultinomialNB())
pred1 = ovr1.fit(X, y).predict(X)

In [96]: np.mean(pred1 == y)
Out[96]: 0.35714285714285715

In [97]: np.mean(pred == y)
Out[97]: 0.35714285714285715

Added tests, raised error when mini-batch classes doesn't contain classes from all_classes.

amueller · 2016-10-07T18:27:26Z

What's the status on this?

kaichogami · 2016-10-08T05:52:53Z

@amueller I don't remember, give me 2-3 days to get back on this.

kaichogami · 2016-10-29T05:09:31Z

I have been busy, if anyone wants to give it a try, please!

srivatsan-ramesh · 2016-10-29T13:35:08Z

I would like to do this work.

srivatsan-ramesh · 2016-10-29T13:39:19Z

Why LabelBinarizer is not having a parameter for number of classes?.. I think if it has that parameter, the fix for this should be easy?

kaichogami · 2016-10-29T16:30:00Z

@srivatsan-ramesh I don't think we can change the API of a class so easily. The first issue wasn't really an issue and second one is solved in this PR, and the 3rd one will work fine as #6202 has been fixed. This just needs rebasing with the master. If you want you can pull my branch and rebase with master and push it in my repo.

srivatsan-ramesh · 2016-10-29T18:13:44Z

@kaichogami I don't think this fix is correct, or maybe i don't understand it?
Check my PR #7786 , I think that should work.

kaichogami · 2016-10-30T04:32:47Z

@srivatsan-ramesh #7786 looks clean and simple. Lets go ahead with that!

kaichogami force-pushed the OvR_partial_fit branch from db62a75 to 59c873a Compare January 29, 2016 18:45

Fixed OvR partial_fit method in various edge cases.

cdd8fa5

Added tests, raised error when mini-batch classes doesn't contain classes from all_classes.

kaichogami force-pushed the OvR_partial_fit branch from 59c873a to cdd8fa5 Compare January 29, 2016 18:49

srivatsan-ramesh mentioned this pull request Oct 29, 2016

[MRG + 1] Fix for OvR partial_fit bug #7786

Merged

kaichogami closed this Oct 30, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix for OvR partial_fit in various edge cases#6239

Fix for OvR partial_fit in various edge cases#6239
kaichogami wants to merge 1 commit intoscikit-learn:masterfrom
kaichogami:OvR_partial_fit

kaichogami commented Jan 27, 2016

Uh oh!

MechCoder commented Jan 27, 2016

Uh oh!

kaichogami commented Jan 28, 2016

Uh oh!

amueller commented Oct 7, 2016

Uh oh!

kaichogami commented Oct 8, 2016

Uh oh!

kaichogami commented Oct 29, 2016

Uh oh!

srivatsan-ramesh commented Oct 29, 2016

Uh oh!

srivatsan-ramesh commented Oct 29, 2016

Uh oh!

kaichogami commented Oct 29, 2016 •

edited

Loading

Uh oh!

srivatsan-ramesh commented Oct 29, 2016

Uh oh!

kaichogami commented Oct 30, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

kaichogami commented Jan 27, 2016

Uh oh!

MechCoder commented Jan 27, 2016

Uh oh!

kaichogami commented Jan 28, 2016

Uh oh!

amueller commented Oct 7, 2016

Uh oh!

kaichogami commented Oct 8, 2016

Uh oh!

kaichogami commented Oct 29, 2016

Uh oh!

srivatsan-ramesh commented Oct 29, 2016

Uh oh!

srivatsan-ramesh commented Oct 29, 2016

Uh oh!

kaichogami commented Oct 29, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

srivatsan-ramesh commented Oct 29, 2016

Uh oh!

kaichogami commented Oct 30, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kaichogami commented Oct 29, 2016 •

edited

Loading