[MRG] Add plotting module with heatmaps for confusion matrix and grid search results by thismlguy · Pull Request #9173 · scikit-learn/scikit-learn

thismlguy · 2017-06-20T06:54:31Z

Continues PR 8082

Changes made:

Added plot functions - plot_confusion_matrix, plot_gridsearch_results
Updated examples - plot_rbf_parameters.py, plot_confusion_matrix.py
added unit tests for new plot modules

…ch results

amueller · 2017-06-20T17:07:49Z

can you fix the flake8 issues?

amueller

very first round ;)

amueller · 2017-06-20T17:26:47Z

doc/modules/classes.rst

+   :no-inherited-members:
+
+This module is experimental. Use at your own risk.
+Use of this module requires the matplotlib library.


Maybe state version? 1.5?

Modified to: "Use of this module requires the matplotlib library,
version 1.5 or later (preferably 2.0)."

amueller · 2017-06-20T17:27:51Z

doc/modules/classes.rst

+   :toctree: generated/
+   :template: function.rst
+
+   plot_heatmap


add functions here

added. this was WIP in previous PR.

I wouldn't count on users to read this message. My two cents is that if you really want users to notice that the plotting module is experimental, you'd have to put it in a sub module "experimental" or "future", and only move it to the main namespace when the API is stable.

It's a bit unclear to me what it would mean for the API to be stable, and I really don't like forcing people to change their code later. I would probably just remove the warning here and then do standard deprecation cycles.

Doing deprecation cycles is forcing people to change their code eventually, with the additional risk that they won't know that this code is experimental :)

Deprecation cycles only sometimes require users to change their code - if they are actually using the feature you're deprecating. That's not very common for most deprecations in scikit-learn. And that is only if there is actually a change.
And I'm not sure what's experimental about this code. The experiment is more having plotting inside scikit-learn. Since it's plotting and therefor user facing, I'd rather have a warning on every call then putting it in a different module.

I guess the thing we are trying to communicate is "don't build long-term projects relying on the presence of plotting in scikit-learn because we might remove it again".

amueller · 2017-06-20T17:28:57Z

sklearn/plot/_confusion_matrix.py

+        generate the heatmap.
+    """
+
+    import matplotlib.pyplot as plt


not sure if we want actual tests of the functionality (I'm leaning no), but I think we want at least smoke-tests.

added smoke-tests which just run the code without asserts.

hm looks like the config we use for coverage doesn't have matplotlib. We should change that...

amueller · 2017-06-20T17:29:05Z

sklearn/plot/_confusion_matrix.py

+    if normalize:
+        values = values.astype('float') / values.sum(axis=1)[:, np.newaxis]
+
+    print(title)


amueller · 2017-06-20T17:29:53Z

sklearn/plot/_gridsearch_results.py

+from sklearn.plot import plot_heatmap
+
+
+def plot_gridsearch_results(cv_results, param_grid, metric='mean_test_score',


This only works for 2d grid-searches, right? I think we should also support 1d, and error for anything else.

amueller · 2017-06-20T17:30:41Z

sklearn/plot/tests/test_heatmap.py

+    except ImportError:
+        raise SkipTest("Not testing plot_heatmap, matplotlib not installed.")
+
+    import matplotlib.pyplot as plt


hm it looks like matplotlib is not installed for the service that computes coverage? hm...

yea this seems to be a problem because my tests cover much more than what Codecov is showing. i wasn't able to check coverage locally as I'm using a mac and running matplotlib from terminal is giving errors which I'm not able to resolve yet.

have you resolved those?

nope these are tricky to resolve. it requires me to re-install python using a different method and then i'll have to setup my scikit-learn development environment again. since sklearn doesn't test a lot on matplotlib, i'm just using a jupyter-notebook for this PR.

We should discuss this in person.

codecov · 2017-06-21T19:39:44Z

Codecov Report

Merging #9173 into master will decrease coverage by 0.17%.
The diff coverage is 23.68%.

@@            Coverage Diff             @@
##           master    #9173      +/-   ##
==========================================
- Coverage    96.3%   96.13%   -0.18%     
==========================================
  Files         332      337       +5     
  Lines       60549    60754     +205     
==========================================
+ Hits        58314    58403      +89     
- Misses       2235     2351     +116

Impacted Files	Coverage Δ
sklearn/plot/__init__.py	`100% <100%> (ø)`
sklearn/plot/tests/test_heatmap.py	`32.43% <32.43%> (ø)`
sklearn/plot/_confusion_matrix.py	`33.33% <33.33%> (ø)`
sklearn/plot/_heatmap.py	`6.66% <6.66%> (ø)`
sklearn/plot/_gridsearch_results.py	`8.57% <8.57%> (ø)`
sklearn/utils/testing.py	`89.5% <0%> (-0.04%)`	⬇️
sklearn/decomposition/tests/test_pca.py	`100% <0%> (ø)`	⬆️
sklearn/decomposition/pca.py	`94.5% <0%> (ø)`	⬆️
... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0d5d842...b5e5823. Read the comment docs.

thismlguy · 2017-06-21T21:56:00Z

I think we should also give the user an option to plot everything as a single line graph. This will allow them to get a good plot results in case number of unique parameters are 3 or more but the total number of cases are within say 20.

Adding x-axis labels will be a challenge. But we can give each combo an ID and then print a table below showing the value of each parameter for each ID on the plot.

Thoughts?

thismlguy · 2017-06-21T22:16:22Z

Also, could you help me understand the circleCI error (details here):

Exception occurred:
File "/home/ubuntu/miniconda/envs/testenv/lib/python2.7/site-packages/docutils/writers/_html_base.py", line 671, in depart_document
assert not self.context, 'len(context) = %s' % len(self.context)
AssertionError: len(context) = 1
The full traceback has been saved in /tmp/sphinx-err-RgKdVr.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at https://github.com/sphinx-doc/sphinx/issues. Thanks!
make: *** [html] Error 1

./build_tools/circle/build_doc.sh returned exit code 2

Action failed: ./build_tools/circle/build_doc.sh

amueller · 2017-06-28T15:37:04Z

@aarshayj can you merge master? This is an issue with old sphinx, I think, which should be fixed in master.

…cikit-learn into plot_confusion_matrix

jeremiedbb · 2026-04-05T13:13:46Z

We now have a display for confusion matrix.
For GridSearch results, no consensus on the scope of the display has been found yet. In addition, the plotting API has changed so much since this PR that we would be better off restarting from a fresh new PR. Anyway, inclusion and feature scope of a display for GridSearch results should be discussed in a dedicated PR first.
With that in mind I'm closing this PR.

amueller and others added 15 commits December 19, 2016 11:41

add plotting module and "plot_heatmap" function

fe41a8b

add plotting module to the API docs

62da4fb

simplify plot_confusion_matrix example

23d8671

add normalizer support to heatmap, use heatmap when plotting gridsear…

67308d4

…ch results

add plot to __all__

40078c6

using pcolormesh + alignment fix

5d43843

added confusion_matrix plot file

221d748

made vmin and vmax pass through in heatplot plot function

74143d5

modified documentation plot_confusion_matrix

0b377e9

updated __init__.py file to include confusion matrix plot

e875d0d

plot confusion matrix example updated to use new function

74bf786

make matrix diagonal

8a1b861

modify documentation

abaaccb

adding grid search results plotting function.

badf387

modified examples/svm/plot_rbf_parameters.py with new function

46c4253

thismlguy mentioned this pull request Jun 20, 2017

Add plotting module with heatmap function #8082

Closed

amueller reviewed Jun 20, 2017

View reviewed changes

aarshayj added 6 commits June 20, 2017 17:50

removed printing confusion matrix

8f4e5f1

remove param_grid argument

d6940f5

adding cases for nparams 1,2, more

18ac9c7

minor fixes

343db9d

fixed typo

d82c4c2

adding tests for confusion matrix and grid search plots

a3ccae9

aarshayj added 2 commits June 21, 2017 17:26

adding test case for normalized

d27c859

updated doc files

b5e5823

aarshayj and others added 22 commits January 12, 2018 17:45

randint correction

bc73643

y_pred and y_true as input in place of matrix

caebd38

fixed typo

0f124f1

fixed example

5e5f741

making classes optional

2e3eec9

some fixes

7f3336f

fixed 1d case of grid_search_results

6be2a92

fixed confusion matrix test

1c8db68

working on axes not plt

f737c01

adding title to plot_heatmap, removing plt.show from within API

170471d

added section to validation curve example

df031bf

sphinx syntax fix

d676be3

matplotlib new figure creation modified

f4d8a64

define axis closer to public layer

a90c0d2

removed plt.draw()

fa2dec7

docstring split lines

876d854

Merge branch 'plot_confusion_matrix' of https://github.com/aarshayj/s…

6e2d812

…cikit-learn into plot_confusion_matrix

add tight_layout to plot confusion matrix examples

fb16ab0

remove second print doc statement

8cc94f1

adding axis format and tight layout

c48b61a

adding tight_layout

2d1e6cf

taking .travis.yml from master and adding matplotlib

01a63a7

amueller mentioned this pull request Jul 14, 2018

[MRG+1] EXA Adding cv indices example #11475

Merged

thomasjpfan mentioned this pull request Mar 14, 2019

Add a sklearn.plot Module #13448

Closed

thomasjpfan mentioned this pull request Jul 14, 2019

[MRG] Plotting API starting with ROC curve #14357

Merged

amueller added the Stalled label Aug 5, 2019

chbrandt mentioned this pull request Jan 29, 2020

[WIP] Add plotting module with heatmaps for confusion matrix and grid search results #16287

Closed

5 tasks

Base automatically changed from master to main January 22, 2021 10:49

jeremiedbb closed this Apr 5, 2026

		from sklearn.plot import plot_heatmap


		def plot_gridsearch_results(cv_results, param_grid, metric='mean_test_score',

Uh oh!

Conversation

thismlguy commented Jun 20, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

amueller commented Jun 20, 2017

Uh oh!

amueller left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

thismlguy Jun 21, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Jun 21, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

thismlguy commented Jun 21, 2017

Uh oh!

thismlguy commented Jun 21, 2017

Uh oh!

amueller commented Jun 28, 2017

Uh oh!

jeremiedbb commented Apr 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

thismlguy commented Jun 20, 2017 •

edited

Loading

thismlguy Jun 21, 2017 •

edited

Loading

codecov bot commented Jun 21, 2017 •

edited

Loading