So we've decided to:
(1) Use sklearn.XXX.plot.plot_XXX, do not allow from sklearn.XXX import plot_XXX
(2) Introduce a Visualizer class and a plot function. The Visualizer class do calculation in __init__ and do plotting in plot.

I think this is a great solution, but I'd like to see more opinions here (>=+3 maybe). Are you able to reach out to more people during the sprint, or maybe a SLEP/mail in the mailing list?

You need to change our matplotlib dependency in readme and install.rst to something like classes end with Visualizer and functions start with plot_.

We need to take care of plot_partial_dependence, right? Maybe it's acceptable to leave plot_tree as it is?

CC @NicolasHug

There's a reimplementation of plot_partial_dependence linked above. I think it's backwards compatible apart from the returned object (which is a bit tricky).

I think it's worth emphasizing that the common simple use-case involves just calling the function, and the user not having to worry about the object. The main motivation for the object is to allow replotting without recomputation.

Maybe if we want to make the SLEPs more lightweight it might make sense to write a slep for this, I'm not sure.

I listed alternatives in #13448 (#13448 (comment))

There's some discussion there as well.

Regarding plot_roc_curve, will it be useful to return the roc_auc_score?

Maybe to summarize, we could:

1. have a plot_X function that takes the results of some computation. Upside: clear separation of computation and testing, backward compatible, pretty obvious interface. Downside: users have to call two functions. (say roc_curve and plot_roc_curve) and need to ensure the arguments match. (an alternative would be to have roc_curve return some ROC object, but that wouldn't be backward compatible and also seems not scikit-learn-like).

2. have a plot_X function that does the computation inside. Upside: simple interface for users. Downside: mixing plotting and computation somewhat. Without further work: can't adjust existing plots easily, can't plot or do anything else with computations and potentially need to compute results again just do show them again.

I like being able to plot a roc curve with a single line. I think that should be our goal, so I'm strongly in favor of 2. Having to recompute, say, partial dependence, for plotting it again is not acceptable to me, though. That means we need to return or store the results of the computation somehow.

There's two immediate solutions (maybe there's better ones that I can't think of):
a) have the plotting one-liner be an object that does the computation and plotting in the __init__ and then stores everything in self and can be plotted again.
b) have a utility function that does the plotting and returns an object that stores the results and allows plotting again.

We opted for b) here because it hides the complexity somewhat from the user. The simple case still remains simple with a function call, not a class construction, which seems more natural to me (the difference is a bit academic, though, for the user it mostly manifests in whether we use CamelCase or snake_case for the thing they call (we could also have plot_roc_curve be a class and just make it look like a function if we wanted to).

docstring tests are failing @thomasjpfan

There's two roc curve examples, one for multi-class and one for cross-validation.
I suggest we merge the multi-class roc_auc first (though that only relates to the multi-class example of course).

The cross-validation example we can pretty easily do with the new function, apart from computing the mean.
For the mean, we could either:

1. just not plot it (I vote this)
2. in the end pull the results out of all the objects and do the interpolation bit
3. add support to the plotting method directly (probably not)

For the multi-class one, I would use the plot function and manually apply it to each label for plotting in OVR fashion, and use the newly added multi-class roc to actually compute the metric.
Maybe showing the different kinds of metrics might be nice, but we can always add this later.

Naming sugestion: changer "Visualizer" to "Display": "RocCurveDisplay"

Hmm, current API design seems strange, I think the visualizer should have a consistent interface compared with the helper function, i.e., it should accept estimator&X&y instead of fpr&tpr, otherwise the visualizer seems useless.

@qinhanmin2014 the ultimate goal here is to enable changing the style of plots without having to recompute values.

We want to keep functions plot_... because we already have these and users are used to them. Should the visualizer have the same API as the function, that would require the visualizer to perform the computations in init. That would also mean there are 2 ways of doing the same thing (use a function or use an object), which we want to avoid.

We want to keep functions plot_... because we already have these and users are used to them. Should the visualizer have the same API as the function, that would require the visualizer to perform the computations in init. That would also mean there are 2 ways of doing the same thing (use a function or use an object), which we want to avoid.

Maybe it's common for a helper function to do similar things compared with the class in scikit-learn?

True, but I guess estimators are intended to be only used once.

The advantage of the current design is that if you realize you want to change the style of the plot after the fact, your code can stay (almost) the same. With your proposal uses have to replace the function call by an object instantiation.

@qinhanmin2014 raises the question in the dev meeting whether we need a visualizer for plot_tree. I don't think that's required but that makes the API for "plotting" somewhat inconsistent. Maybe that's a "different" kind of plotting functions but we should make sure that we have some consistent story.

I kinda like the Display idea by @GaelVaroquaux but not 100% sure.

I think Andy persuaded me during the meeting. I'll vote +1 here.

I don't think that's required but that makes the API for "plotting" somewhat inconsistent.

I agree that we don't need a visualizer for plot_tree, but the inconsistency is indeed a problem.
Maybe we can explain that for some plotting functions which do not require much calculation, we do not provide a visualizer?

I kinda like the Display idea by @GaelVaroquaux but not 100% sure.

+1 for either solution. I don't care much about the name.

Maybe @NicolasHug can organize a vote in the mailing list (since it's a brand new API in scikit-learn) and then we can merge this one.

waiting for @thomasjpfan to post an update and ping

(not sure why I should be the one calling for a vote though?)

(not sure why I should be the one calling for a vote though?)

because you organize the meeting, hmm, strange reason right :)
you can ask me to do this or maybe @thomasjpfan can better summarize this PR.

I have updated this PR and the original post with the following summary:

Summary

The key features of this API is to run calculations once and to have the flexibility to adjust the visualizations after the fact. This logic is encapsulated into a display object where the computed data is stored and the plotting is done in a plot method. The display object's __init__ method contains only the data needed to create the visualization. The plot method takes in parameters that only have to do with visualization, such as a matplotlib axes. The plot method will store the matplotlib artists as attributes allowing for style adjustments through the display object. A plot_* helper function accepts parameters to do the computation and the parameters used for plotting. After the helper function creates the display object with the computed values, it calls the display's plot method. Note that the plot method defines attributes related to matplotlib, such as the line artist. This allows for customizations after calling the plot method.

For example, the RocCurveDisplay defines the following methods:

class RocCurveDisplay:
    def __init__(self, fpr, tpr, roc_auc, estimator_name):
        ...
         self.fpr = fpr
         self.tpr = tpr
         self.roc_auc = roc_auc
         self.estimator_name = estimator_name


    def plot(self, ax=None, name=None, **kwargs):
        ...
        self.line_ = ...
        self.ax_ = ax
        self.figure_ = ax.figure_

Together with a plotting function to create this object:

def plot_roc_curve(estimator, X, y, pos_label=None, sample_weight=None,
                   drop_intermediate=True, response_method="auto",
                   name=None, ax=None, **kwargs):
       # do computation
       viz = RocCurveDisplay(fpr, tpr, roc_auc, 
                             estimator.__class__.__name__)
       return viz.plot(ax=ax, name=name, **kwargs)

I'm not sure if a vote is going to help here, it'll complicate things and this PR would be merged much much later, if we go for that process, which also includes writing a SLEP.

So +1 from Joel, Andy, Adrin, Nicolas and me (maybe Gael not sure, sorry if I miss someone),
I think there's enough consensus here. @thomasjpfan please reply to my reviews (#14357 (review)), though they're very trivial :)
and then let's merge.

The brave new world of plotting?