I don't think we should mutate rcParams. Can we just specify dpi and figsize when saving plots?

seems cannot. We need set it when creating figure, if so we need modify shap source code.

maybe we could try this
https://stackoverflow.com/a/35394637

Yes. I just also found it. Use with matplotlib.rc_context(...)

where do we use this function?

In usage log, we want to log error raised from which evaluator (TODO), for debug purpose. so I add this.

why do we only store the last failed evaluator?

The evaluate API will raise error when it hit the first failed evaluator. (The API designed to be so.) When any evaluator fail evaluate raise error, otherwise return all merged success results

Note:
in shap 0.40, the shap.Explainer argument algo , only when setting 'exact', 'permutation', 'partition', it works fine, the remaining algo (listed in shap doc) are all buggy in my test. So I limit them to be the 3 values.

Can we set the dpi value to something higher than the default of 100? The images look really blurry and out of place in the tracking UI. 300 might be a better option.

pyplot.savefig(artifact_file_local_path, dpi=300)

Or perhaps save in .svg format to ensure that scalability in different zoom levels in browsers will work well?

I increase dpi to 288 (previous in databricks notebook it is 72)

Save fig as svg is not good choice, the Pillow package does not support svg. But we need return artifact as the pillow Image instance.

We need to scale these plots for class count > 6. The example for multiclass that has 10 classes is illegible with numeric wrapping overwrites of values. Setting figsize overrides on the canvas and changing the dpi at figure save time will make a much more clear visualization.

The autologging code also generate confusion matrix plot for training dataset and has similar issue. Let's do a follow-up PR to improve for both cases.

Create a follow-up ticket for this:
https://databricks.atlassian.net/browse/ML-19817

What does the "*" mean here? Do you intend to make it a nameless positional arg like *args? It may worth to spell it out even if it's not used, and put it in the end of the argument list, e.g.
def _evaluate(model, model_type, dataset, actual_run_id, evaluator_name_list, evaluator_name_to_conf_map, *unused_args):

No. The * is a python grammar, not a argument, it means all following arguments must be a key-word-only arguments. https://www.python.org/dev/peps/pep-3102/

Mark all argument as key-word-only is a proposal raised by @dbczumar , so that in future if mlflow containerized model provide "model_type" property, we can remove the "model_type" argument here but keep backwards compatibility

Wait, removing the model_type argumnet is a breaking change, right?

In future, we can change API to be :

def evaluate(
    *, model, dataset, run_id, evaluators=None, evaluator_config=None, **kwargs
):
   if "model_type" in kwargs:
      model_type = kwargs["kwargs"]
      logger.warning("deprecation: ....")
   else:
      model_type = model.model_type
   ...

Then we can keep backwards compatiblity.

I didn't know about this usage pattern. Interesting!
Another way to keep backward compatibility is to use keyword arg for model_type, e.g.

def evaluate(
    model, dataset, run_id, model_type=None, evaluators=None, evaluator_config=None, **kwargs
):

What's the downside of going this more "conventional" way of specifying optional arguments?

In current evaluate implementation, the model_type argument is required. In future we will make it "optional and deprecated" and then remove this argument.

Nice! I really like the documentation here to spell out the expected outcome and limitations. Hopefully we'll make it shorter and easier to navigate over time.