Skip to content

Add Additional Forecasting Evaluation Functions/Classes #671

@RNKuhns

Description

@RNKuhns

Sktime's forecasting performance metrics currently only cover a limited number of metrics discussed in the forecasting literature.

I propose to add the metrics discussed in: Another look at measures of forecast accuracy, which has a nice discussion of different metrics for evaluating forecast accuracy.

I also propose adding root mean squared scaled error, which is related to MASE and the forecast metric used in the M5 Competition

Beyond these performance metrics the availability of an asymmetric forecasting loss function is useful when forecasting errors in one direction have a higher cost than those in another direction. I propose adding an asymmetric error function too.

In addition to these changes, I also propose two changes to make Sktime's performance metrics more compatible with Scikit-learn:

  • Scikit-learn adapted many (all?) of its regression metrics to handle multioutput regression problems. I propose providing this functionality for each of the forecasting metrics in the above resources so that compatability with Scikit-learn's metrics is maintained and so that the functionality exists if future multivariate forecasting functionality is added.
  • Scikit-learn's metrics write out the full name (e.g. mean_absolute_error) and I propose updating Sktime's forecasting performance metrics to do the same

The end result would be the following performance metric functions (each with a corresponding class version for scoring):

  • relative_error
  • mean_asymmetric_error
  • mean_absolute_scaled_error
  • median_absolute_scaled_error
  • root_mean_squared_scaled_error
  • mean_absolute_error
  • mean_squared_error
  • root_mean_squared_error
  • median_absolute_error
  • median_squared_error
  • root_median_squared_error
  • symmetric_mean_absolute_percentage_error
  • symmetric_median_absolute_percentage_error
  • mean_absolute_percentage_error
  • median_absolute_percentage_error
  • mean_squared_percentage_error
  • median_squared_percentage_error
  • root_mean_squared_percentage_error
  • root_median_squared_percentage_error
  • mean_relative_absolute_error
  • median_relative_absolute_error
  • geometric_mean_relative_absolute_error
  • geometric_mean_relative_squared_error

I've already started work on this and will start a draft pull request soon.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions