-
-
Notifications
You must be signed in to change notification settings - Fork 2k
Description
Sktime's forecasting performance metrics currently only cover a limited number of metrics discussed in the forecasting literature.
I propose to add the metrics discussed in: Another look at measures of forecast accuracy, which has a nice discussion of different metrics for evaluating forecast accuracy.
I also propose adding root mean squared scaled error, which is related to MASE and the forecast metric used in the M5 Competition
Beyond these performance metrics the availability of an asymmetric forecasting loss function is useful when forecasting errors in one direction have a higher cost than those in another direction. I propose adding an asymmetric error function too.
In addition to these changes, I also propose two changes to make Sktime's performance metrics more compatible with Scikit-learn:
- Scikit-learn adapted many (all?) of its regression metrics to handle multioutput regression problems. I propose providing this functionality for each of the forecasting metrics in the above resources so that compatability with Scikit-learn's metrics is maintained and so that the functionality exists if future multivariate forecasting functionality is added.
- Scikit-learn's metrics write out the full name (e.g. mean_absolute_error) and I propose updating Sktime's forecasting performance metrics to do the same
The end result would be the following performance metric functions (each with a corresponding class version for scoring):
- relative_error
- mean_asymmetric_error
- mean_absolute_scaled_error
- median_absolute_scaled_error
- root_mean_squared_scaled_error
- mean_absolute_error
- mean_squared_error
- root_mean_squared_error
- median_absolute_error
- median_squared_error
- root_median_squared_error
- symmetric_mean_absolute_percentage_error
- symmetric_median_absolute_percentage_error
- mean_absolute_percentage_error
- median_absolute_percentage_error
- mean_squared_percentage_error
- median_squared_percentage_error
- root_mean_squared_percentage_error
- root_median_squared_percentage_error
- mean_relative_absolute_error
- median_relative_absolute_error
- geometric_mean_relative_absolute_error
- geometric_mean_relative_squared_error
I've already started work on this and will start a draft pull request soon.