Nice starter issue with a recipe.
evaluate_by_index is the interface point of performance metrics that provides contributions per time point without averaging for metrics classes.
Most classes, however, do not have an efficient implementation for this - instead, it is backed out of the aggregate (evaluate), which is unnecessarily complex. For instance, in the mean squared error, instead of providing a frame or series of squares, we take mean squared errors of sub-sequences and subtract them from each other to back out the squares.
(the reason for this is that it could be done by one piece of boilerplate logic, as opposed to doing it per metric)
This should be replaced by efficient implementations of _evaluate_by_index in the metrics classes.
Some notes:
- For mean metrics it should be straightforward; for metrics that are not means (e.g., RMSE), an efficient expression for the jackknife pseudo-sample estimate needs to be worked out.
- the logic should take into account the
multioutput logic - weighting across variables could be non-trivial, but will be just simple averaging in most of the cases.
- whenever the metric has parameters (e.g.,
sp), they need to be accessed, and conditional logic implemented
Recipes or templates PR:
When working on this, kindly post the name of the metric here, to avoid duplication of efforts.
Nice starter issue with a recipe.
evaluate_by_indexis the interface point of performance metrics that provides contributions per time point without averaging for metrics classes.Most classes, however, do not have an efficient implementation for this - instead, it is backed out of the aggregate (
evaluate), which is unnecessarily complex. For instance, in the mean squared error, instead of providing a frame or series of squares, we take mean squared errors of sub-sequences and subtract them from each other to back out the squares.(the reason for this is that it could be done by one piece of boilerplate logic, as opposed to doing it per metric)
This should be replaced by efficient implementations of
_evaluate_by_indexin the metrics classes.Some notes:
multioutputlogic - weighting across variables could be non-trivial, but will be just simple averaging in most of the cases.sp), they need to be accessed, and conditional logic implementedRecipes or templates PR:
MeanAbsoluteError-evaluate_by_index, docstring #4302, where the change is carried out forMeanAbsoluteError. This is a mean loss, so_evaluatedoes not need to be implemented._evaluate_by_indexfor MSE and RMSE (MeanSquaredError) #6248, the RMSE/MSE. This is not a mean loss, so_evaluateand_evaluate_by_indexboth need to be implemented.When working on this, kindly post the name of the metric here, to avoid duplication of efforts.