[ENH] implement efficient _evaluate_by_index for forecast performance metrics #4304 Implementation for GeometricMeanAbsoluteError#6244
[ENH] implement efficient _evaluate_by_index for forecast performance metrics #4304 Implementation for GeometricMeanAbsoluteError#6244KaustubhUp025 wants to merge 43 commits intosktime:mainfrom KaustubhUp025:main
Conversation
…etricMeanAbsoluteError
fkiraly
left a comment
There was a problem hiding this comment.
Thanks!
I think there has been a mistake, you copied the mean absolute error (MAE)?
This metric is supposed to be GMAE, geometric mean absolute error.
Btw, it would also be nice if you provide the formula in the docstring, like in MeanAbsoluteError, that helps avoiding math errors and make the review easier.
|
Thank You @fkiraly , I will check it once. |
…MeanAbsoluteError with documentation as well
|
@fkiraly I have added the changes that I missed earlier. |
fkiraly
left a comment
There was a problem hiding this comment.
Thanks, this looks correct now!
Before we merge, can we discuss numerical stability?
If we implement the formula directly, we use np.prod which can lead to a numerical explosion.
Would it be better to use the exp/log representation instead?
That is, using that geometric averages are arithmetic averages of logarithms, then exp?
For discussion - we could test with a vector of length 100, with value 1000. My conjecture is that the current formula produces nans, while exp/log will be fine.
|
@fkiraly Thank you, I will give it a check and will let you know. |
|
@fkiraly I checked your conjecture and yepp you are right in that. and implementing the given functions to check it, I found this as my result. # Function to compute GMAE using direct product approach And :- def gmae_log_exp(y_true, y_pred, epsilon=1e-10): And so this was the result I got :- |
|
hm, something is not right here - the result should be 1000 |
|
Okay I will check it once. |
|
I see - I meant when the errors are a vector of 1000. To create that situation, you could make |
|
@fkiraly I gave the input as :- test_length = 100 and I got this as the result :-
|
|
hm, is direct more accurate? Surprising. How about you make the errors larger, e.g., 1e-10 and see what happens? |
|
@fkiraly So I used the input values given and here is the output for the same :-
|
|
yes, seems more stable for extreme values. Shall we go with the log/exp approach then? |
|
Yes I have written the code for that as well. I will just add it. |
…MAE. This modification ensures that the errors are strictly positive before taking their logarithm, improving the numerical stability of the calculation.
| n = raw_values.shape[0] | ||
| gmae_values = [] | ||
| for i in range(n): | ||
| excluded_error = np.delete(raw_values, i, axis=0) # Exclude i-th error |
There was a problem hiding this comment.
working with logarithms might make this a bit safer numerically
|
|
||
| n = raw_values.shape[0] | ||
| gmae_values = [] | ||
| for i in range(n): |
There was a problem hiding this comment.
while I wrote the code for PR #6461, I saw this as an error and vectorized this since without vectorizing the gmae_values it was difficult to multiply it with the custom weights in case of multioutput parameter.
Could you please check #6461 once? I think I should close this PR as I have written the correct code taking in reference from this PR
Adding all the changes to current main
Towards #4751 Creating class for MAEP for the private repo
Merge branch 'sktime:main' into Ksktime2
|
for discussion later in our mentoring meeting. I made some corrections and simplifications to the code, but I think it's not done yet.
The implementation in |
|
there are some failures due to wrong type - would be great if you could handle these, @KaustubhUp025. The formula should be correct, but a double check would also be appreciated, might be errors of sign or numerical issues. |
…ometricMeanAbsoluteError` (#7095) Towards #4304 Implementation of efficient `_evaluate_by_index` performance metric for `GeometricMeanAbsoluteError` by taking reference from #4302 Originally in #6244, which was deleted by @KaustubhUp025, this contained changes both by myself and @KaustubhUp025
Reference Issues/PRs
Towards #4304
What does this implement/fix? Explain your changes.
Implementation of efficient
_evaluate_by_indexperformance metrics by taking reference from #4302Does your contribution introduce a new dependency? If yes, which one?
No
What should a reviewer concentrate their feedback on?
Did you add any tests for the change?
No
Any other comments?
PR checklist
For all contributions
How to: add yourself to the all-contributors file in the
sktimeroot directory (not theCONTRIBUTORS.md). Common badges:code- fixing a bug, or adding code logic.doc- writing or improving documentation or docstrings.bug- reporting or diagnosing a bug (get this pluscodeif you also fixed the bug in the PR).maintenance- CI, test framework, release.See here for full badge reference
maintainerstag - do this if you want to become the owner or maintainer of an estimator you added.See here for further details on the algorithm maintainer role.
For new estimators
docs/source/api_reference/taskname.rst, follow the pattern.Examplessection.python_dependenciestag and ensureddependency isolation, see the estimator dependencies guide.