0

I am familiar with the mean absolute error (MAE) for the evaluation of performances in a regression model. Given a set of ground truth (GT) values $y_i$, and a set of the model evaluated values $\hat{y}_i$, the mean average error $E_{mae}$ is defined by: $$E_{mae}=\frac{1}{N}\sum_{i=0}^N|y_i-\hat{y}_i|,$$ where $N$ is the number of points in the dataset, and the best error is $E_{mae}=0$. Recently, I have discussed with a colleague the following suggested error metric $E_{acc}$ (he called it accuracy): $$E_{acc} = \frac{1}{N}\sum_{i=0}^N\frac{y_i}{\hat{y}_i},$$ Where the best error $E_{acc}=1$. The claim here is that this metric is more informative since it also gives a sense of the direction of error. To be clearer, if this accuracy measure is averaged to a value greater than $1$, it means that the model is biased to evaluate a lower value than the GT.

I admit that I am unfamiliar with this type of metric. As far as I am aware, the accuracy measure is a measure for classification tasks and not regression. I could not find any such metric online.

Is there such a metric? Is it common? How does it compare with MSE? How does one encapsulate the direction of error in a regression model? Is there a third matric for that?

havakok
  • 1,067
  • Just to clarify since my answer was only relevant for $E_{acc}$ as a general metric (where it can fail terribly): Is this only applied to a linear regression settings where you minimized the SS? – Claudio Moneo Apr 24 '22 at 11:51
  • SS is the sum of squares. I have an example below. Of course you can extend this to more points – Claudio Moneo Apr 24 '22 at 13:54
  • The question discusses the performance evaluation metric of a regression model, regardless of the minimization function used to create it. – havakok Apr 24 '22 at 14:16

1 Answers1

0

Suppose $y_1 = y_2 = 1$ but you have predicted $\hat y_1 = -r$ and $\hat y_2 = +r$. Then $E_{acc} = 0$ but your model can actually be arbitrarily bad. A better way to identify bias in your predictions is plotting the fitted values against the residuals $r_i = y_i - \hat y_i$. Also note that if you include an intercept in regression, the residuals will always sum up to zero.

Claudio Moneo
  • 2,176
  • Well, I have edited my question to clarify. The best error for the so-called accuracy error is $E_{acc}=1$. In your example, $E_{acc}=0$ indicates that the model is faulty. – havakok Apr 24 '22 at 14:02