2

I'm a novice trying to determine if the use of R-squared is valid for developing an exponential regression (of the form $y=a+b\,e^{cx}$ ).

I've come across several articles online that say R-squared is invalid for nonlinear regression.

That claim makes sense to me.

But I've also noticed that lots of people still use R-squared for nonlinear regression.

  • Several of my statistics books suggest that R-squared can be used for all regression analysis, including nonlinear regression.
  • Most statistics software produces a R-squared value for nonlinear regression.
  • People that are far more educated than me continue to use R-squared for nonlinear regression.

So, I can't help but think that something doesn't add up here. Why do so many people still do it? Are they really all wrong, and I'm right?

My uneducated thoughts are:

  • Sure, it's a common misconception that R-squared is valid for all types of regression, including nonlinear regression.

  • But also, I'm wondering if the wording of "R-squared is invalid for nonlinear regression" might be an oversimplification.

Does the rule need clarification? For example:

  1. R-squared can be used for some nonlinear regressions, such exponential regression, if the equation is first flattened by taking the natural logarithm of both sides.

  2. [removed - misleading]

  3. There are models that appear to be nonlinear, but in this context, they're actually considered to be linear. Examples: parabolic and polynomial. So using R-squared for them is valid.

As mentioned, I'm a novice, I'm not a mathematician. Are clarifications 1-3 above correct? Or have I misunderstood?

Note: I've intentionally posted this question on Math Stack Exchange, instead of on Cross Validated, because I find I get better answers here.

User1974
  • 425
  • What do you mean by valid? And how are you defining $R^2$ in general? Something like $1- \frac{\sum (y_i-\hat y_i)^2}{\sum (y_i-\bar y)^2}$ exists so long as the denominator is not $0$ though it is not always positive – Henry Feb 11 '22 at 11:46

1 Answers1

2

To understand the statement, it is better to start with a model where the $R^2$ arises quite naturally. For a simple linear model $y = \beta_0 + \beta_1 x + \epsilon$, $R^2$ is just the square of the Pearson correlation coefficient. So, it has an intuitive and well defined meaning and support $[0,1]$. Now, let us check what happens in multiple regression, i.e., $y = \beta_0 + \sum_{j=1}^p \beta_j x_j + \epsilon$, then the $R^2$ is the Pearson correlation coefficient between the fitted values of $y$, namely, $\hat{y}$ and $y$. Which is analogously to the simple linear model, since $\hat{y}$ is the orthogonal projection of $y$ onto the affine space spanned by $\{1,x_1,...,x_p\}$. Now, let's look on the statistical definition. The OLS for a model with an intercept $\beta_0$, decompose the total variance (sum of squares) $\sum ( y_i - \bar{y} ) ^2 = $ TSS into RSS + ESS, i.e, $TSS = RSS + ESS$, where $RSS = \sum ( \hat{y} - y ) ^2$, and $ESS = \sum ( \hat y - \bar y) ^2$, thus dividing both sides by $TSS$, you have $$ 1 = \frac{RSS}{TSS} + \frac{ESS}{TSS} = R^2 + \frac{RSS}{TSS} \to R^2 = 1 - \frac{RSS}{TSS}. $$ This decomposition holds with the intuitive meaning of portion of explained variance as long as: (1) The model has an intercept, and (2) the model is linear. Otherwise, you can run into a problem. For example, in linear model without an intercept $R^2$ may be negative, and then it is meaningless to interpret it as a "portion of explained variance" (e.g., what does it mean to explain $-0.1$ of the variance?). For non linear models the very definition of the $R^2$ is questionable. For example, in logistic regression you predict probabilities $p_i$, however you observe only binary realizations $0$ or $1$. So, what does the $RSS$ mean? Therefore, there are many possible pseudo $R^2$ (McFadden's, Efron's, Nagelkerke's, Tjur's, and more). Some of them bounded by a value that is less then $1$, which makes it even harder to interpret.

V. Vancak
  • 16,444