0

When minimising the expected prediction error (EPE) for a linear model $f(x)=x^T\beta$

$$EPE(\beta)=\int(y-x^T\beta)^2Pr(dx,dy)$$

What do we get when we differentiate wrt $\beta$?

I've found conflicting results online.

According to this comment (https://math.stackexchange.com/a/2166977/280789)

$$\frac{\partial EPE}{\partial \beta}=\int 2(y-x^T\beta)(-1)x^T Pr(dx,dy)$$

But this doesn't seem to jive with the fact that;

$$\frac{\partial a^Tx}{\partial x}=a$$

It seems that the $x^T$ should just be an $x$. This matches with what's given for the derivative here (https://waxworksmath.com/Authors/G_M/Hastie/WriteUp/Weatherwax_Epstein_Hastie_Solution_Manual.pdf)

This is unfortunate because the rest of the former analysis was quite nice.

Either way it seems like we basically just moved the partial derivative inside the integral, what enables us to do this?

If we write the derivative with the $x$ on the LHS (which we can do since $(y-x^T\beta)$ is a scalar)

$$\frac{\partial EPE}{\partial \beta}=\int 2x(y-x^T\beta)(-1) Pr(dx,dy)$$

We can derive the result for $\beta$ as follows:

$$\frac{\partial EPE}{\partial \beta}=\int 2x(y-x^T\beta)(-1) Pr(dx,dy)$$ $$=-2\int x(y-x^T\beta) Pr(dx,dy)$$ $$=-2\int xy-xx^T\beta Pr(dx,dy)$$ $$=-2E[xy-xx^T\beta]$$

Setting $\frac{\partial EPE}{\partial \beta}=0$

$$E[xy-xx^T\beta]=0$$ $$E[xy]-E[xx^T\beta]=0$$ $$E[xy]=E[xx^T\beta]$$ $$E[xy]=E[xx^T]\beta$$ $$\beta=E[xx^T]^{-1}E[xy]$$

tail_recursion
  • 411

1 Answers1

1

I think they used Leibniz rule to push the partial diff inside the integral. You can find more info here (https://en.wikipedia.org/wiki/Leibniz_integral_rule#:~:text=This%20formula%20is%20the%20gener). Since the range of the integral isn't dependent on $\beta$, you get the last term in the Leibniz rule in the link given above.

Regarding the vector differentiation, it seems to be a typo but it doesn't matter as long as they are consistent. If $\beta$ is a column vector, then we want the RHS to be column vector as well, so it should be $a$, not $a^T$. However, since $(y-x^T\beta)$ is a scalar, it doesn't matter.

MoneyBall
  • 877