I'm reading Bishop "Pattern Recognition and Machine Learning" and cannot understand how to get the following result:
Imagine we have $\mathbb{E}[L] = \int\int (y(x) - t)^2p(x, t)dxdt$ and want to minimize it choosing y(x). So, we want $\frac{\partial \mathbb{E}[L]}{\partial y(x)} = 0$. And $\frac{\partial \mathbb{E}[L]}{\partial y(x)}$ is given there as $2 \int (y(x) - t) \cdot p(x, t) dt$.
How can I get this result?
$\mathbb{E}[L]$ is not a function but a (linear) functional, i.e. a map that takes a whole function as input and returns a scalar value. Functionals are minimized in the same spirit as function optimization by setting the derivative to zero. However, in this case, one has to take a derivative with respect to a function $y(x)$ (itself dependent of the variable $x$). In order to do so, you need to use the "functional derivative" $\frac{\delta}{\delta y(x)}$ (cf. https://en.wikipedia.org/wiki/Functional_derivative) instead of the usual derivative.
In pratice, it works as follows : the functional differential is given by $$ \delta F[y] = \sum_{n=0}^\infty \frac{\partial^n F[y]}{\partial y^{(n)}} \delta y^{(n)} $$ where $y^{(n)}$ is the $n^{\mathrm{th}}$ derivative of $y$; each derivative of $y$ plays the role of a variable contributing to the variation of the functional $F[y]$. We also have : $$ \frac{\delta y^{(n)}(x')}{\delta y(x)} = \frac{\mathrm{d}^n}{\mathrm{d}x^n}\frac{\delta y(x')}{\delta y(x)} = \frac{\mathrm{d}^n}{\mathrm{d}x^n}\delta(x-x') \equiv (-1)^n\delta(x-x')\frac{\mathrm{d}^n}{\mathrm{d}x^n}, $$ where we used the fact that $\frac{\delta y(x')}{\delta y(x)} = \delta(x-x')$, with $\delta(x-x')$ the Dirac delta function $-$ note the same pattern as in classical calculus : $\frac{\mathrm{d}x_i}{\mathrm{d}x_j} = \delta_{ij}$, with $\delta_{ij}$ the Kronecker delta $-$, as well as the property $\delta'(x) \equiv -\delta(x)\frac{\mathrm{d}}{\mathrm{d}x}$. Finally, we have : $$ \frac{\delta F[y(x')]}{\delta y(x)} = \delta(x-x')\sum_{n=0}^\infty (-1)^n\frac{\mathrm{d}^n}{\mathrm{d}x^n} \frac{\partial^nF}{\partial y^{(n)}} $$
Now we can find the functional derivative of your functional of interest : $$ \frac{\delta \mathbb{E}[L]}{\delta y(x)} = \frac{\delta}{\delta y(x)} \int\mathrm{d}x\int\mathrm{d}t \verb+ + (y(x')-t)^2p(x',t) = \int\mathrm{d}x\int\mathrm{d}t \verb+ + \frac{\delta}{\delta y(x)} (y(x')-t)^2p(x',t) $$ by linearity of the integrals and derivatives. One has then : $$ \begin{array}{rcl} \displaystyle\frac{\delta}{\delta y(x)} (y(x')-t)^2p(x',t) &=& \displaystyle \delta(x-x')\sum_{n=0}^\infty (-1)^n\frac{\mathrm{d}^n}{\mathrm{d}x^n} \frac{\partial^n}{\partial y^{(n)}} (y(x')-t)^2 p(x',t) \\ &=& \displaystyle 2\delta(x-x')(y(x')-t)p(x',t) \end{array} $$ hence $$ \frac{\delta \mathbb{E}[L]}{\delta y(x)} = \int\mathrm{d}x\int\mathrm{d}t \verb+ + 2\delta(x-x')(y(x')-t)p(x',t) = 2\int(y(x)-t)p(x,t)\mathrm{d}t $$ which you can set to zero. Note that $p(x',t)$ is not affected by the functional derivative with respect to $y(x)$ in the same way as independent variables in classical calculus.
