loss function in Machine learning

Question

Would someone be so kind to explain this to me: $F= \text{argmin} E_{x,y} [L(y,F(x)]$

I specially don't understand the $E_{x,y}$. I found this formula in wikipedia: https://en.wikipedia.org/wiki/Gradient_boosting thank you!

Answer 1

A loss function is a method of calculating an "error" between what you predict and what you are suppose to predict. For example, squared error is $$(y-f(x))^2$$ where $y$ is what we want to predict and $f(x)$ is what we did predict for $x$ using the model $f$. An arbitrary loss function would be some function $L$ defined on both $y$ and $f(x)$, like $$L(y,f(x))$$ In general, our objective in machine learning is to minimize loss, more formally we wish to minimize average loss. For example, mean squared error, $$\frac{1}{n}\sum_{k=0}^n (y_k - f(x_k))^2$$ Notice that averaging our loss means to use some subset of our data, the $y_k$'s and $x_k$'s. The most formal way to talk about arbitrary averaging about arbitrary distributions, is called expectation, denoted by $\mathbb{E}$. We now have that $$\hat F = \underset{F}{\text{argmin}}\ \mathbb{E}_{x,y} L(y,F(x)) $$ says formally, for a fixed loss function, $L$, analysis all possible models $F$ and find the one that minimizes the expected (average) loss that is calculated over the distribution of my dataset. The statement is very formal and tells us nothing about what loss function to use or how to actual find the $\hat F$ that minimizes the average loss.

Answer 2

The meaning of $\text{arg min}$ is explained in the answers to this question. That means the value of $F$ which minimises the expression.

$E_{x,y}$ means expectation, which is defined as $$E{x,y}(f(x,y))=\sum_{x,y} p(x,y) f(x,y)$$ where $p(x,y)$ is the probability of $x$ and $y$. The simplest example of an expectation is the average or mean, which is $E(x)=\sum_x xp(x)$. (If $p(x)$ is continuous then $\sum$ is replaced by $\int$.)

So altogether, that expression in Wikipedia means that $\hat F$ is the value of the function $F$ which minimizes the expectation of the loss function. In other words we are trying to find an $F$ which is the least-worst of all possible $F$s.

Further down the page that article says "the method tries to find an approximation $\hat{F}(x)$ that minimizes the average value of the loss function on the training set", so here the $E$ is the averaging of the values, the arg min is the minimising, and $L$ is the loss function.