5

Intuitively, when a random variable $X$ has a low variance, a sufficiently smooth function of that random variable will have an expected value which is close to that same function applied to $E[X]$. That is, $$ f(E[X]) \approx E[f(X)] $$

This is obviously true when $X$ takes on only one value, and appears in a few special cases I have worked out. Is this a well-known theorem? If not, how could this intuition be formalized?

Display Name
  • 671
  • I'm not super confident on this, but viewing the expected value as an integral and $X =g $ as a function, we're asking for when $f (\int g) = \int(f(g))$, or when $f(\int g) - \int(f(g)) < \epsilon$ for some fixed sufficiently small $\epsilon > 0$. – User203940 Feb 26 '19 at 20:39

1 Answers1

11

$E[X]$ is just a number; I will call it $x_0 := E[X]$ to make the Taylor expansion look more familiar.

Here is one set of assumptions that leads to a bound on your approximation.

Suppose we have the Taylor expansion $$f(x) = f(x_0) + f'(x_0) (x-x_0) + f''(\xi_x) (x-x_0)^2$$ where $\xi_x$ lies between $x_0$ and $x$. If we plug in the random variable $X$ for $x$ and take the expectation, the second term on the right-hand side becomes $E[X-x_0] = 0$ since $x_0 := E[X]$, and we have $$E[f(X)] = f(x_0) + E[f''(\xi_X) (X-x_0)^2].$$ If $f$ has bounded second derivative, i.e. $|f''(x)| \le M$ for all $x$, then we have $$|E[f(X)] - f(x_0)| \le M E[(X - x_0)^2] = M \text{Var}(X).$$

So indeed, this is one way of formalizing your "low variance" and "sufficiently smooth" intuitions.

angryavian
  • 89,882
  • 4
    Note that we should have an equality when $f$ is linear, which this bound recovers (as $M$ will be 0). – Mark Schultz-Wu Feb 26 '19 at 20:40
  • @Mark Yes, that's a good observation to point out. – angryavian Feb 26 '19 at 20:42
  • +1 this is an excellent answer. it is not every day that i learn a new aspect of something as basic as Jensen's inequality! this clearly shows both the effect of convexity / concavity ($f'' > 0$ or $< 0$) and the variance. – antkam Feb 26 '19 at 22:51