4

Suppose $X\sim \textrm{Binomal}(n,p)$, show that for $\epsilon\geq 1$ $$P\bigg\{{X/n \over p}\geq \epsilon\bigg\}\leq \exp\bigg[-np(\epsilon(\log \epsilon -1)+1)\bigg]$$

I suppose there is a way of using Chernoff Bound of some kind but I don't quite get the right procedure. Any hint or comment will be appreciated.

Roy Alan
  • 105
  • Sure about this? The RHS goes to $0$ when $\epsilon\lt1$ but the LHS does not. – Did Jan 11 '14 at 08:38
  • @Did It should be restricted to $\epsilon \geq 1$, sorry for that – Roy Alan Jan 11 '14 at 08:44
  • OK. Now you should show what you tried to apply Chernoff. – Did Jan 11 '14 at 08:51
  • @Did Basically it should be bounded by $\exp(-nD_{KL}(\epsilon p||p))$ but I don't see the KL-divergence can be simplified into that term – Roy Alan Jan 11 '14 at 09:25
  • Logically, your next step is to include in your post the formula of $D_{KL}(\epsilon p||p)$ you are trying to bound. – Did Jan 11 '14 at 13:29

1 Answers1

2

The Chernoff inequality asserts $$ \Pr(X > a) \leqslant \min_{t > 0} \mathsf{E}\left(\mathrm{e}^{t (X-a)}\right) \tag{1} $$ Since the moment generating function of the binomial distribution is $$\mathrm{E}(\exp(t X)) = \left(1 + \left(\mathrm{e}^t-1\right)p \right)^n$$ The left-hand-side of the inequality $(1)$ for $a = n p \epsilon$ reads: $$ \min_{t > 0} \left( \mathrm{e}^{-t p \epsilon} \left(1 -p + p \mathrm{e}^{t}\right) \right)^n = \left( \min_{t > 0} \mathrm{e}^{-t p \epsilon} \left(1 -p + p \mathrm{e}^{t}\right) \right)^n $$ Differentiating the objective function with respect to $t$ and solving the equation for extremum gives: $$ \exp(t_\max) = \epsilon \frac{1-p}{1-p \epsilon} $$ Restriction of $t_\max>0$ imposes $\epsilon > 1$. This gives: $$ \Pr\left(X > \epsilon n p\right) \leqslant \left(\left(\epsilon \frac{1-p}{1-p \epsilon}\right)^{-p \epsilon} \frac{1-p}{1-p \epsilon}\right)^n = \mathrm{e}^{-n p \epsilon \log \epsilon} \left(\frac{1-p}{1-p \epsilon}\right)^{n (1-p \epsilon )} \tag{2} $$ Using $$ \left(\frac{1-p}{1-p \epsilon}\right)^{n(1-p \epsilon)} = \left( 1 + \frac{n p(\epsilon-1)}{n (1-p \epsilon) } \right)^{n(1-p \epsilon)} \leqslant \mathrm{e}^{n p (\epsilon-1)} \tag{3} $$ where the last inequality follows from $\forall_{a\geqslant 0,x>0} \left(1+\frac{a}{x}\right)^x \leqslant \mathrm{e}^{a}$.

Combining eqs. $(2)$ and $(3)$ we get the desired inequality: $$ \Pr(X > \epsilon n p) \leqslant \exp\left(- n p \left( \epsilon \log \epsilon + 1 - \epsilon \right) \right) $$

Sasha
  • 70,631