3

Let $X_1,..., X_n$ be a random sample from a distribution with probability density function $f(x;\theta) = (1/2\theta) exp(-|x|/\theta)$ for $-\infty<x<\infty$.

(a) Find the maximum likelihood estimator of $\theta$ and calculate the Fisher (expected) information in the sample.

I've calculated the MLE to be $\sum |X_i|/n$ and I know the definition of Fisher expectation, but I'm getting really stuck with calculating it. I think the $|X_i|$ terms are throwing me off.

Any help in doing this problem would be much appreciated!

Thanks

Brian
  • 63

2 Answers2

4

Recall that $$I(\theta)=-\mathbb{E}\left[\frac{\partial^2}{\partial \theta^2}l(X\,| \,\theta)\right]\,$$ under certain regularity conditions (that apply here), where $I$ is the Fisher information and $l$ is the log-likelihood function of $X$. The log-likelihood function in this case is given by $$\begin{align} l(X\,|\,\theta) &=\text{log}f(X\,|\,\theta) \\&=\log\left(\frac{1}{2\theta}\text{exp}\left(-\frac{|X|}{\theta}\right)\right) \\ &= -\frac{|X|}{\theta} - \text{log}(2\theta)\,\,. \end{align}$$ It follows that $$\frac{\partial}{\partial \theta}l(X \,|\,\theta) = \frac{|X|}{\theta^2}-\frac{1}{\theta} \implies \frac{\partial^2}{\partial \theta^2}l(X \,|\,\theta) = -\frac{2|X|}{\theta^3}+\frac{1}{\theta^2}\,.$$ So, we have $$I(\theta)=-\mathbb{E}\left[-\frac{2|X|}{\theta^3}+\frac{1}{\theta^2}\right]=\mathbb{E}\left[\frac{2|X|}{\theta^3}-\frac{1}{\theta^2}\right]=\frac{2}{\theta^3}\mathbb{E}(\,|X|\,)-\frac{1}{\theta^2}\,.$$ It remains to compute the expectation of $|X|$. To this end, I will set up the integral. By definition of expected value for transformations of continuous random variables, we have $$\mathbb{E}(\,|X|\,)=\int_{-\infty}^{\infty}|x|\,f(x \,|\, \theta)\,\text{d}x=\int_{-\infty}^{\infty}\frac{|x|}{2\theta}\text{exp}\left(-\frac{|x|}{\theta}\right)\,\text{d}x = \theta\,.$$ Note: to compute the integral, alter its form by taking advantage of the fact that $|x|$ is symmetric (and, you can also decompose the integral based on cases for $|x|$).

Thus, the Fisher information is $$I(\theta)= \frac{2}{\theta^3}\mathbb{E}(\,|X|\,)-\frac{1}{\theta^2} = \frac{2}{\theta^2}-\frac{1}{\theta^2}=\frac{1}{\theta^2}$$ For a sample $X_1,X_2,...,X_n$ of size $n$, the Fisher information is then $$I(\theta \,|\,n)=nI(\theta)=\frac{n}{\theta^2}\,.$$ Therefore, by the Cramér–Rao inequality, the variance of any unbiased estimator $\hat{\theta}$ of $\theta$ is bounded by the reciprocal of the Fisher information (this includes the MLE that you have computed, which achieves this lower bound, and is said to be an efficient estimator). In other words, $$\text{Var}(\hat{\theta}) \geq \frac{1}{nI(\theta)} = \frac{\theta^2}{n}\,\,.$$

afedder
  • 2,102
  • but how do I differentiate $|x|$? – Brian May 19 '14 at 10:48
  • Simplify the integral using a symmetric argument, so that you won't have to...but for future reference, look up the signum function $\text{sgn}(x)$. – afedder May 19 '14 at 11:01
  • The substitution I mentioned is one way to compute the integral, but there are other ways...the symmetry of $|x|$ yields the best approach. @Brian – afedder May 19 '14 at 11:02
  • @Brian, in regards to making the integral manageable: what is $|x|$ for $x \geq 0$ and for $x < 0$? You can split the integral into these two cases and then apply the substitution and integration by parts. – afedder May 19 '14 at 11:20
  • thanks, I understand now. Looked up $sgn(x)$ and see why it works now! – Brian May 19 '14 at 11:24
  • No problem whatsoever! Happy to help! @Brian – afedder May 19 '14 at 17:12
  • @Brian note that I computed the information for a single sample of this distribution with parameter $\theta$, but you can generalize to a sample of size $n$ by inserting a sum $\sum_{i=1}^{n}$ before each $|X|$ and replacing each of these with $|X_i|$...however the calculation still reduces to finding the expectation of the $|X_i|$, which is what I have outlined. – afedder May 19 '14 at 21:46
  • yes, thanks, my problem was for a sample size of n but did generalise it – Brian May 21 '14 at 09:30
1

To find MLE of $\theta$ we take likelihood function as $$L(x,\theta) = \frac{1}{(2\theta)^n} \exp \left( \frac{-\sum |x_i|}{\theta} \right).$$ Next we take its $\log$ as $$l(x,\theta) = - n \log 2 \theta - \frac{\sum_{i=1}^n |x_i|}{\theta}.$$ Differentiate w.r.t $\theta$ and equate to zero. $$\frac{\partial l (x,\theta)}{\partial \theta} = \frac{-n}{\theta} + \frac{\sum |x_i|}{\theta^2} = 0$$ yields $$-n \theta + \sum |x_i| = 0$$ hence MLE is $$\theta = \frac{\sum_{i=1}^n |x_i|}{n}.$$

The Fisher information is given as $$I(\theta) = -E \left[ \frac{\partial^2 l(\theta)}{\partial \theta^2} \right]$$ i.e., expected value of the second derivative of the log likelihood $l(\theta)$. $$\frac{\partial^2 l(\theta)}{\partial \theta^2} = \frac{n}{\theta^2} - \frac{ 2 \sum_{i=1}^n | x_i|}{\theta^3}$$ Taking expectation we have $$I(\theta) = - E \left[ \frac{\partial^2 l(\theta)}{\partial \theta^2} \right] = - \left[ \frac{n}{\theta^2} - \frac{2n\theta}{\theta^3} \right] = \frac{n}{\theta^2}.$$

Original images: one, two.

user642796
  • 52,188
SA-255525
  • 187