2

Let $X_1, ..., X_n$ be independent observations of a random variable $X$ that has with probability $p$ the distribution $\mathcal{U}[0,a]$ and with probability $1-p$ the distrbiution $\mathcal{U}[0,b]$ where $a$ and $b$ are known with $a<b$.

  1. What is the maximum-likelihood estimator for $p$?

  2. Is that estimator efficient? (This means, is the Cramér–Rao bound achieved?)

I'm having quite some trouble with this task for several hours now. I think the density of $X$ is given by $$f_X(x) = p \frac{1}{a}1_{[0,a]}(x) + (1-p) \frac{1}{b}1_{[0,b]}(x).$$

For the trivial case $n=1$ this gives $\hat{p}=1_{[0,a]}(x_1)$ where the expected value of $X_1$ is $a/b$. So the estimator is in fact biased. So this doesn't really help with the second task.

I can't even figure out the general likelihood function. I think it should be something like $L(p) = (p \frac{1}{a})^k ... + ((1-p) \frac{1}{b})^m ...$ where $k$ denotes the number of values from the sample $x_1, ..., x_n$ where $x_i\le a$ and $m=n-k$. But I don't understand how to deal with the indicator functions and if maybe I have to add a factor like with the binomial distribution.

However, I took the raw form $L(p) = (p \frac{1}{a})^k + ((1-p) \frac{1}{b})^{(n-k)}$, $k$ like explained above and tried to maximize by using the log-likelihood. But from that I got $\hat p = \frac{an-bN}{an-bn}$ which doesn't strike me as correct as because of $a<b$ it's negative for small $N$. Nonetheless, I tried to calculate the Fisher information and compared it's inverse to the variance of $\hat p$ - I wasn't very surprised not to get an equality...

Could be that I messed up some calculations, too - dealing with the derivatives of the log-likelihood was a little messy. Now I'm rather frustrated.

Can anyone show me how to solve this?

Amarus
  • 1,697

1 Answers1

1

If the indicators functions confuse you, don't use them. Write instead

$$f_X(x)=\begin{cases} \begin{array}{lc} {p \over a} + \frac{1-p}{b} & x\in (0,a)\\ \frac{1-p}{b} & x\in (a,b) \end{array}\\ \end{cases}$$

which, letting $A=1/a$ $B=1/b$ can also be written as

$$f_X(x)=\begin{cases} \begin{array}{lc} p\, (A-B) +B & x\in (0,a)\\ (1-p) \, B & x\in (a,b) \end{array}\\ \end{cases}$$

If you have $n$ samples, of which $k$ fall in $(0,a)$, then the likelihood is

$$L = \left(p\, (A-B) +B\right)^k \left((1-p)B\right)^{n-k}$$

The derivative of the log-likelihood is then and $$ \frac{k (A-B)}{p\, (A-B) +B} - \frac{(n-k)B}{(1-p)B}$$

which gives me

$$p_0=\frac{k}{n}-\left(1-\frac{k}{n}\right)\frac{a}{b-a}=\frac{1}{b-a}\left(\frac{k}{n}b-a\right)$$

Not however that this is just the critical point, which does not necessarily coincide with the ML estimator. ACtually, that is valid only if $k/n > a/b$, and

$$\hat p_{ML}= \begin{cases} \begin{array}{lc} \frac{1}{b-a}\left(\frac{k}{n}b-a\right) & k \ge n \frac{a}{b}\\ 0 & elsewhere \end{array}\\ \end{cases}$$

It doesn't seem to be consistent, as, when $n\to \infty$ $k/n \to p$ (in prob.), hence $\hat p_{ML}$ won't converge to $p$. Edited: sorry for the errors; actually $n/k \to p+(1-p)a/b = [p(b-a)+a]/b$ and hence the estimator should be at least asymtotically unbiased.

leonbloy
  • 63,430
  • Thanks for the input. Is there any more agreeable way to check for efficiency than to actually compute the fisher information and compare it's inverse to $Var(\hat p)$? – Amarus Jun 13 '13 at 01:48
  • I'd say that this estimator is biased (even asymtotically), so it cannot be efficient (actually, the Cramer Rao bound applies only to unbiased estimators – leonbloy Jun 13 '13 at 02:20
  • BTW, you might have better answers at http://stats.stackexchange.com/ – leonbloy Jun 13 '13 at 02:20
  • Why would $\frac{k}{n} \to p$ for $n \to \infty$? Values from $U[0,b]$ have lie within $[0,a]$ with a probability of $\frac{a}{b}$, i.e. $\mathbb{P}(X \leq a) = p + (1-p)\frac{a}{b}$ and thus $\frac{k}{n} \to p + (1-p)\frac{a}{b}$, no? – fgp Jun 13 '13 at 08:38
  • I don't understand why biasedness is a problem. We introduced the CR bound in a general form that makes no use of unbiasedness: $$Var_\theta(T) \ge (\partial/\partial \theta E_\theta(T))^2 / I(\theta),$$ where $I(\theta)$ is the Fisher information. The usual CR bound follows from this in the unbiased case. With this, we can define efficiency independent of unbiasedness, I guess? – Amarus Jun 13 '13 at 09:06
  • leonbloy: The formula at the end is wrong (but the first error is when $p_0$ appears, hence you should trace it easily). – Did Jun 13 '13 at 09:37
  • @fgp You're right, fixed - thanks. – leonbloy Jun 13 '13 at 12:37
  • @Did Fixed, thanks. – leonbloy Jun 13 '13 at 12:39
  • Right. Note that $\hat p$ is unbiased, for every $n\geqslant1$. – Did Jun 13 '13 at 12:42
  • @fgp Re your last comment to your answer, let me quote myself: starting from l'(p). – Did Jun 13 '13 at 14:54
  • @Did: I think the estimator would be unbiased it we'd just choose $\hat p = p_0$; but $\hat p_{ML} = max(p_0,0)$, the truncation should introduce a (asympotically negligible) bias. – leonbloy Jun 13 '13 at 15:03
  • leonbloy: Hmmm... I am stupid and you are absolutely right. Thanks for the explanation. – Did Jun 13 '13 at 15:11
  • @Amarus: The CR bound can be extended to biased estimators, but the concept of "efficient" gets a little trickier. For one thing, the variance now is not an useful efficiency measure (it's trivial to build a zero-variance biased estimator: just pick a constant), one should instead compare mean squared errors. See eg the CR Wikipedia article. – leonbloy Jun 13 '13 at 15:25