2

Suppose you have two sets of data with known population variances and want to test the null hypothesis that two means are equal, ie. $H_{0}: \mu_{1} = \mu_{2}$ against $H_{1}: \mu_{1} > \mu_{2}$. There's a certain way I want to think about it, which is the following:

\begin{align} P(\mu_{1} > \mu_{2}) &= P(-(\mu_{1} - \mu_{2}) < 0) \\ &= P\left(\frac{\bar{x}_{1} - \bar{x}_{2} - (\mu_{1} - \mu_{2})}{\sigma_{\delta \bar{x}}}<\frac{\bar{x}_{1} - \bar{x}_{2}}{\sigma_{\delta \bar{x}}} \right) \\ &=P\left(z < \frac{\bar{x}_{1} - \bar{x}_{2}}{\sigma_{\delta \bar{x}}} \right) \end{align}

To me, this 'derivation' makes it perfectly clear what's actually going on. You're actually calculating the probability that $H_{1}$ is true and not just blindly looking up some $z$-score. However, now suppose that $H_{1}: \mu_{1} \neq \mu_{2}$. The problem with this is that the method I just described doesn't seem to work. If I write

$$ P(\mu_{1} \neq \mu_{2}) = P(\mu_{1} < \mu_{2}) + P(\mu_{1} > \mu_{2}) $$

Then all that happens is $P(\mu_{1} \neq \mu_{2}) = 1$. I think I'm probably not interpreting the above equation correctly.

Mr. G
  • 1,058
  • What do you mean by $\sigma_{\delta\bar x}$? You say there are known population variances, which would mean one variance for $\bar x_1$ and another for $\bar x_2$. What does the lower-case $\delta$ refer to? ${}\qquad{}$ – Michael Hardy Nov 05 '14 at 01:52
  • 1
    @MichaelHardy I mean the standard deviation of $\bar{x}{1} - \bar{x}{2}$. The $\delta$ is a difference. We would have $\sigma_{\delta \bar{x}} = \sqrt{\sigma_{1}^2/n_{1} + \sigma_{2}^2/n_{2}}$. – Mr. G Nov 05 '14 at 01:53
  • You are not finding the probability that $H_1$ is true. – Michael Hardy Nov 05 '14 at 04:00
  • @MichaelHardy Saying you can reject the null hypothesis at the 95% level is something like saying there's a 95% chance the alternate hypothesis is true. – Mr. G Nov 05 '14 at 04:13
  • I think you're confusing "$95%$ level" with "$5%$" level. But at any rate, you are committing the "prosecutor's fallacy" (Google that term) --- a simple error in probability. $1/20$ of the time you get a false positive. So suppose $1/20$ of the time the evidence matches whichever suspect you've arrested. $1/20$ of a million is $50,000$, so in a population of a million, there are $50,000$ who would match. Thus a match would mean a $1/50,000$ probability of guilt, not $1/20$ as you suggest. ${}\qquad{}$ – Michael Hardy Nov 05 '14 at 04:20
  • @MichaelHardy I'm not saying that $P(\mu_{1} > \mu_{2})$ is the according-to-Gauss probability that $H_{1}$ is true, but it's at least suggestive. Furthermore, if you set that expression equal to $0.95$ it's as if you're setting $P(H_{1}) = 0.95$ and as if you're setting $\alpha = 0.05$. Then the null hypothesis is rejected if $P(z > z_{0}) > 0.95$, which is what you would just start with to "actually" do this. If that only works because of a coincidence then okay. – Mr. G Nov 05 '14 at 04:43
  • If it only works because of a coincidence, then how is this better than "blindly looking up some $z$-score"? (For that matter, if you understand how standard statistical tests work, then why would you call any of the steps "blindly looking up" anything? When we look things up, it should be with full awareness of why we do it that way.) – David K Nov 05 '14 at 06:27
  • @DavidK I said it was better than blindly looking up a $z$-score before it was brought to my attention that it only works because of a coincidence. I call it "blindly looking up" because I've never seen any derivation or explanation of why the $z$ score works the way it does, it's just stated as a result. As you say, I wanted to be aware of why it's done that way so I tried to think of a possible argument that leads to the result. If it's wrong then it's wrong and I'll have to change it. I'm not going to literally publish a paper with this in it. – Mr. G Nov 05 '14 at 14:03
  • @Mr.G I see I've been a bit unfair. You seem to have made a good-faith effort to understand some concepts that are often not well explained. Worse, the methods don't quite work the way we would expect (at least not the way I naively would have expected), so they really need explanation. I'll try to come up with something more helpful and edit it into my answer if I can. – David K Nov 05 '14 at 14:29
  • @DavidK Thank you very much. – Mr. G Nov 05 '14 at 14:36
  • I thought I would just point out that if we conduct such a hypothesis test as it was originally conceived by Fisher, the statement $\Pr[\mu_1 > \mu_2]$ is nonsensical: $\mu_1$ and $\mu_2$ are unknown but fixed parameters of the underlying distributions from which the samples were drawn, and as such, are not random variables and have no probability measure associated with them. – heropup Nov 05 '14 at 17:00
  • For example: suppose I have a fair penny and an unfair quarter. The probability that the penny lands heads on a given toss is known to be $p_1 = 0.5$ and the probability the quarter lands heads on a given toss is known to be $p_2 = 0.75$. If I give you these coins for you to inspect, but I don't tell you $p_1$ and $p_2$, then there is a definite answer to the question of whether $p_1 > p_2$ because these values are fixed, not random--they are simply unknown to you. With enough coin flips, you can test the hypothesis, though you can never be 100% certain. – heropup Nov 05 '14 at 17:04

3 Answers3

1

In your way of thinking, you decided to set $$z = \frac{\bar{x}_1 - \bar{x}_2 - (\mu_1 - \mu_2)}{\sigma_{\delta \bar{x}}},$$ and from this you conclude (correctly, I think) that $P(\mu_1 > \mu_2) = P\left(z < \frac{\bar{x}_1 - \bar{x}_2}{\sigma_{\delta \bar{x}}}\right).$ The question is, if you cannot assign a probability distribution to $\mu_1 - \mu_2,$ how do you compute $P\left(z < \frac{\bar{x}_1 - \bar{x}_2}{\sigma_{\delta \bar{x}}}\right)$? And of course if you can assign a probability distribution to $\mu_1 - \mu_2,$ then you can use that distribution to compute $P(\mu_1 - \mu_2 > 0)$ directly.

Note that if $P(\mu_1 = \mu_2) > 0,$ then $$P\left(z = \frac{\bar{x}_1 - \bar{x}_2}{\sigma_{\delta \bar{x}}}\right) > 0,$$ because you have defined $z$ in such a way that it is simply $\frac{\bar{x}_1 - \bar{x}_2}{\sigma_{\delta \bar{x}}}$ if the two populations have the same mean.

When you conclude that $P(\mu_1 \neq \mu_2) = P(\mu_1 < \mu_2) + P(\mu_1 > \mu_2) = 1,$ you are assuming that $z$ has zero probability to equal $\frac{\bar{x}_1 - \bar{x}_2}{\sigma_{\delta \bar{x}}}$ exactly. This would be true if $z - \frac{\bar{x}_1 - \bar{x}_2}{\sigma_{\delta \bar{x}}}$ had a continuous distribution, but how do we know it does?

Edit: There are some quite reasonable motivations (from a practical point of view, if not a mathematical one) for attempting some kind of approach to coming up with a value for $P(\mu_1 > \mu_2)$ as explained in the question. The main motivation is that this is the way we seem to want to be able to think about statistics: just how much weight (i.e. likelihood) should I assign to the possibility that certain facts are true? Unfortunately it's often very difficult to make a convincing case for a particular value of such a likelihood. Instead, what frequentist statistics gives us is an apparently roundabout statement that if a certain fact were not true (that is, if that fact's "null hypothesis" were true instead), it would have been extremely unlikely for us to have made the observations we just made.

A more precise and succinct explanation is given in this answer to another question.

To test the hypothesis that $\mu_1 > \mu_2$ in the posted question, we can define the null hypothesis as $\mu_1 \leq \mu_2.$ Now, having obtained samples from the two populations, how likely is it that we would have gotten samples "like those" if the null hypothesis were true?

If $\bar x_1 \leq \bar x_2$ the answer is "likely enough," so we only have an interesting statistical test in the case where $\bar x_1 > \bar x_2.$ Assuming that's the kind of sample results we got, then among all possible ways the null hypothesis could be true, the one that gives us the best chance to obtain samples "like" the ones we did is if $\mu_1 = \mu_2.$ But if we assume that $\mu_1 = \mu_2,$ then prior to taking the samples, $\frac{\bar{x}_1 - \bar{x}_2}{\sigma_{\delta \bar{x}}}$ was a random variable with a standard normal distribution (mean zero, variance $1$).

Suppose that having taken our samples, we find that $\frac{\bar{x}_1 - \bar{x}_2}{\sigma_{\delta \bar{x}}} = 2.38.$ That's a relatively extreme value for a standard normal variable; $99$ times out of $100$ the value of a standard normal variable will be less than that. In fact, the probability is $0.99134$ that a standard normal variable will have a value less than $2.38.$ (I know this because someone computed that probability and put it in a table, and I looked it up there.) There is therefore less than a $1$% chance that we would have observed a sample mean $\bar x_1$ so much larger than $\bar x_2$ if the population mean $\mu_1$ were not actually at least a little bit larger than $\mu_2.$ We therefore reject the null hypothesis.

Using samples drawn from populations with continuous distributions, it appears impossible to test statistically whether the means of two populations are exactly the same, because even if they were the same, with probability $1$ we would still observe different sample means. (See also this answer on that topic.)

There is another kind of statistics called Bayesian statistics that (as far as I understand it) does assign probabilities to the truth of statements that you might want to prove or disprove, but only by using observations of experiments to modify probability assignments that one was able to make before the experiment.

Community
  • 1
David K
  • 98,388
  • Well I can compute $\bar{x}{1}$ and $\bar{x}{2}$ from the samples and $\sigma_{\delta \bar{x}}$ using the sample sizes and the known population variances. Then I just look up $P(z < z_{0})$ from a standard normal table, where $z_{0}$ is just a number now. For $P(\mu_{1} < \mu_{2}) + P(\mu_{1} > \mu_{2})$ the reason I get zero is that when I apply the same reasoning to each of those two terms, the sum is identically one because continuous distributions "don't care" whether the inequalities are strict or not. – Mr. G Nov 05 '14 at 03:40
  • The point is, you are assuming that $\mu_1 - \mu_2$ is continuously distributed. So of course you get probability $1$ that they're not equal. In fact, the whole idea of computing $P(\mu_1 - \mu_2)$ ignores all of frequentist statistics, and the idea of computing it without a prior ignores all of Bayesian statistics, so it's really unclear what kind of statistics you're doing. – David K Nov 05 '14 at 06:20
  • @Mr.G In fairness (and because I do have sympathy for your point of view, since I had similar misgivings about statistical methods when I first studied them mathematically), I've attempted to explain why the standard techniques are performed the way they are. I'm sure there's more to be said on that topic, and perhaps there are better explanations elsewhere. (Unfortunately it seems there are also worse explanations in some places.) – David K Nov 06 '14 at 05:42
  • Thank you. This makes much more sense. – Mr. G Nov 17 '14 at 00:35
0

If is probably that $\mu_1<\mu_2$, then $\mu_1\geq\mu_2$ is rejected, then not is possible that holds $\mu_1<\mu_2$ and $\mu_1>\mu_2$ simultaneusly.

AsdrubalBeltran
  • 4,212
0

You have $$ \frac{\bar X_1-\bar X_2}{\sigma_{\delta \bar X}}\sim N(0,1). $$ If you want to use $\bar X_1-\bar X_2$ as the test statistic, then you would reject the null hypothesis if $\bar X_1-\bar X_2>c$, for some critical value $c$. Often one chooses $c$ by deciding how much probability of Type I error one is willing to tolerate --- a subjective economic decision. One can then think about probability of Type II error as a function of $\mu_1-\mu_2$, and that probability would depend on the value of $c$ and on the two sample sizes.

None of this gives you a probability distribution of $\mu_1-\mu_2$. For that you'd need a prior distribution, and then you'd have to find the posterior distribution, i.e. the conditional distribution given the two samples.

PS: A bit of back-of-the-envelope scratchwork suggests that the likelihood-ratio test statistic may be: $$ \left.\begin{cases} \left(\dfrac{\bar X_1 - \bar X}{\sigma_2/\sqrt{n_1}}\right)^2 + \left(\dfrac{\bar X_2-\bar X}{\sigma_2/\sqrt{n_2}}\right)^2 & \text{if }\bar X_1\ge \bar X_2, \\[10pt] {}\qquad 0 & \text{otherwise}, \end{cases}\right\} $$ where $\bar X$ is simply the grand mean $\dfrac{n_1 \bar X_1 + n_2 \bar X_2}{n_1+n_2}$.

Michael Hardy
  • 1
  • I'm just wondering why the set of algebraic manipulations I outlined for the first part don't work for the second part. Are you saying it's just a coincidence that they work for the first part? – Mr. G Nov 05 '14 at 02:35
  • @Mr.G : Your "algebraic manipulations" start out be referring to "$P(\mu_1>\mu_2)$" and right there it's already nonsense, so it doesn't work for any "part". ${}\qquad{}$ – Michael Hardy Nov 05 '14 at 15:34