5

So, I've been reviewing some of my old stats courses in preparation for an interview I have in a couple of days. I'm a bit stuck on a particular question and hope you could help.

A drug trial gives the result that the drug works better than the placebo, with 95% confidence. What exactly does this statement mean? What further assumptions are needed to be able to deduce that the probability of the drug working is actually 95%?

My answer to the first part is... 95% confidence means that there is a 1 in 20 chance that the difference could have been observed by chance i.e. if the experiment was conducted many times.

Any suggestion for part 2?

Thanks in advance.

M Turgeon
  • 10,419
Youler
  • 71

4 Answers4

1

95% confidence means that 95% of the time, your test will produce intervals that contain the true mean.

In your case, it means that the actual success of the drug is higher than the actual success of the placebo 95% of the time (from a large number of tests).

The assumptions are: the sample size is adequately large, there do not exist biases in how the test was conducted, etc.

zzzzzzzzzzz
  • 1,072
  • Do confidence levels and intervals have meaning only when the results are normally distributed and the sample size is large etc or can they be used in non-normal small-sample situations as well? – Dilip Sarwate Jun 05 '12 at 16:40
  • Confidence intervals have the same meaning in context when the distribution is non-normal and the sample size is small. Yet, they are calculated differently than a confidence interval for a normal distribution. It might be difficult to attain a 95% confidence interval with those conditions though (as an increased sample size increases confidence levels).
    Look at the derivation for more information http://en.wikipedia.org/wiki/Confidence_interval#Theoretical_example     – zzzzzzzzzzz Jun 05 '12 at 16:56
1

First part: The statement means there is a $\lt5\%$ probability that the observed success rate or better would occur (by chance), given that the drug is only as effective as the placebo. The other answers explain this well.

Second part: I assume that the question means "What further assumptions are needed to be able to deduce that the probability of the drug working better than the placebo is actually 95%?

Let $H_0$ be the null hypothesis: the drug is only as effective as the placebo. Let $H_1$ by the hypothesis: the drug is more effective than the placebo. Let S be the event that the p-value is less than 0.05 (i.e. the event that our trial gives the result that the drug works better than the placebo, with 95% confidence).

We have $\mathbb{P}(S |H_0) = 0.05$, from the definition of p-value.

By Bayes' Theorem: $\mathbb{P}(H_1 | S) = \frac{\mathbb{P}(S | H_1)\mathbb{P}(H_1)}{\mathbb{P}(S | H_1)\mathbb{P}(H_1) + \mathbb{P}(S | H_0)\mathbb{P}(H_0)}$

Therefore we need to know two things: $\mathbb{P}(H_1)$, the prior probability of hypothesis $H_1$ being true, and $\mathbb{P}(S | H_1) = 1- \mathbb{P}(S^c | H_1) = 1 - \beta$, where $\beta$ is the probability of making a type II error (failing to reject the null hypothesis when it is false).

Ronnie268
  • 303
0

I assume you still mean that the drug works better than placebo with the second part of your question. That question is not addressed directly using frequentist statistics. If you use the Bayesian framework of inference you can put a prior probability on the hypothesis that the drug works better than placebo. Then given the observed results you compute the likelihood function assuming some parametric distribution for the difference between the drug effect and the placebo effect, apply Bayes rule and determine a posteriori probaility that the drug works. It will depend on the prior probability and the strength of the data supporting the hypothesis that the drug works to determine whether or not this aposteriori probability is >0.95 or not.

The key part of this that was not mentioned in the question is that there is some measure of effectiveness that can be estimated for subjects taking the drug and subjects taken the placebo and that the difference between their averages can be used as a measure of whether or not the drug works well. There are no assumptions of normality needed, but the method testing the hypothesis and drawing inference will depend on whether or not you make parametric assumptions.

Michael R. Chernick
  • 4,639
  • 2
  • 19
  • 24
0

First part: the probability that the trial result or something better would be seen by chance (i.e. the proportion if the trial was repeated a very large number of times) , if the drug is in fact just as good as a placebo, is less than or equal to 5%.

Second part: A key assumption (or estimate from past experience) is an a priori figure for how often drugs in fact work better than placebos.

Henry
  • 157,058