3

Apologies if this is not the correct area to post this but, I have a question about the specific wording of an assignment. We have been given this sentence, basically

The occurrence of false positives [in some experiment] is 40%

What does this mean? Does it mean that 40% of positives are false? Or 40% of all tested patients are specifically both false and positive? Or say 40% of those that are false are positive? I'm quite confused as to what this means, and none of the people I have asked understand either. Thanks

Gabe
  • 832

3 Answers3

4

For binary classifiers (that output Negative or Positive), there are four types of outcomes:

True Positive -> The classifier reports positive and is correct.

False Positive -> The classifier reports positive but is wrong.

True Negative -> The classifier reports negative and is correct.

False Negative -> The classifier reports negative and is wrong.

40% false positives (fp) out of all responses does not tell you much about the other responses. But all together they sum up to 100%. Also: it does not automatically tell you how it will perform in the future. Also, the result is always relative to the true label distribution: If you tested it on 60 positive and 40 negative labels, 40% false positives means, that it misclassified 100% of the negatives, since there are only 100 instances and the positives cannot be classified as false positives, they need to come from the negatives. Since the percentage is always relative to the whole population, 40% of 100 are 40 which leaves no negatives that are classified as true negatives.

So you need all 4 types of outcomes and also the label ratios of the test set to judge your result.

TP + TN + FP + FN = 1.0 TP + TN = Rate of correctly classified over both outcomes, positive and negative classes. FP / TP = Error rate for positives to be missclassified. Since you are missing the TP, you cannot say how many positives would be missclassified.

Maybe something that could also help to understand: Think about a classifier that always predicts positive, no matter what. What would be your TP, TN, FP, FN be for the two following cases:

a) You give it 60% positive and 40% negative instances. b) You give it 99% positive and 1% negative instances.

Answer:

! a) TP = 60%, TN = 0%, FP = 40%, FN = 0%

! b) TP = 99%, TN = 0%, FP = 1%, FN = 0%

Community
  • 1
Decrayer
  • 56
  • 2
  • So you are saying given the wording of the question, FP / 100% = 40%? To give some context, this is a medical situation where the occurrence of false positives compared to all patients should be quite low. – Gabe Nov 25 '19 at 15:44
  • Medical or not, FP is one of the four possible outcomes and the percentage value is always relative to the total number of instances in your test, not just the negatives. Therefore it is very important to know how many have been tested on and how the balance between positives and negatives in that set was. Also, to not confuse oneself, it is better to use absolute numbers for TP, TN, FP and FN. That would also convey some understanding about how reliable the numbers are. 40% if you only tested on 10 people is different from testing on 100.000. – Decrayer Nov 25 '19 at 15:52
  • One clarification though: There is a difference between False Positives(FP) and False Positive Rate (FPR)! False positive rate is usually FP / (TN+FP). But since the question explicitly says "False Positives" and not "False Positive Rate" I assume it means the first. – Decrayer Nov 25 '19 at 15:58
  • Thank you. This is exactly what I needed to hear. Some problems in the packet contained True Positive and some contained True Positive Rate. That makes a lot more sense then. – Gabe Nov 25 '19 at 15:59
3

It means (mostly in medical testing) that the binary outcome of the model is falsely indicated as positive. Meaning that the test says that the condition is present, while in reality, it is not.

This is important in statistical hypothesis testing and especially in medical testing, because it can create some false interpretation of the accuracy of a test. Let me clarify with an example:

I recently read an article in a Belgian newspaper that said that a test was $99\%$ accuracy, but if you tested positive, you only had $1\%$ chance that you actually had the disease. The ratio of people that had this disease was $$\frac{1}{10000}$$ and claimed to be $99\%$ accurate. So let's assume you have $1.000.000$ people, $100$ people should have the disease. But if you were out to test it, $99$ of these $100$ would get a true positive and $1$ of these $100$ people would get a false negative. On the other hand, of the $999.900$ people who don't have the disease, $989.901$ people would get a true negative and $9999$ people would get a false positive. This means that the probability you actually would have the disease if you were to test positive is only :

$$\frac{99}{10098} \approx 0.98 \%..%$$

Steven01123581321
  • 1,419
1

Let $D$ indicate the presence of a disease. Let $+$ indicate that the experiment shows a positive and a $-$ shows a negative. Thus,

($+ \mid D$) indicates that the person has the disease, and the experiment showed a positive result, i.e a true positive.

($- \mid D$) indicates that the person has the disease, and the experiment showed a negative result, i.e a false negative.

($+ \mid D^c$) indicates that the person does not have the disease, and the experiment showed a positive, i.e a false positive, which has the 40% chance.

($- \mid D^c$) indicates that the person does not have the disease, and the experiment showed a negative, i.e a true negative.

Thus, the statement "The occurrence of false positives [in some experiment] is 40%" indicates that a 40% of people that are not sick with the disease will show a positive in the test. In other words Pr($+ \mid D^c)=$0.4=40%$

sirferrum
  • 339
  • 2
  • 17
  • 1
    Your explanation doesn't seem to gibe with your conclusion. When you say $+\cap D^c$ has the $40%$ rate, that seems to mean, at least to me, that $40%$ of the people examined will test positive, although they don't have the disease: $\Pr(+\cap D^c)$, whereas your conclusion seems to talk about $\Pr(+|D^c)$. – saulspatz Nov 25 '19 at 15:20
  • @saulspatz You are completely correct, any suggestions on how to correct it? – sirferrum Nov 25 '19 at 15:23
  • 1
    I'm afraid I don't know which is correct, the explanation or the conclusion. I would suggest that you couch the entire answer in terms of either joint or conditional probabilities, whichever is appropriate. – saulspatz Nov 25 '19 at 15:26
  • @saulspatz I agree. – sirferrum Nov 25 '19 at 15:26