1

I have recently seen some graphs like this in the context of classificaiton:

enter image description here

Source: http://on-demand.gputechconf.com/gtc/2014/webinar/gtc-express-convolutional-networks-webinar.pdf, page 85

How should one interpret these graphs?

My try

In this case, it is about face recognition I guess there is some threshold parameter which leads to a decision if something is a face. The lower this parameter is, the more parts of images get "recognized" as a face. So both, the true positive value and the false positive value increase.

Now the two definitions

\begin{align} \text{true positive rate} &:= \frac{TP}{TP+FN}\\ \text{false positive rate} &:= \frac{FP}{FP+TN} \end{align}

The true positive rate gives the proportion of actual positives which are correctly identified as such. (Higher is better)

The false positive rate gives the proportion of falsely identified positives amongst all actual negatives. (lower is better)

Obviously, the most right curve (combined Joint Baysian) is worst, because for a fixed true positive rate it has always the highest false positive rate.

But how would one decide if the red or the black curve is better? Isn't there the need of a "cost" of false positives / negatives to decide that? What information can I derive from such diagrams?

Martin Thoma
  • 9,821
  • I am quite sure that the tag is wrong. However, I don't know what good tags would be. Please correct it if you know it. Also, English is not my mother tongue. Feel free to correct any sentence that seems to be a language mistake. – Martin Thoma Sep 27 '14 at 17:50

1 Answers1

2

For decision problems, you do need a loss function for each type of error. However, this type of graph is useful because it identifies "dominated" and "dominating" solutions. Ill explain below:

The ideal form of the curve would be a vertical step function of height 1 at false-positive rate = 0. The worst possible curve would be the inverse of this.

Now, lets say there are two curves of true positive rate as a function of false positive rate $(\alpha)$, $f_1(\alpha)$, $f_2(\alpha)$. If $f_1\geq f_2\;\;\forall \alpha$, then $f_1$ will incur less loss than $f_2$ regardless of the loss functions, as long as the losses (L) for each type of error (False positive=FP, false negative=FN) are additive:

$L_{f_1}=(1-f_1)L_{FN}+\alpha L_{FP}$

$L_{f_2}=(1-f_2)L_{FN}+\alpha L_{FP}$

$L_{f_2}-L_{f_1}= L_{FN}[f_1-f_2]$

$f_1(\alpha)\geq f_2(\alpha) \implies L_{f_2}-L_{f_1}\geq 0$

Thus, the condition $f_1(\alpha)\geq f_2(\alpha)$ indicates that $f_1$ will lead to less loss than $f_2$ regardless of the loss functions.

Applying this to your actual data shows that Deep-Face Ensemble dominates all other approaches except human-cropped, which appears to be the best choice if very low fase positives are needed, but inferior for higher allowable false positives.