5

I am trying to rank my neural network, which is trained for binary classification. That is, given a set of input signals, it outputs either a 1 or a 0.

I have a training set, where I have the actual desired outcomes (of 1 or 0).

After I train my network, I check the output to the input. From this, I can easily see how many true positives (TP), false positives (FP), true negatives (TN) and false negatives (FN) I have.

From the TP, FP, TN, FN, I can compute the TPR and the FPR (true and false positive rates).

But I do not know how to compute the AUC score from this data.

I would appreciate any help

Thanks Lyle

Chill2Macht
  • 20,920
Lyle Coder
  • 51

2 Answers2

1

Often, a system is tested with a variety of parameter settings, so that you get a list of [FPR, TPR] pairs, which you plot, and then the area under the curve (AUC) is visually obvious because you can see the "curve" plotted.

When you only have one point, you have to interpolate. So:

  • Let's say you have your classifier which outputs TPR value $t$, FPR value $f$, so the coordinate is $[f,t]$.
  • As well as this, you also know that you could act stupid and always output 1, irrespective of the input data. This gives you a perfect TPR :) but it also gives you a perfect FPR :( so there's another coordinate you can plot: $[1,1]$.
  • By similar reasoning, you can always output zeroes, and achieve $[0,0]$.

You can now interpolate between these three possibilities: draw a piecewise linear curve from $[0,0]$ to $[f,t]$ to $[1,1]$. Then calculate the area under this curve.

If you do this graphically it should be straightforward to see that you get:

$$ \text{AUC} = \frac{t - f + 1}{2} $$

Note that this relies on the piecewise linear interpolation, which is plausible (see Wikipedia article linked above) but not the only way to do it.

Dan Stowell
  • 369
0

To expand a little bit on Dan S' answer, let's talk for a second about how to generate an ROC curve. First, recall that any binary classifier can be thought of as computing a test statistic $T(x)$ on your input data and comparing it to a threshold value $T_0$. The decision rule is then

$$ D(x) = \left\{\begin{array}{ll} 1 & \text{if }T(x)>T_0\\ 0 & \text{else}\end{array}\right. $$ This is true no matter what classifier you use - in your case, $T(x)$ is computed using a neural network. At any rate, the ROC curve is, as stated before, a plot of the (FPF,TPF) values as $T_0$ is varied. So you can't just generate one (FPF,TPF) pair, you'll need to generate many of them. Once you've generated that curve, you could then apply some form of numerical integration (i.e. Riemann sums) to estimate the area under the curve.

However, since you are estimating TPF and FPF from data, these values are actually random variables. Hence any estimate of the AUC is also a random variable, and hence should always come with an estimate of variance or a confidence interval. To do this...

...you should read the classic 1982 paper by Hanley & McNeil, as a start. It's very accessible. It discusses how the AUC is actually equivalent to the probability of getting making the correct choice in a 2AFC (2 alternative forced choice) test. Furthermore, one can estimate this probability using a Wilcoxian statistic, whose variance is known and so a confidence interval can be computed.

This stats stack exchange question is also highly informative.

Community
  • 1
icurays1
  • 17,161