1

Suppose I want to build a model to predict if someone has a certain disease. 1 means they have the disease, 0 means they don't. From what I've read, my model's output domain is all real numbers. So my model's output might be

Patient Number, Model Prediction
1, -3
2, 2
3, 0.2
4, -3
5, 0.2

Basically I'm totally confused as to how this makes sense. How would I draw a roc curve for this? (Assume the actual results were 1,1,1,0,0.) This problem stems from a Kaggle competition that I'm trying to understand here

Ben
  • 369

1 Answers1

0

For each patient (let us call him $p_i$) you have tabulated a score $s_i$. For example, the patient $p_3=3$ has received the score $s_3=0.2$.

The computation of the score can be obtained, for example by a logistic regression. In your case the scores are already given in the question but we do not know how did you produced such numbers.

The output of your whole set of computations/analysis would be a mapping

$$D:\{p_i\}_{i=1,\dots,5}\rightarrow \{0,1\},$$

which gives the health status $D(p_i)$ of each patient $p_i$. If the patient has the disease his status would be equal to $1$; if he is healthy then the status would be equal to $0$.

It seems that in your case the heath status is already given, as you talk about "actual results", specifying the string $(1,1,1,0,0)$.

In your case

  • The scores $s_i$ are given
  • The health status for each patient is given (called actual results)

You are ready to draw the ROC curve. To do it you need to vary the score $s_i$ from its minimum to its maximum while counting the number of patients with $0$ and $1$ as health status. If your model is sensitive, then you expect that all patients with a low score are healthy, while those with a higher score are all ill, or viceversa depending on the definition of your score.

A complete separation between the two sets can be obtained only in theory. A "real world" model will always show a certain number of "errors", which lower the AUROC value, i.e. the surface under the ROC curve.

Avitus
  • 14,018