ROC curves and AUC: judging a classifier at every threshold
A ROC curve plots the true positive rate against the false positive rate at every classification threshold a model can use. The area under that curve, AUC, expresses the probability that a random positive example is ranked above a random negative one. AUC helps compare models on balanced data, while the right threshold depends on the costs of false positives and false negatives.
ROC and AUC basics
A receiver operating characteristic (ROC) curve is a visual representation of how a binary classifier performs across all classification thresholds, according to Google's Machine Learning Crash Course. To construct the curve, one computes the true positive rate and false positive rate at each threshold and then graphs the former against the latter.
The area under that curve, AUC, represents the probability that a randomly chosen positive example is ranked higher than a randomly chosen negative example. A perfect model has an AUC of 1.0, while a random guesser has an AUC of 0.5 and a diagonal ROC curve.
The role of classification thresholds
A classification threshold is the probability value that separates the positive class from the negative class. As explained in Google's thresholding guide, examples with a predicted probability above the threshold are assigned to the positive class, and those below are assigned to the negative class. A threshold of 0.5 is intuitive but not always ideal when classes are imbalanced or error costs differ.
Different thresholds produce different numbers of true positives, false positives, true negatives, and false negatives, which are summarized in a confusion matrix. The threshold that offers the best trade-off depends on the specific costs of each type of error.
Practical considerations for model comparison
According to Microsoft's AutoML evaluation guide, ROC AUC and average precision compare ranking across decision thresholds, while accuracy, precision, recall, and F1 evaluate predictions at a selected threshold. For imbalanced data, macro or per-class results and the confusion matrix are more informative than accuracy or micro averages.
The Machine Learning Glossary points out that accuracy is highly misleading for class-imbalanced datasets. A model that always predicts the majority class can still achieve a very high accuracy while having no predictive power.
Selecting a scoring function
The scikit-learn model evaluation guide advises that the choice of scoring function should align with the intended purpose of the prediction. If a decision has to be made from a predicted probability, metrics derived from the confusion matrix measure different aspects of that decision, such as false positives and false negatives.
The guide distinguishes between predicting the outcome and making a decision. For classification, using a strictly consistent scoring function, such as Brier score or log loss, is recommended for training and evaluation. The selection of a scoring function should reflect the task's objective.
| Metric | What it measures | When to use |
|---|---|---|
| ROC AUC | Ranking quality across thresholds | Compare how well models rank positives above negatives |
| Accuracy | Proportion of correct predictions | When class frequencies and error costs are balanced |
| Average precision | Summary of the precision-recall curve | Compare ranking when the positive class is uncommon |
| F1 score | Harmonic mean of precision and recall | Balance false positives and false negatives |
| Log loss | Negative log-likelihood of true labels | Evaluate probability quality and penalize confident errors |
Frequently asked questions
What does a ROC curve show?
A ROC curve shows the trade-off between true positive rate and false positive rate for every possible classification threshold.
What does AUC represent?
AUC is the probability that a randomly chosen positive example is ranked higher than a randomly chosen negative example.
When is AUC a reliable metric?
AUC is reliable for comparing models on roughly balanced datasets; for imbalanced data, precision-recall curves may be better.
How do you choose a classification threshold?
The threshold depends on the costs of false positives and false negatives; points on the ROC curve near (0,1) offer good trade-offs.
Why is accuracy misleading on imbalanced datasets?
Because a model that always predicts the majority class can have high accuracy but no predictive power.