The confusion matrix: true and false positives and negatives explained
The confusion matrix is a table that lays out the four possible outcomes of a binary classifier: true positives, false positives, true negatives, and false negatives. These outcomes are used to calculate metrics such as accuracy, precision, recall, and the false positive rate. The choice of classification threshold directly affects the counts in each cell, so understanding the matrix helps in selecting the right threshold for a given task.
What the confusion matrix shows
A confusion matrix is a two-by-two table that compares a model's predictions with the actual ground truth for a binary classification problem. The rows represent the predicted class, and the columns represent the actual class. Each cell counts one of four outcomes: true positive, false positive, false negative, or true negative, as described in Google's thresholds and confusion matrix guide.
In the spam email example, a true positive is a spam email correctly sent to the spam folder, while a false positive is a legitimate email that is mistakenly filtered as spam. A false negative is a spam email that slips into the inbox, and a true negative is a legitimate email correctly delivered. The row totals give all predicted positives and negatives, while the column totals give all actual positives and negatives, regardless of the model's classification.
The matrix is a foundation for many evaluation metrics. For instance, accuracy is the sum of true positives and true negatives divided by the total number of predictions, as defined in Google's machine learning glossary. The glossary also notes that accuracy can be misleading for imbalanced datasets, where one class is much rarer than the other.
True and false positives and negatives in detail
True positives are actual positive examples that the model correctly identifies as positive. False positives are actual negative examples that the model incorrectly labels as positive. False negatives are actual positive examples that the model misses and labels as negative. True negatives are actual negative examples that the model correctly labels as negative. These four categories are the building blocks for all classification metrics.
The meaning of each outcome depends on the context. In a medical test, a false negative might mean a disease is missed, while a false positive might cause unnecessary worry. The cost of each type of error can differ, which is why the confusion matrix is useful for evaluating models beyond simple accuracy.
The scikit-learn model evaluation documentation explains that many scoring functions for classification are derived from the confusion matrix. It also discusses how the choice of a scoring function should align with the main goal of the prediction task, whether that is ranking, probability estimation, or decision making.
How the classification threshold changes the matrix
A classification model often outputs a probability score, and a threshold converts that score into a class label. For example, with a threshold of 0.5, any score above 0.5 is predicted as positive, and any score below is predicted as negative. Changing the threshold changes the number of true and false positives and negatives.
If the threshold is lowered, more examples are predicted as positive, which tends to increase true positives and false positives while decreasing false negatives and true negatives. Raising the threshold has the opposite effect. The optimal threshold depends on the relative costs of false positives and false negatives, as well as the class balance in the data.
The Google classification metrics page notes that all metrics calculated from the confusion matrix are computed at a single fixed threshold and change when the threshold changes. Users often tune the threshold to optimize a specific metric, such as recall or precision.
Metrics derived from the matrix
Accuracy, precision, recall, and the false positive rate are all derived from the four counts in the confusion matrix. Accuracy is the proportion of all correct predictions. Precision is the proportion of predicted positives that are actually positive, calculated as true positives divided by all predicted positives. Recall, also called the true positive rate, is the proportion of actual positives that are correctly identified, calculated as true positives divided by all actual positives.
The false positive rate is the proportion of actual negatives that are incorrectly classified as positive, calculated as false positives divided by all actual negatives. These metrics help evaluate different aspects of model performance. For imbalanced datasets, precision and recall are often more informative than accuracy, as explained in Google's classification metrics guide.
The Microsoft AutoML evaluation documentation recommends using the confusion matrix and per-class precision and recall when dealing with imbalanced classification, because accuracy and micro averages can hide failures in the minority class. It also suggests using threshold-independent metrics like ROC AUC for ranking, and threshold-specific metrics like precision and recall for evaluating predictions at a selected threshold.
| Outcome | Predicted class | Actual class | Meaning |
|---|---|---|---|
| True positive (TP) | Positive | Positive | Correctly identified positive |
| False positive (FP) | Positive | Negative | Incorrectly identified as positive |
| False negative (FN) | Negative | Positive | Missed positive |
| True negative (TN) | Negative | Negative | Correctly identified negative |
Frequently asked questions
What is a true positive?
A true positive is an actual positive example that the model correctly classifies as positive. For example, a spam email that is correctly identified as spam.
What is a false positive?
A false positive is an actual negative example that the model incorrectly classifies as positive. For example, a legitimate email that is mistakenly filtered as spam.
What is a false negative?
A false negative is an actual positive example that the model incorrectly classifies as negative. For example, a spam email that is not caught and ends up in the inbox.
What is a true negative?
A true negative is an actual negative example that the model correctly classifies as negative. For example, a legitimate email that is correctly delivered to the inbox.
How does the classification threshold affect the confusion matrix?
The threshold determines which probability scores are labeled as positive. Lowering the threshold increases true positives and false positives, while raising it increases true negatives and false negatives. The optimal threshold depends on the costs of different errors.