Overfitting and underfitting: memorizing noise or missing the pattern
Overfitting occurs when a model memorizes training data noise and fails on new data, while underfitting happens when a model is too simple to capture the underlying pattern. Both problems stem from an imbalance between bias and variance. Detection relies on comparing training and validation performance, and prevention includes regularization, cross-validation, and adjusting model complexity.
Overfitting and underfitting defined
Overfitting occurs when a machine learning model fits too closely to its training data, learning irrelevant noise instead of the underlying pattern. IBM's definition of overfitting states that the model becomes unable to make accurate predictions from any data other than the training data, which defeats the purpose of generalization IBM's definition of overfitting.
Underfitting is the opposite error. IBM's explanation of underfitting describes a data model that cannot capture the relationship between input and output variables, producing high error rates on both the training set and unseen data IBM's explanation of underfitting.
The two conditions illustrate the bias-variance tradeoff. An underfitted model has high bias and low variance, while an overfitted model has low bias and high variance, as AWS describes in its overview of overfitting AWS's overview of overfitting.
Detecting overfitting
A common diagnostic is the generalization curve, which plots training and validation loss against training iterations. Google's crash course on overfitting explains that when the training loss continues to decline while the validation loss rises, the model is likely overfitting Google's crash course on overfitting.
Loss curves can also reveal other issues. Google's guide to interpreting loss curves notes that an oscillating loss curve might be improved by checking data against a schema, reducing the learning rate, or reducing the training set to a small number of trustworthy examples Google's guide to loss curves.
Comparing training and test accuracy is another approach. Microsoft Learn's guidance on overfitting prevention states that a large gap between the two indicates a problem, as with Model C, which has 99.9 percent training accuracy but only 45 percent test accuracy Microsoft's automated ML guidance.
Causes and prevention
Model complexity is a key factor. scikit-learn's example of polynomial regression demonstrates that a degree 1 polynomial underfits the data, while a high degree polynomial, such as degree 15, overfits by learning the noise scikit-learn's underfitting vs overfitting example.
Prevention strategies include early stopping, which halts training before the model begins to memorize noise, and regularization, which penalizes complex models. Data augmentation and feature selection also help, but each approach must be balanced against the risk of underfitting.
The goal is to find a model complexity that achieves low bias and low variance. This balance allows the model to generalize well to new data, which is the primary purpose of machine learning.
Finding the balance
The bias-variance tradeoff is central to model fitting. As a model learns, bias decreases but variance can increase, and the ideal model lies between underfitting and overfitting.
To achieve good generalization, practitioners should regularly evaluate the model on a validation set or use cross-validation to adjust complexity and training duration.
| Aspect | Overfitting | Underfitting |
|---|---|---|
| Training error | Low | High |
| Test error | High | High |
| Bias | Low | High |
| Variance | High | Low |
| Typical cause | Too complex or too much training | Too simple or too little training |
Frequently asked questions
What is overfitting in machine learning?
Overfitting happens when a model learns the training data too well, including noise, and fails to generalize to new data.
How can overfitting be detected?
Overfitting is detected by comparing training and validation loss curves, or by using cross-validation to measure performance on held-out data.
What is the bias-variance tradeoff?
The bias-variance tradeoff describes the tension between a model's error due to simplicity (bias) and error due to sensitivity to training data (variance).
How can underfitting be avoided?
Underfitting can be reduced by increasing model complexity, such as adding features or using a more flexible algorithm, and by training for a longer duration.