The bias-variance tradeoff: balancing simple and complex models
The bias-variance tradeoff describes how model complexity affects prediction error. Simple models tend to underfit and have high bias, while complex models overfit and have high variance. The goal is to find a middle ground that minimizes total error.
Bias and variance as sources of error
Bias measures how far a model's predictions are, on average, from the true values. Variance measures how much predictions change when the model is trained on different datasets. The Amazon MLU Explain bias-variance guide explains that a high-bias model is too simple to capture the underlying pattern, leading to underfitting, while a high-variance model is too sensitive to noise, leading to overfitting.
The total prediction error can be decomposed into bias squared, variance, and an irreducible noise term. The Cornell lecture note on the bias-variance tradeoff derives this decomposition mathematically. Noise is inherent to the data and cannot be reduced by changing the model, but bias and variance can be adjusted.
Underfitting occurs when the model is too simple to account for the training data, so both training and test errors are high. Overfitting occurs when the model memorizes patterns that exist only in the training set, so training error is low but test error is high. These two failure modes are the extremes of the bias-variance tradeoff.
Model complexity and the tradeoff curve
As model complexity increases, bias tends to decrease while variance tends to increase. The total test error therefore follows a U-shaped curve: it starts high due to bias, drops to a minimum at a moderate complexity, and then rises again due to variance. This shape is illustrated by the IBM bias-variance tradeoff explainer with polynomial fits of degree 1, 4, and 25.
In IBM's example, a degree 1 polynomial has high bias and an MSE of 0.2929, a degree 4 polynomial reaches a lower MSE of 0.0714, and a degree 25 polynomial overfits the training data with an MSE of 0.059 but generalizes poorly. The overly complex model reacts strongly to noise and would change significantly with a new sample of data.
The tradeoff is not about eliminating either bias or variance but about finding the complexity that minimizes total error. For a given dataset, the best model typically lies between the simple and complex extremes.
Finding the balance with validation techniques
Practitioners use validation curves to diagnose whether a model is underfitting or overfitting. The scikit-learn user guide on validation curves explains that if both training and validation scores are low, the estimator is underfitting; if the training score is high and validation score is low, it is overfitting. Plotting validation curves for a hyperparameter, such as a regularization strength, can reveal a good operating point.
Learning curves show how training and validation scores change with the number of training samples. The scikit-learn guide notes that if adding more data does not improve the validation score, the model likely suffers from bias; if the training score is much higher than the validation score, more data should reduce variance.
Cross-validation is a common method for estimating the validation error. The Cornell lecture note on model selection describes k-fold cross-validation, where the training data is split into k partitions and each is used once for validation. A telescopic search over a regularization parameter can then find a suitable level of complexity.
When larger models seem to defy the tradeoff
A phenomenon called double descent appears to contradict the classical U-shaped curve. The Amazon MLU Explain double descent article shows that test error can rise after the traditional minimum but then fall again as the model becomes extremely overparameterized. In this interpolation regime, models that pass exactly through every training point can achieve very low test error.
Despite this apparent contradiction, the article concludes that double descent reinforces the bias-variance tradeoff. The overparameterized region behaves differently from the classical region, but bias and variance still help explain the error structure. Understanding when double descent occurs depends on the dataset and the model architecture.
| Observation | Likely cause | Suggested action |
|---|---|---|
| Training and validation scores both low | Underfitting | Increase model complexity or add features |
| Training score high, validation score low | Overfitting | Reduce complexity, add regularization, or collect more data |
| Training and validation scores both high | Appropriate fit | Proceed with model |
Frequently asked questions
What is the bias-variance tradeoff?
The bias-variance tradeoff is the balance between a model's systematic error (bias) and its sensitivity to training data (variance). Choosing model complexity affects both; simple models have high bias, complex models have high variance, and the total error is minimized at a middle ground.
How can one know if a model is underfitting or overfitting?
If both training and validation scores are low, the model is underfitting. If training score is high but validation score is low, it is overfitting. Use validation or learning curves to check.
What is double descent?
Double descent is a phenomenon where test error rises with complexity past a point, then falls again as the model becomes extremely overparameterized. It appears in some deep learning settings and is discussed in Amazon MLU's explainer.
How can cross-validation help with the tradeoff?
Cross-validation estimates test error by training on subsets of the data. It helps tune hyperparameters like regularization strength to find a model complexity that balances bias and variance.
Why is the tradeoff called a tradeoff?
Because reducing bias usually increases variance and vice versa. The goal is to choose a point where the sum of bias squared and variance (plus noise) is lowest.