Regularization: L1, L2 and Other Ways to Keep Models from Overfitting

Updated October 21, 2026

Regularization reduces overfitting by adding a penalty for model complexity, trading a bit of training accuracy for better generalization. L1 and L2 penalties shrink coefficients, while early stopping and tree-specific parameters also control complexity.

What regularization does

Overfitting occurs when a model fits training data too closely and fails on new data. Regularization is a set of methods to reduce overfitting by trading a small increase in training error for a larger improvement in test accuracy. As the IBM regularization overview explains, regularization increases generalizability at the cost of higher training error.

The bias-variance tradeoff describes this behavior. Bias measures average error on training data, while variance measures error on unseen data. Regularization decreases variance by increasing bias, a concession that the same IBM overview describes in detail.

Simpler models often generalize better. The preference for simplicity goes back to Occam's razor, a principle that favors simple explanations. The Google model complexity unit notes that complex models typically outperform simple ones on training data but lose on test data.

L2 regularization

L2 regularization adds a penalty based on the sum of squared weights. The formula is the sum of squares of all model weights. The Google L2 regularization page gives this formula and shows that large weights contribute disproportionately to the penalty.

In practice, L2 encourages weights toward zero but never exactly to zero. A regularization rate lambda controls the strength. High lambda leads to simpler models; low lambda risks overfitting. The same Google page also describes weight histograms for different lambda values.

The scikit-learn documentation for Ridge regression shows the penalty added to the residual sum of squares, with alpha controlling shrinkage. The scikit-learn linear models guide states that larger alpha values increase shrinkage and make coefficients less sensitive to collinearity.

L1 regularization

L1 regularization, also known as Lasso, uses the sum of absolute weights as a penalty. The Google model complexity unit lists L1 and L2 as two common ways to quantify model complexity.

Unlike L2, L1 can drive some weights exactly to zero, which effectively removes those features. This makes L1 useful for creating sparse models and performing feature selection, though the detailed mechanics are described in the same complexity unit as an alternative to L2.

Alternatives and additional controls

Early stopping is a regularization method that ends training before the model fully converges. The Google logistic regression loss and regularization page notes that logistic regression models, because of their asymptotic nature, often rely on L2 regularization or early stopping to prevent overfitting.

Gradient boosted trees, unlike random forests, can also overfit. The Google decision forests overfitting guide lists common regularization parameters for these models, including maximum tree depth, shrinkage rate, ratio of attributes tested at each node, and L1/L2 coefficients on the loss.

Frequently asked questions

What is overfitting?

Overfitting is when a model performs well on training data but poorly on new data. Regularization helps prevent it by penalizing complexity.

What is the difference between L1 and L2 regularization?

L2 adds squared weights and shrinks them toward zero but never to zero. L1 adds absolute weights and can set some weights exactly to zero, providing feature selection.

How is the regularization rate chosen?

The regularization rate lambda is tuned using validation data. Higher lambda gives simpler models, while lower lambda risks overfitting. The optimal value depends on the dataset.

What is early stopping?

Early stopping ends training before the model fully converges, often when validation loss starts to increase. It is a quick, although rarely optimal, form of regularization.

Do tree-based models need regularization?

Yes, gradient boosted trees can overfit. Regularization parameters include tree depth, shrinkage, and attribute test ratio.