Gradient boosting: a strong model built from many weak trees
Gradient boosting is an ensemble method that builds a strong model by sequentially adding weak decision trees, each trained to correct the errors of the current ensemble. The process minimizes a loss function and uses shrinkage to control learning speed. Popular implementations such as XGBoost and LightGBM add efficiency techniques for large datasets.
What gradient boosting is
Gradient boosting is an ensemble learning algorithm that combines multiple weak prediction models, typically decision trees, into a single strong model. It belongs to the boosting family of methods that train models sequentially to correct previous mistakes. IBM's gradient boosting overview
In contrast to bagging, which trains models in parallel on different data subsets, boosting trains weak learners one after another. The goal is to reduce bias by focusing on samples that previous models misclassified. AWS's explanation of boosting
A weak learner has low accuracy, often only slightly better than random guessing. Gradient boosting converts these weak learners into a strong learner by combining their predictions.
How gradient boosting iteratively improves
In gradient boosting, each iteration trains a new weak model to predict the error of the current strong model, then subtracts that prediction to reduce the error. Shrinkage scales each weak model's contribution, limiting learning speed and mitigating overfitting. Google's gradient boosted decision trees
This procedure can be viewed as gradient descent in function space: the weak learners are regression trees fitted to the negative gradient of the loss function with respect to the current prediction. A small fixed step size is used. Cornell's boosting lecture notes
Objective function and regularization
The objective function in gradient boosting combines a training loss with a regularization term. The training loss measures predictive accuracy, and the regularization term penalizes model complexity to avoid overfitting. XGBoost's introduction to boosted trees
For regression, mean squared error is common; for classification, logistic loss is often used. The tradeoff between simplicity and predictive power is known as the bias-variance tradeoff.
Efficient implementations
XGBoost is a widely used implementation that introduces a sparsity-aware algorithm and a weighted quantile sketch for approximate tree learning. It scales beyond billions of examples with fewer resources than earlier systems. arXiv paper on XGBoost
LightGBM, another popular implementation, uses Gradient-based One-Side Sampling (GOSS) and Exclusive Feature Bundling (EFB) to speed up training. These techniques reduce data instances and feature dimensions, achieving training speeds over 20 times faster than conventional GBDT while maintaining accuracy. NeurIPS paper on LightGBM
These implementations make gradient boosting practical for large, high-dimensional datasets. They are available in Python libraries and are used for both regression and classification tasks.
| Method | Training approach | Key characteristic |
|---|---|---|
| Bagging | Trains models in parallel on random data subsets | Reduces variance by averaging predictions |
| AdaBoost | Trains sequentially, increases weights on misclassified samples | Reduces bias by focusing on difficult points |
| Gradient boosting | Trains sequentially, fits trees to residual errors | Minimizes a loss function with gradient descent |
Frequently asked questions
What is a weak learner in gradient boosting?
A weak learner is a simple model, often a decision tree with few leaves, that performs only slightly better than random guessing. Gradient boosting combines many weak learners to create a strong model.
How does gradient boosting prevent overfitting?
It uses shrinkage to limit each tree's contribution, regularization to penalize complexity, and early stopping based on validation performance.
What is the difference between gradient boosting and AdaBoost?
AdaBoost adjusts weights of misclassified samples, while gradient boosting fits new trees to the residual errors of the current ensemble to minimize a loss function.
What are some popular implementations of gradient boosting?
XGBoost and LightGBM are widely used. XGBoost uses sparsity-aware learning, and LightGBM uses GOSS and EFB for faster training.