AI glossary · Basic · Classic ML, recsys & search
Gradient boosting
Gradient boosting is a technique that builds a strong prediction model by adding small decision trees one after another, training each new tree to make up for what the previous trees got wrong. XGBoost, LightGBM and CatBoost are well-known implementations.
About this term
- Level
- Basic · Technical terms you often meet in AI news
- Category
- Data, ops & platforms
- Added
- 2026-10-08
How it works
It starts with a very simple prediction (for example, the overall average). It computes the difference between the actual values and the prediction, that is, the error, and the next tree is trained to predict this error rather than the original answer. The new tree's prediction is added to the existing prediction not all at once but a little at a time (by the learning rate), reducing the error. Repeating this hundreds of times combines individually weak trees into an accurate model. The 'gradient' in the name means following the direction that reduces the loss function fastest; it applies the same idea as gradient descent to stacking trees.
Stacking trees to make up for errors
Until the error is small enough
- 1Current prediction
- 2Compute error
- 3Train new tree
- 4Add a little at a time
- 1. Current prediction Starts with a simple value such as the average
- 2. Compute error Difference from actual values
- 3. Train new tree To predict the error
- 4. Add a little at a time Applied by the learning rate
Why it matters
On tabular data organized in rows and columns, such as customer information, transaction records and sensor readings, it still often predicts as well as or better than deep learning. That is why it is a go-to tool in data analysis competitions and in finance and e-commerce. Training and prediction are relatively fast, and it can show the importance of each item (feature) in predictions, which makes results easier to explain. Being able to run on ordinary servers without large GPUs is also a big practical advantage.
Things to know
Stacking too many trees or making them too deep causes overfitting, where the model memorizes even the noise in the training data. So 'early stopping', which halts training when performance on held-out validation data stops improving, is commonly used. There are many settings to tune, such as the number of trees, depth and learning rate, so it takes a fair amount of effort. Deep learning generally does better on unstructured data such as images, text and audio, and because trees must be built in sequence, parallel processing is harder than with random forests, which build many trees at once.
Example
An e-commerce company feeds tabular data such as customers' visit counts, most recent purchase dates and shopping cart history into XGBoost or LightGBM to predict 'customers likely to churn next month', and sends coupons to high-risk customers. Gradient boosting models are also widely used for bank loan scoring, ad click prediction and demand forecasting, with feature importance reported alongside to explain why a decision was made.