Visualize gradient boosting step by step. See how each new decision tree fits residual errors and improves the combined prediction.
1A team that fixes each other's mistakes
A forest's trees work side by side. Boosting's trees work in a line: each one looks at what the team still gets wrong and fixes a bit of it.
2Fit the leftovers, not the data
Start with a flat guess. The gaps between the guess and the points (residuals, in red) are what's still wrong. The next tree is trained on those gaps.
3Small steps beat big jumps
Each tree is added at only a fraction of its size (the learning rate). Small steps need more trees but follow the real pattern instead of the noise.
Watch it learn
Boost a model, one small tree at a time
The task: draw a curve through 40 noisy points. The model starts as a flat line at the average. Each new small tree is trained on what's still wrong (the residuals in the lower plot) and only a fraction of it, the learning rate, is added. The first three trees are shown in slow motion.
The model after 0 trees
What's still wrong (residuals)
- Trees so far
- 0
- Training error (MSE)
- 2.604
- Error on 400 new points
- 2.808
- One deep tree, new points
- 0.429
MSE is the mean squared error: the average of (true − predicted)². Lower is better. The new points come from the same curve, with the same amount of noise, so even a perfect model can't reach 0.
Press Step to see the first tree in three parts: residuals, fit, add.
The big question
When to stop adding trees
Training error only ever goes down as trees are added. Error on new points goes down at first, reaches its lowest point, then creeps back up as the trees start fitting the noise. That turning point is the number of trees to keep. With a learning rate of 1 and deep trees, it's reached almost immediately.
In practice you hold back some data and stop when its error stops improving (“early stopping”); scikit-learn's histogram-based boosting does this by default once there are more than 10,000 samples.
No magic, just averages
The maths, step by step
One round of boosting on five points, (1, 1), (2, 3), (3, 2), (4, 5), (5, 4), with a one-question tree (a stump) and a learning rate of 0.5. Every number is calculated by the same code as the plots above.
- 1
Start with the average
Before any trees, the best single number to predict everywhere is the mean of y.F₀ = (1 + 3 + 2 + 5 + 4) ÷ 5 = 3.0000
error (MSE) = mean of (y − 3)² = (4 + 0 + 1 + 4 + 1) ÷ 5 = 2.0000
- 2
Residuals: what's still wrong
For each point, subtract the prediction from the true value. Positive means the model is too low there, negative means too high.r = y − F₀ = (-2, 0, -1, 2, 1)
Why “gradient”? With the loss ½(y − F)², the slope with respect to F is −(y − F). So the residual is exactly the negative gradient: the downhill direction for each prediction. Each tree is a step of gradient descent.
- 3
Fit a small tree to the residuals
Try every cut between neighbouring x values and keep the one whose two sides are most uniform (lowest squared error). Here that's x ≤ 3.5. Each side predicts the mean of its residuals.left (x ≤ 3.5) = (-2 + 0 + -1) ÷ 3 = -1.0000
right (x > 3.5) = (2 + 1) ÷ 2 = 1.5000
- 4
Add a shrunken copy of the tree
Don't add the whole tree: multiply it by the learning rate (0.5) first. That's the “shrinkage” that stops each tree from overcorrecting.F₁ for x ≤ 3.5 = 3 + 0.5 × (-1) = 2.5000
F₁ for x > 3.5 = 3 + 0.5 × 1.5 = 3.7500
- 5
The error went down
Measure again. The new residuals are smaller, and the next tree would be fitted to these.new residuals = (-1.5, 0.5, -0.5, 1.25, 0.25)
error (MSE) = mean of new residuals² = 0.8750
From 2 to 0.875 with one tree. Repeat this hundreds of times and you have gradient boosting. With squared error and a learning rate between 0 and 1, the training error can never go up from one tree to the next.
The family
AdaBoost, and the tools people use
AdaBoost: the original boosting
Instead of fitting residuals, AdaBoost changes how much each training point matters. Every point starts with the same weight (1/N). After each weak learner (usually a tiny tree), the points it got wrong get more weight and the ones it got right get less, so the next learner concentrates on the hard cases. The final answer is a weighted vote of all the learners.
Red points were misclassified, so they grow (more weight) each round.
What people use in practice
- scikit-learn's HistGradientBoosting: groups values into bins (typically 256), handles missing values itself, and can be orders of magnitude faster than the classic version once there are tens of thousands of rows.
- XGBoost, LightGBM, CatBoost: popular libraries built on the same idea, adding each tree to fix what the previous ones got wrong.
- Subsampling: training each tree on a random half of the rows (scikit-learn's typical subsample of 0.5) is called stochastic gradient boosting.
Random forest vs gradient boosting
| Random forest | Gradient boosting | |
|---|---|---|
| How trees are built | Side by side, independently | One after another, each fixing the last |
| Each tree | Deep, on a random sample | Small and shallow, on the residuals |
| Combined by | Averaging | Adding, each scaled by the learning rate |
| More trees | Doesn't overfit; just slower | Can overfit: stop at the best number |
| Tuning | Works well out of the box | Needs learning rate, depth and tree count |
| Parallel training | Yes | No (trees depend on each other) |
How it works: Gradient Boosting
Step by step
- The model so far
- Residuals (what's still wrong)
- The new tree
Press Start or Step: the step the animation is on lights up below.
- 1
Start from the average
The first model predicts the mean of y everywhere.
F₀(x) = mean(y) - 2
Measure what's still wrong
For every point, the residual is the true value minus the current prediction.
rᵢ = yᵢ − F(xᵢ) - 3
Fit a small tree to the residuals
Not to y: to the errors. The tree learns where the model is too low or too high.
h = small_tree(x, r) - 4
Add a little of it
Move the model towards the fix, scaled down by the learning rate.
F ← F + lr · h - 5
Stop at the right number of trees
Keep the number of trees where the error on new data was lowest; past that it starts fitting noise.
pick n_trees by validation error
A random forest grows many trees side by side and averages them. Boosting grows them one after another, and each new tree has one job: fix what the trees so far still get wrong.
It starts with a very simple guess, the average. Then it measures the leftover errors (the residuals: true value minus prediction), fits a small tree to those errors, and adds a little of that tree to the model. Repeat, and the model creeps towards the data, one small correction at a time.
“Gradient” comes from gradient descent: for squared error, the residual is exactly the downhill direction for each prediction, so each tree is a step downhill. The learning rate shrinks every step, which needs more trees but usually generalises better.
Where you'll meet it
- Tabular data (spreadsheets, databases): scikit-learn calls it “an excellent model” for exactly this. XGBoost, LightGBM, CatBoost and scikit-learn's HistGradientBoosting are the popular tools.
- Regression and classification alike; only the loss, and so the “residual” each tree fits, changes.
- When you can spend a little time tuning the learning rate, tree depth and number of trees.
Each tree is small, but they have to be grown one after another, because each depends on the errors left by all the ones before it.