Explore random forests through interactive decision trees, bootstrap samples and out-of-bag evaluation. See how individual trees vote on a prediction.
1Ask a crowd, not one person
One tree makes jumpy mistakes. Many different trees make different mistakes, and averaging their answers cancels much of them out.
2Each tree gets a random sample
Pick rows at random, putting each one back after picking (a bootstrap sample). Some rows come up twice and about a third are never picked: those are that tree's “out-of-bag” rows.
3Each question looks at only some features
At every split, a tree may only choose from a random few of the features. That stops every tree from asking the same first question, so the trees really are different.
Watch it learn
Grow a forest, one tree at a time
Each tree below learned from its own random sample of the points (filled dots; hollow ones were left out) and could only ask about one randomly chosen feature at each question. On their own they're jumpy. The big plot shows the forest: the average of every tree so far. Press Start and watch it settle.
The forest: average of 1 tree
Stronger colour means more of the trees agree.
Four of the trees, each on its own
- One full tree, new points
- 66.3%
- Forest (1), new points
- 59.3%
- Out-of-bag estimate
- 62.5%
- Rows with an estimate
- 32 / 80
“New points” are 400 fresh points from the same shape that no tree ever saw. The out-of-bag estimate uses only the training points, each judged by the trees that left it out.
Two groups with some points given the wrong label. A deep tree memorises the noise; a shallow one ignores it.
The forest starts as a single tree. Press Start to add the other 49 one at a time.
More trees, better?
How many trees?
Accuracy on new points for a forest of 1 to 50 trees. It climbs quickly, then levels off. Adding trees doesn't make a forest overfit (Breiman: “Random forests does not overfit. You can run as many trees as you want.”); after the curve flattens, more trees just cost time. The dashed line is a single full tree.
No magic, just probability
The maths, step by step
Four small steps explain why a forest works. Every number is calculated by the same code that grows the forest above.
- 1
Why about a third of the rows are left out
A bootstrap sample makes n picks from n rows, putting each row back after it's picked. On any single pick, one particular row is missed with chance (1 − 1/n). To be left out completely it has to be missed on all n picks.chance a row is never picked = (1 − 1/n)ⁿ
n = 10 = (1 − 1/10)¹⁰ = 0.9¹⁰ = 0.3487
n = 80 (this page) = (1 − 1/80)⁸⁰ = 0.3656
n = 1,000,000 = (1 − 1/1,000,000)^1,000,000 = 0.3679
As n grows, this gets closer and closer to 1/e = 0.3679: about 37%, or roughly a third. Those left-out rows are each tree's out-of-bag rows.
- 2
Why many okay voters beat one
Imagine 25 trees that are each right 70% of the time, and suppose (for now) that their mistakes are completely independent. The forest is wrong only if 13 or more of them are wrong at once, which is unlikely.P(majority right) = Σ over k = 13…25 of C(25, k) · 0.7ᵏ · 0.3²⁵⁻ᵏ
1 tree = right 70% of the time = 0.7000
5 trees = at least 3 of 5 right = 0.8369
25 trees = at least 13 of 25 right = 0.9825
From 70% to about 98%, just by voting. That's the “wisdom of the crowd”.
- 3
The catch: the trees aren't independent
Step 2 assumed the trees make different mistakes. Trees grown on the same data tend to make the same ones, and if they all agree on a wrong answer, voting can't fix it. Real forests gain much less than step 2 suggests.That's exactly why random forests add randomness on purpose:
· each tree gets a different bootstrap sample of the rows
· each question may only look at a random subset of the features
Both make the trees disagree more, so their errors overlap less and the average improves.
- 4
How the votes are combined
scikit-learn doesn't count one vote per tree. It averages each tree's probabilities (the class shares in the leaf the point lands in), then picks the class with the biggest average. Here are three trees' answers for one point:tree 1: teal 0.9, violet 0.1
tree 2: teal 0.6, violet 0.4
tree 3: teal 0.2, violet 0.8
average teal = (0.9 + 0.6 + 0.2) ÷ 3 = 0.5667
average violet = (0.1 + 0.4 + 0.8) ÷ 3 = 0.4333
Teal wins, 0.567 to 0.433. A plain vote would also say teal (2 trees to 1), but averaging keeps track of how sure each tree was.
In practice
Pros and cons
Good at
- Much steadier and usually more accurate on new data than one tree: averaging reduces the variance.
- Works well out of the box, with little tuning.
- A free accuracy estimate from the out-of-bag rows.
- Trees are independent, so they can be grown in parallel.
Watch out for
- Hard to read: you can't follow 50 trees by hand the way you can follow one.
- Slower and bigger than one tree, both to train and to use.
- Can be slightly more biased than one tree (scikit-learn notes this trade for much less variance).
- Built-in feature importances favour features with many distinct values; scikit-learn suggests permutation importance instead.
How it works: Random Forest
Step by step
- Class: teal
- Class: violet
- Left out of a tree's sample
Press Start or Step: the step the animation is on lights up below.
- 1
Grow one tree on a bootstrap sample
Draw n rows with replacement. Grow a full tree on them, looking at only a random subset of features at each question.
sample = random rows, with replacement - 2
Add more trees, each different
Every new tree gets its own random sample and its own random questions. The forest's answer is the average of all their probabilities.
p(x) = mean over trees of tree.probs(x) - 3
Predict with the average, check with out-of-bag
The class with the highest average probability wins. Each row can be tested by the trees that never saw it.
predict = argmax p(x)
One decision tree is jumpy: change the data a little and you get a very different tree. A random forest grows many trees and averages their answers. Their individual mistakes point in different directions, so averaging cancels much of them out, like asking a crowd instead of one person.
Averaging only helps if the trees disagree. Two tricks make them different. Each tree learns from a bootstrap sample: rows picked at random with replacement, so some rows appear twice and about a third are left out. And at every question, each tree may only look at a random few of the features.
The left-out rows are a free test. Each tree never saw about a third of the data, so you can check the forest on those rows (the out-of-bag estimate) without keeping a separate test set.
Where you'll meet it
- Tabular data (spreadsheets, databases) where you want strong accuracy with very little tuning.
- When one decision tree overfits or keeps changing with the data.
- As a strong baseline. Gradient boosting is the other top choice for tabular data; it needs more tuning.
Each tree is grown exactly like a decision tree, just on a resampled copy of the data, and the trees are independent, so they can be grown in parallel.