Move a threshold and watch a confusion matrix fill. Explore precision, recall, F1, ROC curves, AUC and the accuracy trap on imbalanced data.
1A score, then a cut
A classifier doesn't say yes or no. It gives each email a score. A threshold turns scores into decisions: everything to the right is flagged as spam.
2Four kinds of answer
Compare each decision with the truth. Right answers are true positives and true negatives. The two mistakes are different: a false positive hides a real email; a false negative lets spam through.
3A trade-off, not one number
Raise the threshold and the filter only flags what it's sure about: precise, but it misses spam. Lower it and it catches more spam, with more false alarms. You choose the point.
Watch it get judged
Move the threshold, fill the table
A spam filter has scored 20 emails (made up for this page, so you can count them). Each dot is one email: spam on the top row, normal email below. Everything right of the dashed line gets flagged. Press Step to judge the emails one by one and watch each land in its cell, or drag the threshold and see every number change at once.
Tip: click anywhere on the strip to put the threshold there. Hover a dot to see the email.
At 0.50, the filter flags 9 emails: 6 spam and 3 normal. Press Step to judge them one at a time.
Confusion matrix
all 20 emails
- Precision · When it flags, is it right?
- 66.7%TP ÷ (TP + FP)
- Recall · How much spam does it catch?
- 75.0%TP ÷ (TP + FN)
- F1 score · Both at once
- 70.6%2·P·R ÷ (P + R)
- Accuracy · How many right overall?
- 75.0%(TP + TN) ÷ all
Hover a metric to see which cells it uses.
The big trade-off
Precision vs recall
One threshold, two numbers that pull in opposite directions. The chart shows both at every threshold from 0 to 1; the amber line is yours (click the chart to move it). There is no setting that wins both: you choose which mistake you'd rather make.
- Precision
- 66.7%
- Recall
- 75.0%
- F1
- 70.6%
- High threshold: only the surest emails are flagged. Few false alarms (high precision), but spam slips through (low recall).
- Low threshold: nearly everything is flagged. All the spam is caught (high recall), along with real email (low precision).
- Past the highest score nothing is flagged, so precision has no value and its line stops.
Every threshold at once
The ROC curve and AUC
Slide the threshold from 1 down to 0 and plot where you are each time: across, the share of normal email wrongly flagged; up, the share of spam caught. Each spam email you pass steps the curve up; each normal email steps it right. A good filter climbs first and hugs the top-left corner. The area under the curve (AUC) sums it up in one number.
Mostly right, with a few normal emails that look spammy and a few spam emails that slip under.
- AUC
- 0.854
- Recall (up)
- 75.0%
- False positive rate
- 25.0%
The amber dot is your threshold. AUC is also the chance that a random spam email scores higher than a random normal one: here 82 of the 96 spam–normal pairs.
- Bottom-left corner: threshold above every score, nothing flagged.
- Top-right corner: threshold at 0, everything flagged.
- A perfect filter goes straight up, then across: AUC 1. The diagonal is guessing: AUC 0.5.
When accuracy lies
The accuracy trap: a rare condition
1,000 people are screened for a condition that 10 of them have. Each square below is one person. Compare a “model” that simply says “healthy” to everyone with a real test, and look at which one has the better accuracy.
- Accuracy
- 99.0%
- Recall: sick people found
- 0.0%
- Precision: flagged people who are sick
- – (no one flagged)
A “model” that never flags anyone. It is right about all 990 healthy people and misses every sick one.
99% accuracy, and every sick person is sent home. When one class is rare, accuracy mostly measures how rare it is.
No magic, just counting
The maths, step by step
The same 20 emails at threshold 0.5. Every metric is a fraction of the four counts, so once you have the table, the rest is division. Every number below comes from the same code as the plots above.
- 1
Count the four cells
Sort every email by two questions: was it flagged, and is it really spam?TP = 6 (spam, flagged) · FN = 2 (spam, missed)
FP = 3 (normal, flagged) · TN = 9 (normal, left alone)
6 + 2 = 8 spam, 3 + 9 = 12 normal: all 20 emails, each counted once.
- 2
Precision: trust in a flag
Look only at the flagged column. What share of it is really spam?precision = TP ÷ (TP + FP) = 6 ÷ (6 + 3) = 0.6667
So when the filter flags an email, it's right about 67% of the time. False positives pull it down.
- 3
Recall: how much spam is caught
Look only at the spam row. What share of it was flagged?recall = TP ÷ (TP + FN) = 6 ÷ (6 + 2) = 0.7500
It catches 75% of the spam. False negatives pull it down. Recall is also called sensitivity, or the true positive rate.
- 4
F1: why not just average them?
F1 is the harmonic mean, which stays low unless both numbers are high.F1 = 2 · P · R ÷ (P + R) = 2 · 0.6667 · 0.7500 ÷ (0.6667 + 0.7500) = 0.7059
Now a filter that flags just one email, correctly: precision 1, recall 0.05. The plain average is 0.525, which sounds fine. F1 is 0.095, which says what's really going on: it catches almost nothing.
- 5
Accuracy and specificity
Accuracy uses all four cells. Specificity is recall for the normal class.accuracy = (TP + TN) ÷ all = (6 + 9) ÷ 20 = 0.7500
specificity = TN ÷ (TN + FP) = 9 ÷ (9 + 3) = 0.7500
false positive rate = FP ÷ (FP + TN) = 3 ÷ (3 + 9) = 1 − specificity = 0.2500
- 6
AUC: one number for every threshold
Pair every spam email with every normal email. How often does the spam score higher?8 spam × 12 normal = 96 pairs. For example, “You won a free cruise” (0.95) outscores all 12 normal emails.
AUC = pairs the spam wins ÷ all pairs = 82 ÷ 96 = 0.8542
The area under the ROC curve, measured with trapezoids, gives the same 0.8542. 1.0 means every spam email outscores every normal one; 0.5 is a coin flip. AUC never looks at a threshold, so it judges the ranking, not the cut.
In practice
Which metric should I use?
Start from the question: which mistake costs more?
Spam filter → precision
A real email lost in the spam folder costs more than one spam email reaching the inbox. Keep false positives rare.
Cancer screening → recall
Missing a sick person is far worse than a false alarm that a second test will clear. Keep false negatives rare.
Fraud, rare events → precision, recall and F1
Accuracy is meaningless when positives are rare. Report precision and recall (or the precision–recall curve), and F1 when you need one number.
Comparing models → AUC
AUC judges how well the scores rank positives above negatives, before any threshold is chosen. Pick the threshold afterwards.
Balanced classes, equal costs → accuracy
When both classes are common and both mistakes cost the same, accuracy is a fair, easy-to-read summary.
Predicting numbers → MSE, MAE, R²
Regression has no confusion matrix. See the Linear Regression page for mean squared error, mean absolute error and R². Open Linear Regression →
How it works: Evaluation Metrics
Step by step
- True positive (caught)
- True negative (left alone)
- False positive (false alarm)
- False negative (missed)
Press Start or Step: the step the animation is on lights up below.
- 1
Pick a threshold
Every score at or above it becomes a positive prediction. Everything else is negative.
pred = score >= threshold - 2
Sort each example into a cell
Compare the prediction with the truth. Each example lands in exactly one of TP, FP, FN or TN.
cell[(pred, truth)] += 1 - 3
Precision
Of everything predicted positive, the share that really was. False positives lower it.
precision = TP / (TP + FP) - 4
Recall
Of everything really positive, the share that was found. False negatives lower it.
recall = TP / (TP + FN) - 5
F1
The harmonic mean of the two, so it is only high when both are.
f1 = 2 * p * r / (p + r) - 6
Accuracy
Right answers over all answers. Useful when classes are balanced and both mistakes cost the same.
accuracy = (TP + TN) / n
A classifier gives every example a score. A threshold turns the score into a decision, and comparing each decision with the truth sorts every example into one of four cells: caught (true positive), wrongly flagged (false positive), missed (false negative) or correctly left alone (true negative).
Every metric is a ratio of those four counts. Precision asks how often a flag is right. Recall asks how many of the real positives were found. They pull against each other: raise the threshold and precision usually rises while recall falls.
Accuracy counts every right answer, which sounds ideal but hides which mistakes are made, and it collapses when one class is rare. The ROC curve and its area (AUC) step back further: they judge how well the scores rank positives above negatives, across every threshold at once.
Where you'll meet it
- Choosing a threshold for any yes/no model: spam filters, fraud alerts, medical screening, content moderation.
- Comparing models: AUC for ranking quality, F1 when you need one number that respects both kinds of mistake.
- Interview favourites: “why is 99% accuracy bad here?”, “precision or recall for cancer screening?”, “what does an AUC of 0.5 mean?”.
Counting the cells is one pass. The Python here recounts at every distinct score, which keeps it short. Sorting the scores once and updating the counts as the threshold drops, as scikit-learn does, brings the ROC curve down to O(n log n).