Evaluation Metrics

Move a threshold and watch a confusion matrix fill. Explore precision, recall, F1, ROC curves, AUC and the accuracy trap on imbalanced data.

really spamreally normal00.51← inboxflagged →

1A score, then a cut

A classifier doesn't say yes or no. It gives each email a score. A threshold turns scores into decisions: everything to the right is flagged as spam.

flaggednot flaggedreally spamreally normal6true positive2false negative3false positive9true negative

2Four kinds of answer

Compare each decision with the truth. Right answers are true positives and true negatives. The two mistakes are different: a false positive hides a real email; a false negative lets spam through.

precisionrecalllow thresholdhigh thresholdcatch more spamfewer false alarms

3A trade-off, not one number

Raise the threshold and the filter only flags what it's sure about: precise, but it misses spam. Lower it and it catches more spam, with more false alarms. You choose the point.

Watch it get judged

Move the threshold, fill the table

A spam filter has scored 20 emails (made up for this page, so you can count them). Each dot is one email: spam on the top row, normal email below. Everything right of the dashed line gets flagged. Press Step to judge the emails one by one and watch each land in its cell, or drag the threshold and see every number change at once.

really spamreally normal00.250.50.751scoreYou won a free cruise: score 0.95, TPClaim your cash prize now: score 0.91, TPCheap pills, limited offer: score 0.86, TPYour account will be closed: score 0.78, TPHot deals inside: score 0.71, TPWork from home, earn big: score 0.62, TPQuick question about your order: score 0.44, FNMeeting notes (see link): score 0.27, FNFree lunch on Friday!: score 0.83, FPYour prize-draw ticket from the fair: score 0.66, FPLimited seats for the workshop: score 0.55, FPInvoice for March: score 0.48, TNCan you review my draft?: score 0.38, TNFlight confirmation: score 0.33, TNTeam photo from the offsite: score 0.25, TNDinner on Saturday?: score 0.19, TNProject update: score 0.14, TNHappy birthday!: score 0.09, TNMinutes from Monday: score 0.06, TNLibrary book due: score 0.03, TNthreshold 0.50 → flagged
caught (TP)left alone (TN)false alarm (FP)missed (FN)
0.50

Tip: click anywhere on the strip to put the threshold there. Hover a dot to see the email.

Step 0 / 25
900 ms

At 0.50, the filter flags 9 emails: 6 spam and 3 normal. Press Step to judge them one at a time.

Confusion matrix

all 20 emails

flagged as spam
not flagged
really spam
6
True positive (TP)
spam, caught
2
False negative (FN)
spam, missed
8
really normal
3
False positive (FP)
normal, wrongly flagged
9
True negative (TN)
normal, left alone
12
9
11
Precision · When it flags, is it right?
66.7%TP ÷ (TP + FP)
Recall · How much spam does it catch?
75.0%TP ÷ (TP + FN)
F1 score · Both at once
70.6%2·P·R ÷ (P + R)
Accuracy · How many right overall?
75.0%(TP + TN) ÷ all

Hover a metric to see which cells it uses.

The big trade-off

Precision vs recall

One threshold, two numbers that pull in opposite directions. The chart shows both at every threshold from 0 to 1; the amber line is yours (click the chart to move it). There is no setting that wins both: you choose which mistake you'd rather make.

00.250.50.75100.250.50.751
precision recall F1across: threshold
0.50
Precision
66.7%
Recall
75.0%
F1
70.6%
  • High threshold: only the surest emails are flagged. Few false alarms (high precision), but spam slips through (low recall).
  • Low threshold: nearly everything is flagged. All the spam is caught (high recall), along with real email (low precision).
  • Past the highest score nothing is flagged, so precision has no value and its line stops.

Every threshold at once

The ROC curve and AUC

Slide the threshold from 1 down to 0 and plot where you are each time: across, the share of normal email wrongly flagged; up, the share of spam caught. Each spam email you pass steps the curve up; each normal email steps it right. A good filter climbs first and hugs the top-left corner. The area under the curve (AUC) sums it up in one number.

Filter

Mostly right, with a few normal emails that look spammy and a few spam emails that slip under.

really spamreally normal00.250.50.751scoreYou won a free cruise: score 0.95, TPClaim your cash prize now: score 0.91, TPCheap pills, limited offer: score 0.86, TPYour account will be closed: score 0.78, TPHot deals inside: score 0.71, TPWork from home, earn big: score 0.62, TPQuick question about your order: score 0.44, FNMeeting notes (see link): score 0.27, FNFree lunch on Friday!: score 0.83, FPYour prize-draw ticket from the fair: score 0.66, FPLimited seats for the workshop: score 0.55, FPInvoice for March: score 0.48, TNCan you review my draft?: score 0.38, TNFlight confirmation: score 0.33, TNTeam photo from the offsite: score 0.25, TNDinner on Saturday?: score 0.19, TNProject update: score 0.14, TNHappy birthday!: score 0.09, TNMinutes from Monday: score 0.06, TNLibrary book due: score 0.03, TNthreshold 0.50 → flagged
0.50
AUC
0.854
Recall (up)
75.0%
False positive rate
25.0%

The amber dot is your threshold. AUC is also the chance that a random spam email scores higher than a random normal one: here 82 of the 96 spam–normal pairs.

  • Bottom-left corner: threshold above every score, nothing flagged.
  • Top-right corner: threshold at 0, everything flagged.
  • A perfect filter goes straight up, then across: AUC 1. The diagonal is guessing: AUC 0.5.
00.250.50.75100.250.50.751coin flipfalse positive rate →↑ true positive rate (recall)

When accuracy lies

The accuracy trap: a rare condition

1,000 people are screened for a condition that 10 of them have. Each square below is one person. Compare a “model” that simply says “healthy” to everyone with a real test, and look at which one has the better accuracy.

Model
sick, caught (0) sick, missed (10) healthy, flagged (0) healthy, cleared (990)
Accuracy
99.0%
Recall: sick people found
0.0%
Precision: flagged people who are sick
– (no one flagged)

A “model” that never flags anyone. It is right about all 990 healthy people and misses every sick one.

99% accuracy, and every sick person is sent home. When one class is rare, accuracy mostly measures how rare it is.

No magic, just counting

The maths, step by step

The same 20 emails at threshold 0.5. Every metric is a fraction of the four counts, so once you have the table, the rest is division. Every number below comes from the same code as the plots above.

  1. 1

    Count the four cells

    Sort every email by two questions: was it flagged, and is it really spam?

    TP = 6 (spam, flagged) · FN = 2 (spam, missed)

    FP = 3 (normal, flagged) · TN = 9 (normal, left alone)

    6 + 2 = 8 spam, 3 + 9 = 12 normal: all 20 emails, each counted once.

  2. 2

    Precision: trust in a flag

    Look only at the flagged column. What share of it is really spam?

    precision = TP ÷ (TP + FP) = 6 ÷ (6 + 3) = 0.6667

    So when the filter flags an email, it's right about 67% of the time. False positives pull it down.

  3. 3

    Recall: how much spam is caught

    Look only at the spam row. What share of it was flagged?

    recall = TP ÷ (TP + FN) = 6 ÷ (6 + 2) = 0.7500

    It catches 75% of the spam. False negatives pull it down. Recall is also called sensitivity, or the true positive rate.

  4. 4

    F1: why not just average them?

    F1 is the harmonic mean, which stays low unless both numbers are high.

    F1 = 2 · P · R ÷ (P + R) = 2 · 0.6667 · 0.7500 ÷ (0.6667 + 0.7500) = 0.7059

    Now a filter that flags just one email, correctly: precision 1, recall 0.05. The plain average is 0.525, which sounds fine. F1 is 0.095, which says what's really going on: it catches almost nothing.

  5. 5

    Accuracy and specificity

    Accuracy uses all four cells. Specificity is recall for the normal class.

    accuracy = (TP + TN) ÷ all = (6 + 9) ÷ 20 = 0.7500

    specificity = TN ÷ (TN + FP) = 9 ÷ (9 + 3) = 0.7500

    false positive rate = FP ÷ (FP + TN) = 3 ÷ (3 + 9) = 1 − specificity = 0.2500

  6. 6

    AUC: one number for every threshold

    Pair every spam email with every normal email. How often does the spam score higher?

    8 spam × 12 normal = 96 pairs. For example, “You won a free cruise” (0.95) outscores all 12 normal emails.

    AUC = pairs the spam wins ÷ all pairs = 82 ÷ 96 = 0.8542

    The area under the ROC curve, measured with trapezoids, gives the same 0.8542. 1.0 means every spam email outscores every normal one; 0.5 is a coin flip. AUC never looks at a threshold, so it judges the ranking, not the cut.

In practice

Which metric should I use?

Start from the question: which mistake costs more?

Spam filter → precision

A real email lost in the spam folder costs more than one spam email reaching the inbox. Keep false positives rare.

Cancer screening → recall

Missing a sick person is far worse than a false alarm that a second test will clear. Keep false negatives rare.

Fraud, rare events → precision, recall and F1

Accuracy is meaningless when positives are rare. Report precision and recall (or the precision–recall curve), and F1 when you need one number.

Comparing models → AUC

AUC judges how well the scores rank positives above negatives, before any threshold is chosen. Pick the threshold afterwards.

Balanced classes, equal costs → accuracy

When both classes are common and both mistakes cost the same, accuracy is a fair, easy-to-read summary.

Predicting numbers → MSE, MAE, R²

Regression has no confusion matrix. See the Linear Regression page for mean squared error, mean absolute error and R². Open Linear Regression →

How it works: Evaluation Metrics

Step by step

  • True positive (caught)
  • True negative (left alone)
  • False positive (false alarm)
  • False negative (missed)

Press Start or Step: the step the animation is on lights up below.

  1. 1

    Pick a threshold

    Every score at or above it becomes a positive prediction. Everything else is negative.

    pred = score >= threshold
  2. 2

    Sort each example into a cell

    Compare the prediction with the truth. Each example lands in exactly one of TP, FP, FN or TN.

    cell[(pred, truth)] += 1
  3. 3

    Precision

    Of everything predicted positive, the share that really was. False positives lower it.

    precision = TP / (TP + FP)
  4. 4

    Recall

    Of everything really positive, the share that was found. False negatives lower it.

    recall = TP / (TP + FN)
  5. 5

    F1

    The harmonic mean of the two, so it is only high when both are.

    f1 = 2 * p * r / (p + r)
  6. 6

    Accuracy

    Right answers over all answers. Useful when classes are balanced and both mistakes cost the same.

    accuracy = (TP + TN) / n

A classifier gives every example a score. A threshold turns the score into a decision, and comparing each decision with the truth sorts every example into one of four cells: caught (true positive), wrongly flagged (false positive), missed (false negative) or correctly left alone (true negative).

Every metric is a ratio of those four counts. Precision asks how often a flag is right. Recall asks how many of the real positives were found. They pull against each other: raise the threshold and precision usually rises while recall falls.

Accuracy counts every right answer, which sounds ideal but hides which mistakes are made, and it collapses when one class is rare. The ROC curve and its area (AUC) step back further: they judge how well the scores rank positives above negatives, across every threshold at once.

Where you'll meet it

  • Choosing a threshold for any yes/no model: spam filters, fraud alerts, medical screening, content moderation.
  • Comparing models: AUC for ranking quality, F1 when you need one number that respects both kinds of mistake.
  • Interview favourites: “why is 99% accuracy bad here?”, “precision or recall for cancer screening?”, “what does an AUC of 0.5 mean?”.
time O(n) per threshold; O(n²) for the ROC curve as writtenspace O(n)

Counting the cells is one pass. The Python here recounts at every distinct score, which keeps it short. Sorting the scores once and updating the counts as the threshold drops, as scikit-learn does, brings the ROC curve down to O(n log n).