Fit a line interactively and explore linear regression. See how predictions, residuals and loss change as the model learns from data.
1A line is a guess: y = w·x + b
w is the slope: how much y rises for each step in x. b is where the line crosses the y-axis.
2The errors are the vertical gaps
For each point, measure how far the line misses it. Square each gap and average them: the mean squared error.
3Nudge w and b to shrink the total
Gradient descent tilts and lifts the line a little each step, in whichever direction lowers the error, until it can't do better.
Watch it learn
Fit a line with gradient descent
The line starts flat and gradient descent nudges its slope and intercept, step by step, to shrink the error. The dashed line is the exact best answer, worked out in one go by the formula in the maths section; watch descent arrive on it. Click the plot to add points.
- Step
- 0
- Error (MSE)
- 17.832
- Best possible
- y = 0.76x + 0.59
- R² (1 is perfect)
- -3.890
Press Start to fit the line, or Step through the first few moves slowly.
No magic, just averages
The maths, step by step
For a straight line there's an exact answer, no stepping needed. Here it is, worked by hand on five points: (1, 2), (2, 4), (3, 5), (4, 4), (5, 5). Every number is calculated by the same code as the plot above.
- 1
Find the middle of the data
Average the x values and the y values. The best line always passes through this point.x̄ (mean of x) = (1 + 2 + 3 + 4 + 5) ÷ 5 = 3.0000
ȳ (mean of y) = (2 + 4 + 5 + 4 + 5) ÷ 5 = 4.0000
- 2
The slope: how x and y move together
For each point, multiply how far x is from its mean by how far y is from its mean. Add those up, and divide by the total of x's squared distances. If y tends to be above average when x is, the slope is positive.slope = Σ (x − x̄)(y − ȳ) ÷ Σ (x − x̄)²
x y x − x̄ y − ȳ product (x − x̄)² 1 2 -2 -2 4 4 2 4 -1 0 0 1 3 5 0 1 0 0 4 4 1 0 0 1 5 5 2 1 2 4 totals 6 10 slope = 6 ÷ 10 = 0.6000
- 3
The intercept
The line goes through the middle point (x̄, ȳ), so the intercept is whatever makes that true.intercept = ȳ − slope × x̄ = 4 − 0.6 × 3 = 2.2000
So the best line is y = 0.6x + 2.2.
- 4
Measure the misses
Predict each point with the line, subtract, square, and average. That's the mean squared error.x = 1: (y − prediction)² = (2 − 2.8)² = (-0.8)² = 0.6400
x = 2: (y − prediction)² = (4 − 3.4)² = 0.6² = 0.3600
x = 3: (y − prediction)² = (5 − 4)² = 1² = 1.0000
x = 4: (y − prediction)² = (4 − 4.6)² = (-0.6)² = 0.3600
x = 5: (y − prediction)² = (5 − 5.2)² = (-0.2)² = 0.0400
MSE = 2.4 ÷ 5 = 0.4800
No other straight line has a smaller total of squared misses for these points: that's what “least squares” means.
- 5
How good is it? R²
Compare the line's squared misses with those of the laziest possible model: always predicting the mean, ȳ. R² is the share of that spread the line explains.SS_res (line's misses²) = from step 4 = 2.4000
SS_tot (misses² of always guessing ȳ) = Σ (y − ȳ)² = 6.0000
R² = 1 − SS_res ÷ SS_tot = 1 − 2.4 ÷ 6 = 0.6000
The line explains 60% of the variation in y. R² is 1 for a perfect fit, 0 for a line no better than the mean, and negative for a line that's worse than just guessing the mean.
- 6
Why squares, and what about many features?
Squaring stops misses above and below from cancelling out, punishes big misses most, and makes the maths smooth enough to have an exact answer.With many features the same idea becomes the normal equation:
(XᵀX) β = Xᵀy
where each row of X is one example's features (plus a 1 for the intercept) and β holds all the weights. Libraries don't invert XᵀX directly; scikit-learn's LinearRegression solves it with a more stable method (singular value decomposition).
Is it any good?
R²: compared with just guessing the average
MSE tells you how far off the line is, but in squared units of y, so “0.8” means nothing on its own. R² compares the line with the laziest model there is: always predicting the average of y. Pick a line to see its score.
- R²
- 0.70
- MSE
- 1.10
The least-squares line: no straight line scores higher on these points.
R² = 1 − (line's squared misses) ÷ (squared misses of always guessing the average)
- 1 means every point is exactly on the line.
- 0 means the line is no better than the average. Despite the “²”, R² can be negative: the model is worse than the average.
- Mean absolute error (MAE) is another common score: the average size of the misses, in y's own units. It's never larger than the square root of the MSE, because squaring makes big misses count extra.
One bad point
What a single outlier does
Squaring makes a big miss count a lot: a point 4 away costs 16, as much as sixteen points 1 away. So the line bends towards a point that's far from the rest, even if it's just a typo.
- Slope without it
- 0.76
- Slope now
- 0.76
14 points that follow a clear upward trend.
What to do about it
- Plot your data first, and check whether an odd point is a mistake.
- Use a loss that punishes big misses less. Absolute error grows in a straight line instead of a square; the Huber loss is squared for small misses and absolute for big ones. Both let an outlier pull the line less.
Bending the line
Curves are linear regression too
Add x², x³ … as extra features and the same least-squares maths fits a curve, because the model is still a weighted sum (linear in its weights). The danger is giving it too much freedom. This is scikit-learn's classic example: 30 noisy points from a cosine curve, fitted with degree 1, 4 and 15.
Dashed: the true curve the points came from. Solid: the fitted polynomial.
- Error on the 30 training points
- 0.011
- Error on 1000 new points
- 0.012
About right: it follows the shape of the true curve without chasing every wobble.
What “New sample” shows
- Degree 15 always has the lowest training error: more freedom can only fit the training points better.
- On new points it did worse than degree 4 in all 300 samples we tested, but how much worse jumps around from sample to sample. Sometimes it's close; sometimes it's wildly off.
- So training error can't pick the degree. Measure error on data the model hasn't seen; scikit-learn's example uses cross-validation for this. Ridge regression is another fix: it shrinks the weights, which calms the wiggles.
Good at
- Easy to explain: each weight says how much y changes when that feature goes up by one.
- Fast, with an exact answer: no learning rate to tune if you solve it directly.
- A strong first baseline before trying anything fancier.
Watch out for
- It assumes a straight-line relationship (in its features). Plot the data first.
- Outliers pull the line, because misses are squared.
- Features that move together (say, height in cm and height in inches) make the weights unstable: tiny changes in the data can swing them wildly. Ridge regression helps here too.
How it works: Linear Regression
Step by step
- Current line
- Errors (residuals)
- Exact best fit (dashed)
Press Start or Step: the step the animation is on lights up below.
- 1
Guess a line
Start with w = 0 and b = 0, a flat line at zero. Every prediction is wrong, which is fine.
w, b = 0, 0 - 2
Measure the misses
Each red gap is a residual: prediction minus actual. Square them and average to get the MSE.
mse = mean((w*x + b - y)²) - 3
Find which way to nudge
The gradient tells us how the MSE changes as w or b changes. Positive means “decrease it”.
dw = 2·mean(err·x); db = 2·mean(err) - 4
Nudge and repeat
Move w and b a learning-rate-sized step against their gradients. The line tilts and lifts into place.
w -= lr*dw; b -= lr*db - 5
Settled
The error stops changing, and the line lands on the exact least-squares answer (dashed).
if Δmse < tol: stop - 6
Or: it blew up
Too large a learning rate makes every step overshoot further. Lower it and try again.
mse → ∞ (lr too big)
Linear regression draws the straight line that best fits a cloud of points, so you can predict y from x. The line is y = w·x + b: w is the slope (how much y rises per unit of x) and b is where it crosses the y-axis.
“Best” needs a number. For each point, measure the vertical gap between the point and the line, square it so misses above and below both count, and average them. That's the mean squared error (MSE), and a smaller one means a better line.
To shrink the error we use gradient descent: work out whether nudging w and b up or down would lower the MSE, nudge them a little that way, and repeat. For a straight line there's also an exact formula, shown dashed, so you can see descent arrive at the same answer.
Where you'll meet it
- Predicting a number: price from size, sales from ad spend, time from distance.
- As a baseline before trying anything fancier: if a line already explains the data, a neural network won't help much.
- It's the smallest complete example of the training loop (predict, measure error, compute gradient, update) that every larger model repeats.
Each step looks at every point once to compute the error and its gradient. The exact formula is O(n) for one feature too, but it only exists for models that are linear in their weights (which includes the polynomial curves on this page); gradient descent works for any model.