Explore gradient descent with interactive learning rates, loss contours and optimizer comparisons. See how SGD, momentum and Adam move toward a minimum.
1Error is a landscape
Each spot is one choice of weights; its height is how wrong the model is there. Training means finding the lowest point.
2The slope points uphill, so step the other way
The gradient is the slope under your feet. Moving against it is the fastest way down from where you stand.
3Step size (learning rate) matters
Too small crawls. Just right walks straight down. Too big jumps past the bottom and climbs out the other side.
Start simple
One weight, one curve
Imagine a model with a single setting, a weight called w. The curve shows how wrong the model is for every value of w: here the error is (w − 3)², so the best answer is w = 3. The model doesn't know that. All it can do is measure the slope where it stands and step downhill.
Every step so far
| step | w | slope | error |
|---|---|---|---|
| 0 | 0.000 | -6.000 | 9.000 |
Learning rate 0.1: Creeps closer each step: the gap shrinks to 80% every time.
Press Step: first the slope is measured (dashed line), then the ball moves along the arrow.
No magic, just arithmetic
The maths, step by step
We use the same curve as above: error(w) = (w − 3)², starting from w = 0 with a learning rate of η = 0.1 (η, “eta”, is the usual symbol for the learning rate). Every number here is calculated by the same code that runs the ball above.
- 1
What “slope” means
The slope tells you how fast the error changes when you nudge w. You can measure it directly: move w a tiny bit, see how much the error moves, and divide.error(0) = (0 − 3)² = 9.0000
error(0.001) = (0.001 − 3)² = 8.9940
slope ≈ change ÷ nudge = (8.994001 − 9) ÷ 0.001 = -5.9990
Calculus gives the exact slope without nudging: for (w − 3)² it is 2·(w − 3), which at w = 0 is -6. The nudge gets almost the same answer.
- 2
The one rule
Step against the slope. If the slope is positive, the error rises as w grows, so make w smaller. If it's negative, make w bigger. The learning rate η decides how big the step is.new w = w − η × slope
That's all gradient descent is. Everything else on this page is a variation on how big the step should be and which way it points.
- 3
Run it with real numbers
Apply the rule three times from w = 0. Each line uses the slope at the current w.w1 = w0 − η × 2·(w0 − 3) = 0 − 0.1 × (-6) = 0.6000
w2 = w1 − η × 2·(w1 − 3) = 0.6 − 0.1 × (-4.8) = 1.0800
w3 = w2 − η × 2·(w2 − 3) = 1.08 − 0.1 × (-3.84) = 1.4640
The error falls each time: 9 → 5.76 → 3.686 → 2.359. Steps get smaller on their own, because the slope gets flatter near the bottom.
- 4
The pattern: the gap shrinks by the same factor
Put the slope into the rule and simplify. The distance to the answer is multiplied by the same number, (1 − 2η), every single step.new w − 3 = (w − 3) − η·2·(w − 3) = (1 − 2η)·(w − 3)
factor for η = 0.1 = 1 − 2 × 0.1 = 0.8000
Gaps to the answer: 3 → 2.4 → 1.92 → 1.536 → 1.229, each one 0.8 times the last.
- 5
Why the learning rate matters so much
Because the gap is multiplied by (1 − 2η) every step, that one number decides what happens. If it's between −1 and 1 the gap shrinks; outside that, it grows.η 1 − 2η What happens 0.1 0.8 Small steps, approaches from one side 0.5 0 Lands exactly on 3 in one step 0.8 -0.6 Overshoots, bounces, still settles 1 -1 Bounces between two points forever 1.05 -1.1 Each bounce is bigger: blows up So for this curve, any η between 0 and 1 works and anything above 1 fails. For any bowl-shaped curve like this one, the limit is 2 ÷ (how sharply it bends, its second derivative). This curve bends by 2, so the limit is 1; a steeper bowl has a smaller limit.
- 6
More than one weight: the gradient
Real models have many weights. You measure one slope per weight (each one says how the error changes if only that weight moves). That list of slopes is called the gradient, written ∇. The rule is the same for every weight.error(x, y) = x² + y² → gradient = (2x, 2y)
gradient at (-2.4, 1.5) = (2 × (-2.4), 2 × 1.5) = (-4.8, 3)
new x = x − η × slope_x = -2.4 − 0.1 × (-4.8) = -1.9200
new y = y − η × slope_y = 1.5 − 0.1 × 3 = 1.2000
The gradient points straight uphill on the map, so stepping against it is the steepest way down. The map below shows exactly this, for two weights at once.
Two weights
The map view
With two weights, the error becomes a landscape seen from above, like a hiking map: each ring is one height, and darker means lower error. The amber arrow is the downhill direction (against the gradient). Click anywhere on the map to drop the ball there.
- Step
- 0
- Error
- 8.010
- Position
- -2.40, 1.50
- Slope size
- 5.660
One minimum, equally steep in every direction. The easy case.
Plain descent settles for any learning rate below 1. At exactly 1 it bounces forever, and above 1 it blows up.
Press Start to roll the ball downhill, or Step to go one move at a time.
How much data per step
Batch, stochastic and mini-batch
To measure the slope exactly, you'd check every example in your data, which is slow when there are millions. The three versions of gradient descent differ only in how many examples they look at before each step. Here all three fit a line to the same 40 points, from the same start, with the same learning rate (0.15).
Batch
- Points per step
- 40
- Points looked at
- 0
- Error now
- 6.420
Every step looks at all the data. The direction is exact, so the path is smooth, but each step is expensive.
Stochastic (SGD)
- Points per step
- 1
- Points looked at
- 0
- Error now
- 6.420
Every step looks at one random point. Steps are cheap but noisy, so the path wobbles.
Mini-batch (8)
- Points per step
- 8
- Points looked at
- 0
- Error now
- 6.420
Every step looks at a small random group. Less noisy than SGD, much cheaper than batch.
Press Start: the teal path walks smoothly, the red one wobbles, the amber one is in between.
In practice almost everyone uses mini-batches: they're nearly as cheap as single examples, the noise is small, and computers process a group of examples at once very efficiently. Common batch sizes are between about 50 and 256. People often say “SGD” when they actually mean mini-batch. Because the stochastic versions never see the exact slope, they keep jittering near the bottom unless the learning rate is lowered over time (see schedules below).
Smarter steps
Momentum, Nesterov, AdaGrad, RMSProp and Adam
Plain gradient descent has two weaknesses: in a narrow valley it zigzags across and crawls along, and it uses the same step size for every weight. Each method below fixes one or both. Pick a surface and watch them race from the same start.
Tap a method to show or hide it. “Bottom” is the step where it first got within 0.001 of the lowest error.
Each method has its own learning rate, picked to suit it: adaptive methods (AdaGrad, RMSProp, Adam) scale their steps differently, so the same number would not be a fair comparison. Momentum and Nesterov use β = 0.9; Adam uses its defaults for β₁, β₂ and ε.
Steep across, shallow along. Plain descent zigzags across and crawls along. Momentum builds up speed along the floor and gets to the bottom much sooner, though it swings from side to side on the way.
Plain gradient descent
Step straight downhill, the same size every time (learning rate × slope).
x ← x − lr · g
No extra settings: just the learning rate.
Momentum
Remember the last move and keep most of it, like a heavy ball rolling downhill. Steps that keep pointing the same way build up speed; steps that flip back and forth cancel out.
v ← β · v − lr · g
x ← x + v
β (how much of the last move to keep) is usually 0.9.
Nesterov momentum
Momentum, but measure the slope at the spot the momentum is about to carry you to, not where you are now. It's like looking ahead before you step, so it brakes earlier when it's about to overshoot.
g ← slope at (x + β · v)
v ← β · v − lr · g
x ← x + v
β is usually 0.9, the same as momentum.
AdaGrad
Give every weight its own step size. Weights that have had big slopes get smaller steps; weights with small slopes get bigger ones. The catch: it adds up every slope it has ever seen, so the steps keep shrinking and can stall.
G ← G + g² (running total)
x ← x − lr · g / (√G + ε)
A learning rate of 0.01 is the common default; ε is a tiny number (like 1e-8) to avoid dividing by zero.
RMSProp
AdaGrad's per-weight step sizes, but with a moving average of recent squared slopes instead of an ever-growing total, so it never grinds to a halt.
E ← ρ · E + (1 − ρ) · g² (moving average)
x ← x − lr · g / (√E + ε)
ρ = 0.9 and a learning rate of 0.001 are the suggested defaults.
Adam
Momentum and RMSProp together: a moving average of the slopes (which way to go) and of the squared slopes (how big a step each weight should take). It also corrects both averages at the start, because they begin at 0.
m ← β₁ · m + (1 − β₁) · g
s ← β₂ · s + (1 − β₂) · g²
m̂ = m / (1 − β₁ᵗ), ŝ = s / (1 − β₂ᵗ)
x ← x − lr · m̂ / (√ŝ + ε)
lr = 0.001, β₁ = 0.9, β₂ = 0.999, ε = 1e-8 (the paper's and PyTorch's defaults).
AdamW
Adam plus weight decay, done properly. Weight decay gently shrinks every weight towards 0 each step, which helps a model not to overfit. Adam's built-in version adds the shrink to the slope, so it goes through Adam's averaging and per-weight rescaling like everything else. AdamW keeps it separate and applies it straight to the weights, which works more like true weight decay.
x ← x − lr · λ · x (shrink first)
then the usual Adam step
PyTorch's defaults: lr = 0.001, β₁ = 0.9, β₂ = 0.999, weight decay λ = 0.01. Not shown in the race, because weight decay only matters for real models with many weights.
Note on notation: this page writes momentum the way CS231n does (the learning rate inside the velocity). PyTorch writes it as v ← β·v + g, x ← x − lr·v. They are the same idea with the learning rate in a different place.
Changing the step size over time
Learning rate schedules
Big steps help early, when you're far from the bottom. Near the bottom they make you overshoot, and with mini-batches the noise keeps you jittering. A schedule starts with a larger learning rate and lowers it as training goes on. Three common ones:
Constant
lr(t) = lr₀
Step decay
halve every 10 epochs
Exponential
lr(t) = lr₀ · e^(−k·t), k = 0.07
1/t decay
lr(t) = lr₀ / (1 + k·t), k = 0.2
Starting from 0.1. Step decay is the simplest and a common choice, for example halving the rate every few epochs or dividing it by 10 every 20. An epoch is one full pass through the training data.
When it goes wrong
Common problems, and what helps
Local minima
A dip that isn't the lowest point. Plain descent stops in whichever dip it reaches first, because every direction there is uphill.
Try it: Map view, “Two valleys”: plain descent at learning rate 0.05 stops in the shallower valley; with momentum it rolls on into the deeper one.
Saddle points
Flat in the middle, downhill one way and uphill the other, like a horse saddle. The slope is almost 0, so steps become tiny. In big networks, research suggests saddle points slow training down more often than local minima.
Try it: Race, “Saddle point”: plain descent hangs around the middle for about 40 steps; Adam and AdaGrad hardly pause.
Narrow valleys
Steep across and gentle along. The steep side limits how big the step can be, and with that small step the gentle side takes forever. This often happens when inputs have very different scales.
Try it: Map view, “Narrow valley”, or the race: compare Plain with Momentum and Adam.
Feature scaling
If one input is in the thousands and another is between 0 and 1, the error surface becomes a narrow valley. Rescaling inputs to similar ranges (for example, subtracting the mean and dividing by the standard deviation) makes it rounder, so plain descent works much better.
Try it: The Linear Regression page does exactly this before fitting.
In practice
Which one should I use?
- Start with Adam (or AdamW) at its default settings. Stanford's CS231n course recommends it as the default, and it needs the least tuning.
- Try SGD with momentum or Nesterov as the alternative. With a well-tuned learning rate and a schedule, it's the other widely used choice.
- Use mini-batches (commonly 50 to 256 examples), and shuffle the data each epoch.
- Tune the learning rate first. It's the setting that makes the biggest difference on every page here: if the loss goes up or jumps around, it's too big; if it barely moves, it's too small.
How it works: Gradient Descent
Step by step
- Current position
- Downhill direction
- Path so far
- Minimum
Press Start or Step: the step the animation is on lights up below.
- 1
Pick a starting point
Models start from random weights. Here you can click the map to choose where to drop the ball.
w = initial_guess - 2
Measure the slope
The gradient says how fast the loss grows in each direction. The arrow points the way we'll move: straight downhill, against the gradient.
g = grad(loss, w) - 3
Step downhill
Move by learning rate × gradient. With momentum, add most of the previous move too, so the ball keeps rolling.
w = w - lr * g - 4
Stop when it's flat (or out of steps)
Near a minimum the slope is close to zero, so steps shrink on their own. We stop when they're tiny or we hit the step limit.
if norm(g) < tol: break - 5
Or: it blew up
Each step overshot so far that the loss grew instead of shrinking. The fix is a smaller learning rate.
loss → ∞ (lr too big)
A model learns by adjusting its settings, called weights, to make its mistakes smaller. Picture the mistakes as a hilly landscape: each spot is one choice of weights, and the height is how wrong the model is there. Learning means finding a low spot.
Imagine standing on that hill in thick fog. You can't see the bottom, but you can feel which way the ground slopes under your feet. So you take a step downhill, feel again, and repeat. That's gradient descent. The slope is called the gradient, and the size of each step is the learning rate.
The learning rate is the one setting that matters most. Too small and you crawl; too big and you jump past the bottom and land higher up the other side, again and again, until you fly off. Improvements like momentum and Adam change how big each step is and which way it points, but the idea stays the same.
Where you'll meet it
- Training almost every model: linear and logistic regression, neural networks, and large language models all learn by some form of gradient descent.
- Variants you'll hear about: stochastic (SGD, one mini-batch at a time), momentum, and Adam, which adapts the step size per weight.
- Interview favourites: “why is my loss going up?” (learning rate too high), “why does training stall?” (too low, a flat region, or a local minimum).
Each step needs one gradient computation. For a neural network that's one forward and one backward pass over the data (or a mini-batch of it).