Train a neural network interactively. Explore forward propagation, backpropagation and weight updates to see how a model learns from data.
1One neuron: weigh, add, squash
Multiply each input by a weight, add them up with a bias, then pass the total through an S-shaped activation (tanh).
2One line can't, a layer can
In XOR, opposite corners match. No single straight line separates the colours, but a hidden layer combines several lines into a curved boundary.
3Learn by passing blame backwards
Forward: data flows in, a prediction comes out. Backward: the error flows back, and each weight learns how much it was to blame, then nudges.
Background shading is the network's guess at every spot. Red rings mark training points it currently gets wrong. “Training accuracy” counts these 120 points; “on 400 new points” tests fresh points from the same shape that it never trained on.
- Training step
- 0
- Loss
- 0.721
- Training accuracy
- 59.2%
- On 400 new points
- 57.0%
Opposite corners match. No single straight line works, so it needs a hidden layer.
Press Step to walk through one training step (forward, loss, backward, update), then Start to train.
Inside every neuron
Activation functions
After a neuron adds up its weighted inputs (call the total z), it passes z through an activation function. That's the step that lets a network learn curves instead of only straight lines. Pick one below and drag z to see its output and its slope. The slope matters because it's what carries the learning signal backwards during training.
- Output f(1.00)
- 0.7616
- Slope f′(1.00)
- 0.4200
Tanh
- Formula
- tanh(z) = (eᶻ − e⁻ᶻ) / (eᶻ + e⁻ᶻ)
- Slope (derivative)
- tanh′(z) = 1 − tanh²(z)
- Output range
- −1 to 1
Good
- Centred on 0, so it trains better than sigmoid in hidden layers.
- It's a stretched sigmoid: tanh(z) = 2σ(2z) − 1, and its slope reaches 1 at z = 0.
Watch out
- Still saturates: far from 0 the curve flattens and the slope approaches 0.
Where it's used: Hidden layers of small networks (this page's default) and inside recurrent networks (RNNs, LSTMs).
Why not just skip the activation?
Without one, a layer only multiplies and adds. Two layers that only multiply and add can always be merged into a single layer that does the same thing, so stacking them gains nothing: the network can still only draw straight lines. Here's the check with real numbers, for the input x = (1, 2):
Layer 1: h = (5, 5)
Layer 2: 0.5·5 + (-1)·5 + 2 = -0.5
One merged layer: w = (2, -2.5), b = 2.5
2·1 + (-2.5)·2 + 2.5 = -0.5, the same answer.
An activation between the layers breaks this merge, and that's what lets the network bend its boundary.
Why the slope matters: vanishing gradients
When the network learns, the error signal travels backwards and is multiplied by each layer's activation slope (and its weights). Sigmoid's slope is never more than 0.25, so after a few sigmoid layers the signal is tiny and the early layers barely learn. ReLU's slope is exactly 1 for positive inputs, so the activation doesn't shrink it there.
The most that n sigmoid slopes multiplied together can be: 0.25ⁿ. Weights multiply in too; this is only the activations' share.
No magic, just arithmetic
The maths, step by step
Here is one complete training step for the smallest interesting network: 2 inputs, 2 hidden neurons (A and B) and 1 output. We use one example, x = (1, 0.5), whose correct answer is y = 1, and a learning rate of η = 0.5. Every number below is calculated live by the same code this page trains with.
w are the weights into the hidden neurons, b their biases, v the weights into the output and c its bias.
These start as small random numbers. Training is the job of nudging them until the output is right.
● forward pass● backward pass● update
- 1
Weigh and add
Each hidden neuron multiplies each input by its weight, adds them up, and adds its bias. Call the total z.z_A = w_A1·x1 + w_A2·x2 + b_A = 0.6·1 + (-0.4)·0.5 + 0.1 = 0.5000
z_B = w_B1·x1 + w_B2·x2 + b_B = (-0.3)·1 + 0.8·0.5 + (-0.2) = -0.1000
- 2
Squash with the activation
Each neuron passes its total through tanh, which bends it into the range −1 to 1. This bend is what lets the network learn curves.h_A = tanh(z_A) = tanh(0.5000) = 0.4621
h_B = tanh(z_B) = tanh((-0.1000)) = -0.0997
- 3
The output neuron gives a probability
The output does the same weigh-and-add with the hidden values, then a sigmoid turns the total into a probability between 0 and 1: how sure it is that the answer is 1.z_out = v_A·h_A + v_B·h_B + c = 0.9·0.4621 + (-0.5)·(-0.0997) + 0.1 = 0.5657
p = σ(z_out) = 1 / (1 + e^(−z_out)) = 1 / (1 + e^(−0.5657)) = 0.6378
- 4
Score the answer: the loss
The right answer is 1, and the network said 63.8%. Cross-entropy loss is −log of the probability given to the right answer: 0 when the network is certain and right, and it grows without limit when it is confident and wrong.L = −[ y·ln(p) + (1 − y)·ln(1 − p) ]
L = −ln(p) (because y = 1) = −ln(0.6378) = 0.4498
- 5
Go backwards with the chain rule
Now we ask: if we changed z_out a little, how much would the loss change? The chain rule says rates of change multiply along a path. (Here L changes by -1.57 for every 1 that p changes, and p changes by 0.23 for every 1 that z_out changes, so L changes by -1.57 × 0.23 for every 1 that z_out changes.) The two parts simplify to something very neat: p − y.∂L/∂p = −y/p + (1 − y)/(1 − p) = −1/0.6378 + 0/0.3622 = -1.5679
∂p/∂z_out = p·(1 − p) (sigmoid's slope) = 0.6378·0.3622 = 0.2310
∂L/∂z_out = ∂L/∂p · ∂p/∂z_out = p − y = 0.6378 − 1 = -0.3622
Negative means “raising z_out would lower the loss”, which makes sense: the answer should be 1 and we said only 0.6378.
- 6
Blame for the output weights
z_out = v_A·h_A + …, so raising v_A by 1 raises z_out by h_A. Multiply that by the blame on z_out from step 5. The bias c adds to z_out directly, so its blame is just p − y.∂L/∂v_A = (p − y)·h_A = (-0.3622)·0.4621 = -0.1674
∂L/∂v_B = (p − y)·h_B = (-0.3622)·(-0.0997) = 0.0361
∂L/∂c = p − y = -0.3622
- 7
Pass the blame back into the hidden layer
Same idea, one more link at a time. A hidden neuron's output reaches the loss through its output weight v. Then we go back through tanh, whose slope is 1 − tanh², and finally through each input weight, whose effect is the input value x.∂L/∂h_A = (p − y)·v_A = (-0.3622)·0.9 = -0.3260
∂L/∂z_A = ∂L/∂h_A · (1 − h_A²) = (-0.3260)·(1 − 0.4621²) = -0.2564
∂L/∂w_A1 = ∂L/∂z_A · x1 = (-0.2564)·1 = -0.2564
∂L/∂w_A2 = ∂L/∂z_A · x2 = (-0.2564)·0.5 = -0.1282
∂L/∂b_A = ∂L/∂z_A = -0.2564
Neuron B works the same way:
∂L/∂z_B = (p − y)·v_B · (1 − h_B²) = (-0.3622)·(-0.5)·(1 − (-0.0997)²) = 0.1793
∂L/∂w_B1 = ∂L/∂z_B · x1 = 0.1793
∂L/∂w_B2 = ∂L/∂z_B · x2 = 0.0897
∂L/∂b_B = ∂L/∂z_B = 0.1793
- 8
Update every weight
Each weight takes a small step against its blame: new = old − η × blame, with η = 0.5. A negative blame means the weight goes up.Weight Old Blame New w_A1 0.6000 -0.2564 0.7282 w_A2 -0.4000 -0.1282 -0.3359 b_A 0.1000 -0.2564 0.2282 w_B1 -0.3000 0.1793 -0.3897 w_B2 0.8000 0.0897 0.7552 b_B -0.2000 0.1793 -0.2897 v_A 0.9000 -0.1674 0.9837 v_B -0.5000 0.0361 -0.5181 c 0.1000 -0.3622 0.2811 Run the forward pass again with the new weights: p goes from 0.6378 to 0.7464, and the loss falls from 0.4498 to 0.2925. The network is a little less wrong. Training repeats this many times.
- 9
Check it: the nudge test
How do we know backprop got the right answer? Nudge one weight up and down by a tiny amount, measure how much the loss changes, and divide. Real libraries use this “gradient check” to test their maths.Loss with v_A + 0.0001: 0.44974594
Loss with v_A − 0.0001: 0.44977942
(first − second) ÷ 0.0002 = -0.167388
Backprop said ∂L/∂v_A = -0.167388. They match.
How the trainer above differs: it does exactly these steps, but for all 120 points at once. It averages their blames before each update (full-batch gradient descent), and it can use other activations in the hidden layer, in which case step 7 uses that activation's slope instead of 1 − tanh².
How it works: Neural Network
Step by step
- Class A (teal)
- Class B (violet)
- Gradient (blame)
- Negative weight
Press Start or Step: the step the animation is on lights up below.
- 1
Forward pass
Each hidden neuron computes its weighted sum z = w₁x + w₂y + b and applies the activation; the output neuron combines them and the sigmoid turns that into a probability.
h = act(W1·x + b1); p = σ(W2·h + b2) - 2
Measure the loss
Cross-entropy: −log of the probability given to the right answer. Confident mistakes cost the most.
L = −mean(y·log p + (1−y)·log(1−p)) - 3
Backward pass (backprop)
Start from the output error, p − y, and pass it back through each weight and each activation's slope with the chain rule.
dW2 = (p−y)·h; dW1 = (p−y)·W2·act′(z)·x - 4
Update every weight
Subtract learning rate × gradient from each weight. That's one training step.
W -= lr * dW - 5
Repeat
Forward, loss, backward, update, over and over. Watch the boundary bend and the loss fall.
for epoch in range(N): ... - 6
Trained
The background shading is the network's prediction at every spot, including places it never saw a point.
predict(x) = p ≥ 0.5
A neuron takes some numbers, multiplies each by a weight, adds them up with a bias, and passes the total through an activation function (tanh by default here). On its own, one neuron can only draw a straight line between two classes.
Put a layer of neurons in the middle and their outputs become new features that the output neuron combines. Each hidden neuron responds to one straight-line direction in the input; because the activation bends each response, the output can mix them into a curved boundary. That is how the network solves XOR and the circle.
Training is a loop. Forward: push the data through and get predictions. Loss: score how wrong they are. Backward (backpropagation): use the chain rule to work out every weight's share of the blame. Update: nudge every weight against its blame. A few hundred repeats later, the shaded background shows what the network believes everywhere. The maths section below works through one full step with real numbers.
Where you'll meet it
- Images, speech, text: anywhere the useful features are too complex to design by hand. Large language models are this, scaled up enormously.
- On small tabular data, simpler models (trees, logistic regression) are often just as good and easier to explain.
- Interview staples: “explain backprop”, “why do we need non-linear activations?”, “what is a vanishing gradient?”.
Every training step runs every point forward and backward through every weight. The backward pass costs roughly twice as much as the forward pass, so one training step is about three forward passes' worth of work.