Follow one sentence through a real Transformer layer: embeddings, positional encoding, multi-head attention, Add & Norm and feed-forward. Explore masking, the full encoder–decoder and the 2017 paper's numbers.
1Every word at once
Older translation models read a sentence one word at a time. The Transformer reads every word at once, and attention lets any word use information from any other word in a single step.
2One layer: two sub-layers
Each layer does two things: attention (words exchange information) and a feed-forward network (each word thinks on its own). Around both: a shortcut that adds the input back, then a normalisation.
3No peeking ahead
When the decoder writes a translation, each position may only attend to the positions before it. A mask sets the future scores to −∞, so softmax gives them exactly zero weight.
Watch the data move
One sentence through one layer
This is a real Transformer layer, computed in your browser, with the paper's steps in the paper's order. Only the sizes are shrunk so you can read every number: 8 numbers per word instead of 512, 2 heads instead of 8. The weights are random, not trained, so watch the path and the shapes, not the particular values. The map of the layer shows where you are.
One row per token from here on. Everything the layer does keeps that row structure: row 3 is always about the third word.
Press Step to send the sentence into the layer, one stage at a time.
Order matters
Why positional encoding?
Attention compares every word with every other, but nothing in that comparison says which word came first. Shuffle the words and each word's output stays the same, just in a different row. The paper's fix is to add a fixed pattern of sines and cosines to each word's embedding: every position gets its own fingerprint.
Teal is positive, red negative. Each pair of columns is one wave: the left columns flip every few positions, the right ones barely change across 40. Together they give every position a unique code, and nearby positions similar ones.
The two “the”s in “the cat sat on the mat”
- Positions off: biggest difference between their outputs
- 0 (identical)
- Positions on: biggest difference between their outputs
- 0.749
Without positions, “the” at the start and “the” before “mat” are the same word in the same sentence, so the layer gives them exactly the same output. With positions they differ, so later layers can tell “the cat” from “the mat”.
Several views at once
Many heads
Instead of one attention with all 8 dimensions, the paper splits them between 2 heads here (8 in the paper). Each head has its own projections, so each can learn to look for something different, at the same total cost. In a trained model, some heads track the previous word, some link pronouns to nouns, some follow syntax. These are untrained, but you can already see two different patterns from the same input.
Uses the sentence and settings chosen above. Row = the word looking, column = the word looked at.
The full picture
The whole model: encoder and decoder
The paper built the Transformer for translation. The encoder reads the source sentence; the decoder writes the translation one token at a time, each step reading both what it has written so far and the encoder's output. Select a box to see what it does.
Encoder–decoder attention
Queries come from the decoder; keys and values come from the encoder's output. This is where each word being written looks back at the source sentence: the translation step.
From the paper
The paper's numbers
The sizes from Table 3 of the paper, with parameters counted from the formulas on this page. The count lands close to what the paper reports; the exact figure depends on details such as the vocabulary size (about 37,000 shared English–German tokens).
| Transformer (base) | Transformer (big) | this page | |
|---|---|---|---|
| layers (encoder, decoder) | 6 + 6 | 6 + 6 | 1 |
| d_model (numbers per token) | 512 | 1024 | 8 |
| heads | 8 | 16 | 2 |
| d_k = d_model / heads | 64 | 64 | 4 |
| d_ff (feed-forward width) | 2048 | 4096 | 32 |
| parameters, counted | 63.1M | 214.2M | – |
| parameters, paper | 65M | 213M | – |
| BLEU, English→German | 27.3 | 28.4 | – |
Where the base model's parameters are:
Inside each encoder layer, the feed-forward network (2.1M) holds twice the weights of attention (1.1M).
The learning-rate schedule (Equation 3)
Across: training step, in thousands. Up: learning rate, in thousandths (0.6 = 0.0006).
lr = d_model^−0.5 · min(step^−0.5, step · 4000^−1.5)
It rises in a straight line for 4,000 steps, then falls as 1/√step. Starting small keeps the early, random updates from wrecking the model. The paper trained with Adam (β₁ = 0.9, β₂ = 0.98, ε = 10⁻⁹), dropout 0.1 and label smoothing 0.1; the base model trained for 100,000 steps, about 12 hours on 8 P100 GPUs.
No magic, just arithmetic
The maths, step by step
Follow the word “cat” (position 1, counting from 0 as the paper does) through the layer, on the sentence “the cat sat”. Every number below comes from the same code as the stepper above.
- 1
Positional encoding
Even columns use sine, odd columns cosine, each pair with a longer wavelength than the last.PE(1, 0) = sin(pos / 10000^(0/8)) = sin(1) = 0.8415
PE(1, 1) = cos(pos / 10000^(0/8)) = cos(1) = 0.5403
PE(1, 2) = sin(pos / 10000^(2/8)) = sin(1 / 10) = 0.0998
Position 0 is always [0, 1, 0, 1, …]. Nearby positions get similar codes; far-apart ones differ more.
- 2
One attention score
How well does “cat”'s query match “the”'s key? A dot product, then divide by √d_k.q = [-1.0941, -1.0302, 0.0658, -0.4107]
k = [1.0544, -0.0962, 0.7606, 0.7286]
q · k = Σ qᵢ kᵢ = -1.3037
score = q · k ÷ √4 = -1.3037 ÷ 2 = -0.6519
Why divide? With d_k numbers multiplied and added, the dot product grows with d_k, and large scores push softmax into putting almost all the weight on one word, where its gradients vanish.
- 3
Softmax over the row
Exponentiate every score in the row and divide by the total, so the row adds up to 1.scores for “cat”: [-0.6519, 0.3256, -0.3970]
weight → “the” = e^score ÷ Σ e^score = 0.2021
weight → “cat” = e^score ÷ Σ e^score = 0.5371
weight → “sat” = e^score ÷ Σ e^score = 0.2608
- 4
Add & Norm on one row
Add the input back, then subtract the row's mean and divide by its standard deviation.mean = Σ ÷ 8 = 0.3546
variance = Σ (v − mean)² ÷ 8 = 1.1444
first value = (v − mean) ÷ √(variance + ε) = (1.3226 − 0.3546) ÷ 1.0697 = 0.9049
A trained model then multiplies by a learned γ and adds a learned β for each column. They start at 1 and 0, which is what this untrained layer uses.
- 5
The feed-forward network
Two matrix multiplications with a ReLU between them, on this word alone.8 numbers → × W₁ (8 × 32) + b₁ → 32 numbers → ReLU → 21 of them become 0 → × W₂ (32 × 8) + b₂ → 8 numbers
Then Add & Norm once more. The result for “cat”: [0.90, -0.40, -0.12, 1.57, -2.05, 0.47, -0.37, -0.01].
What happened next
Then and now
The layer on this page is still the core of today's language models, with a few common changes.
Decoder-only
GPT-style models drop the encoder and cross-attention, keeping masked self-attention: every model that writes text one token at a time.
Encoder-only
BERT-style models keep only the encoder, reading the whole input in both directions. Common for search, classification and embeddings.
Pre-norm
Many later models normalise before each sub-layer, x + sublayer(norm(x)), instead of after. It tends to make deep stacks easier to train.
Other position schemes
The fixed sine waves were largely replaced by learned positions or by rotary embeddings (RoPE), which build position into the query–key comparison.
Longer contexts
Attention compares every pair of tokens, so its cost grows with the square of the length. Much later work aims to make long inputs cheaper.
Same building blocks
Tokens, embeddings, attention, feed-forward, residuals and normalisation: each has its own page in this section. Tokenization · Self-Attention · Sampling
How it works: The Transformer
Step by step
- Positive value
- Negative value
- Attention weight
- Add & Norm / residual
Press Start or Step: the step the animation is on lights up below.
- 1
Tokens in
The sentence arrives as token ids, one row per token from here on.
ids = tokenizer(text) - 2
Embed
Look up each token's vector and multiply by √d_model.
x = E[ids] * sqrt(d_model) - 3
Add positions
Add a fixed sine/cosine pattern so the model can tell positions apart.
x = x + PE[:n] - 4
Project Q, K, V
Each head multiplies x by its own three matrices.
Q, K, V = x Wq, x Wk, x Wv - 5
Score and mask
Every query against every key, divided by √d_k. A causal mask sets future positions to −∞.
S = Q Kᵀ / sqrt(d_k) + mask - 6
Softmax
Each row becomes attention weights that add up to 1.
A = softmax(S) - 7
Mix the values
Each word's output is its weighted average of the value rows.
head = A V - 8
Concat heads, W_O
Heads side by side, then one projection back to d_model.
attn = concat(heads) Wo - 9
Add & Norm
Residual connection, then layer normalisation on each row.
a = LayerNorm(x + attn) - 10
Feed-forward
Widen, ReLU, shrink, on each word separately.
f = max(0, a W1 + b1) W2 + b2 - 11
Add & Norm
Same again. The output has the input's shape, so layers stack.
out = LayerNorm(a + f)
“Attention Is All You Need” (Vaswani et al., 2017) replaced the recurrent networks used for translation with a model built from attention alone. Every word is processed at the same time, and attention lets any word draw on any other in a single step, which made training far more parallel.
A layer has two sub-layers. Multi-head self-attention lets words exchange information, several times in parallel with different learned projections. A feed-forward network then processes each word on its own. Each sub-layer is wrapped in a residual connection and layer normalisation, so the layer only has to learn a change to its input.
Attention ignores word order, so positional encodings are added to the embeddings first. The original model stacks 6 such layers in an encoder and 6 in a decoder; the decoder adds a causal mask and an attention step that reads the encoder's output.
Where you'll meet it
- The architecture behind today's language models: GPT-style models keep only the decoder (masked self-attention), BERT-style models keep only the encoder.
- Beyond text: Vision Transformers cut images into patches, and speech and protein models use the same layer.
- Interview favourites: “why divide by √d_k?”, “why multiple heads?”, “what are positional encodings for?”, “what does the causal mask do?”.
The attention scores compare every word with every other (n²), and the projections and feed-forward cost about n · d² for n words of width d. The n² term is why long contexts are expensive.