The Transformer

Follow one sentence through a real Transformer layer: embeddings, positional encoding, multi-head attention, Add & Norm and feed-forward. Explore masking, the full encoder–decoder and the 2017 paper's numbers.

one after anotherall at once, with attention

1Every word at once

Older translation models read a sentence one word at a time. The Transformer reads every word at once, and attention lets any word use information from any other word in a single step.

inputattentionadd & normfeed-forwardadd & normx +sub(x)

2One layer: two sub-layers

Each layer does two things: attention (words exchange information) and a feed-forward network (each word thinks on its own). Around both: a shortcut that adds the input back, then a normalisation.

looks at →Iamverytired.Iamverytired.−∞hidden

3No peeking ahead

When the decoder writes a translation, each position may only attend to the positions before it. A mask sets the future scores to −∞, so softmax gives them exactly zero weight.

Watch the data move

One sentence through one layer

This is a real Transformer layer, computed in your browser, with the paper's steps in the paper's order. Only the sizes are shrunk so you can read every number: 8 numbers per word instead of 512, 2 heads instead of 8. The weights are random, not trained, so watch the path and the shapes, not the particular values. The map of the layer shows where you are.

residualInput tokensEmbedding × √d_model+ Positional encodingMulti-head attentionAdd & NormFeed-forwardAdd & NormQ K VscoressoftmaxmixW_O
1the2cat3sat4on5the6mat

One row per token from here on. Everything the layer does keeps that row structure: row 3 is always about the third word.

Sentence
Head shown
Attention
Positional encoding
Step 0 / 11
1800 ms

Press Step to send the sentence into the layer, one stage at a time.

Order matters

Why positional encoding?

Attention compares every word with every other, but nothing in that comparison says which word came first. Shuffle the words and each word's output stays the same, just in a different row. The paper's fix is to add a fixed pattern of sines and cosines to each word's embedding: every position gets its own fingerprint.

positional encoding: 40 positions (rows) × 32 dimensions (columns)40 × 32

Teal is positive, red negative. Each pair of columns is one wave: the left columns flip every few positions, the right ones barely change across 40. Together they give every position a unique code, and nearby positions similar ones.

The two “the”s in “the cat sat on the mat”

Positions off: biggest difference between their outputs
0 (identical)
Positions on: biggest difference between their outputs
0.749

Without positions, “the” at the start and “the” before “mat” are the same word in the same sentence, so the layer gives them exactly the same output. With positions they differ, so later layers can tell “the cat” from “the mat”.

Several views at once

Many heads

Instead of one attention with all 8 dimensions, the paper splits them between 2 heads here (8 in the paper). Each head has its own projections, so each can learn to look for something different, at the same total cost. In a trained model, some heads track the previous word, some link pronouns to nouns, some follow syntax. These are untrained, but you can already see two different patterns from the same input.

head 1: attention weights6 × 6
thecatsatonthemat
head 2: attention weights6 × 6
thecatsatonthemat

Uses the sentence and settings chosen above. Row = the word looking, column = the word looked at.

The full picture

The whole model: encoder and decoder

The paper built the Transformer for translation. The encoder reads the source sentence; the decoder writes the translation one token at a time, each step reading both what it has written so far and the encoder's output. Select a box to see what it does.

encoder layer × 6decoder layer × 6Source tokens + positionsSelf-attention (+ Add & Norm)Feed-forward (+ Add & Norm)Target tokens (shifted right)Masked self-attentionEncoder–decoder attentionFeed-forward (+ Add & Norm)Linear + softmax → next tokenkeys & values“Das Haus ist …” so far“The house is small”Select any box

Encoder–decoder attention

Queries come from the decoder; keys and values come from the encoder's output. This is where each word being written looks back at the source sentence: the translation step.

From the paper

The paper's numbers

The sizes from Table 3 of the paper, with parameters counted from the formulas on this page. The count lands close to what the paper reports; the exact figure depends on details such as the vocabulary size (about 37,000 shared English–German tokens).

Transformer (base)Transformer (big)this page
layers (encoder, decoder)6 + 66 + 61
d_model (numbers per token)51210248
heads8162
d_k = d_model / heads64644
d_ff (feed-forward width)2048409632
parameters, counted63.1M214.2M–
parameters, paper65M213M–
BLEU, English→German27.328.4–

Where the base model's parameters are:

shared embeddings18.9M
6 encoder layers18.9M
6 decoder layers25.2M

Inside each encoder layer, the feed-forward network (2.1M) holds twice the weights of attention (1.1M).

The learning-rate schedule (Equation 3)

0102030405000.20.40.6warm-up ends, step 4,000

Across: training step, in thousands. Up: learning rate, in thousandths (0.6 = 0.0006).

lr = d_model^−0.5 · min(step^−0.5, step · 4000^−1.5)

It rises in a straight line for 4,000 steps, then falls as 1/√step. Starting small keeps the early, random updates from wrecking the model. The paper trained with Adam (β₁ = 0.9, β₂ = 0.98, ε = 10⁻⁹), dropout 0.1 and label smoothing 0.1; the base model trained for 100,000 steps, about 12 hours on 8 P100 GPUs.

No magic, just arithmetic

The maths, step by step

Follow the word “cat” (position 1, counting from 0 as the paper does) through the layer, on the sentence “the cat sat”. Every number below comes from the same code as the stepper above.

  1. 1

    Positional encoding

    Even columns use sine, odd columns cosine, each pair with a longer wavelength than the last.

    PE(1, 0) = sin(pos / 10000^(0/8)) = sin(1) = 0.8415

    PE(1, 1) = cos(pos / 10000^(0/8)) = cos(1) = 0.5403

    PE(1, 2) = sin(pos / 10000^(2/8)) = sin(1 / 10) = 0.0998

    Position 0 is always [0, 1, 0, 1, …]. Nearby positions get similar codes; far-apart ones differ more.

  2. 2

    One attention score

    How well does “cat”'s query match “the”'s key? A dot product, then divide by √d_k.

    q = [-1.0941, -1.0302, 0.0658, -0.4107]

    k = [1.0544, -0.0962, 0.7606, 0.7286]

    q · k = Σ qᵢ kᵢ = -1.3037

    score = q · k ÷ √4 = -1.3037 ÷ 2 = -0.6519

    Why divide? With d_k numbers multiplied and added, the dot product grows with d_k, and large scores push softmax into putting almost all the weight on one word, where its gradients vanish.

  3. 3

    Softmax over the row

    Exponentiate every score in the row and divide by the total, so the row adds up to 1.

    scores for “cat”: [-0.6519, 0.3256, -0.3970]

    weight → “the” = e^score ÷ Σ e^score = 0.2021

    weight → “cat” = e^score ÷ Σ e^score = 0.5371

    weight → “sat” = e^score ÷ Σ e^score = 0.2608

  4. 4

    Add & Norm on one row

    Add the input back, then subtract the row's mean and divide by its standard deviation.

    mean = Σ ÷ 8 = 0.3546

    variance = Σ (v − mean)² ÷ 8 = 1.1444

    first value = (v − mean) ÷ √(variance + ε) = (1.3226 − 0.3546) ÷ 1.0697 = 0.9049

    A trained model then multiplies by a learned γ and adds a learned β for each column. They start at 1 and 0, which is what this untrained layer uses.

  5. 5

    The feed-forward network

    Two matrix multiplications with a ReLU between them, on this word alone.

    8 numbers → × W₁ (8 × 32) + b₁ → 32 numbers → ReLU → 21 of them become 0 → × W₂ (32 × 8) + b₂ → 8 numbers

    Then Add & Norm once more. The result for “cat”: [0.90, -0.40, -0.12, 1.57, -2.05, 0.47, -0.37, -0.01].

What happened next

Then and now

The layer on this page is still the core of today's language models, with a few common changes.

Decoder-only

GPT-style models drop the encoder and cross-attention, keeping masked self-attention: every model that writes text one token at a time.

Encoder-only

BERT-style models keep only the encoder, reading the whole input in both directions. Common for search, classification and embeddings.

Pre-norm

Many later models normalise before each sub-layer, x + sublayer(norm(x)), instead of after. It tends to make deep stacks easier to train.

Other position schemes

The fixed sine waves were largely replaced by learned positions or by rotary embeddings (RoPE), which build position into the query–key comparison.

Longer contexts

Attention compares every pair of tokens, so its cost grows with the square of the length. Much later work aims to make long inputs cheaper.

Same building blocks

Tokens, embeddings, attention, feed-forward, residuals and normalisation: each has its own page in this section. Tokenization · Self-Attention · Sampling

How it works: The Transformer

Step by step

  • Positive value
  • Negative value
  • Attention weight
  • Add & Norm / residual

Press Start or Step: the step the animation is on lights up below.

  1. 1

    Tokens in

    The sentence arrives as token ids, one row per token from here on.

    ids = tokenizer(text)
  2. 2

    Embed

    Look up each token's vector and multiply by √d_model.

    x = E[ids] * sqrt(d_model)
  3. 3

    Add positions

    Add a fixed sine/cosine pattern so the model can tell positions apart.

    x = x + PE[:n]
  4. 4

    Project Q, K, V

    Each head multiplies x by its own three matrices.

    Q, K, V = x Wq, x Wk, x Wv
  5. 5

    Score and mask

    Every query against every key, divided by √d_k. A causal mask sets future positions to −∞.

    S = Q Kᵀ / sqrt(d_k) + mask
  6. 6

    Softmax

    Each row becomes attention weights that add up to 1.

    A = softmax(S)
  7. 7

    Mix the values

    Each word's output is its weighted average of the value rows.

    head = A V
  8. 8

    Concat heads, W_O

    Heads side by side, then one projection back to d_model.

    attn = concat(heads) Wo
  9. 9

    Add & Norm

    Residual connection, then layer normalisation on each row.

    a = LayerNorm(x + attn)
  10. 10

    Feed-forward

    Widen, ReLU, shrink, on each word separately.

    f = max(0, a W1 + b1) W2 + b2
  11. 11

    Add & Norm

    Same again. The output has the input's shape, so layers stack.

    out = LayerNorm(a + f)

“Attention Is All You Need” (Vaswani et al., 2017) replaced the recurrent networks used for translation with a model built from attention alone. Every word is processed at the same time, and attention lets any word draw on any other in a single step, which made training far more parallel.

A layer has two sub-layers. Multi-head self-attention lets words exchange information, several times in parallel with different learned projections. A feed-forward network then processes each word on its own. Each sub-layer is wrapped in a residual connection and layer normalisation, so the layer only has to learn a change to its input.

Attention ignores word order, so positional encodings are added to the embeddings first. The original model stacks 6 such layers in an encoder and 6 in a decoder; the decoder adds a causal mask and an attention step that reads the encoder's output.

Where you'll meet it

  • The architecture behind today's language models: GPT-style models keep only the decoder (masked self-attention), BERT-style models keep only the encoder.
  • Beyond text: Vision Transformers cut images into patches, and speech and protein models use the same layer.
  • Interview favourites: “why divide by √d_k?”, “why multiple heads?”, “what are positional encodings for?”, “what does the causal mask do?”.
time O(n² · d + n · d²) per layerspace O(n² + n · d)

The attention scores compare every word with every other (n²), and the projections and feed-forward cost about n · d² for n words of width d. The n² term is why long contexts are expensive.