Tokenization

Train a byte-pair-encoding tokenizer merge by merge, then encode your own text into token ids. See why rare words split and why other languages cost more tokens.

“the cats sat”the29cat33s19sat32tokens ↑ · ids ↓ · what the model actually gets

1Models read numbers, not letters

Before a language model sees your text, a tokenizer cuts it into pieces called tokens and swaps each one for its number in a fixed vocabulary. The ids here come from this page's own tokenizer.

letters · 10thecatssatwords · 3 (but “cats” needs its own entry)thecatssatsubwords · 4thecatssat

2Three ways to cut text

Letters need only a tiny vocabulary but make text very long. Whole words make it short but need a huge vocabulary, and any new word is unknown. Subwords sit in between: common words stay whole, rare ones split into familiar parts.

pair countse s9s t9l o7newestnewestnewest

3Merge the commonest pair

Byte-pair encoding learns the pieces from data. Count every pair of neighbouring tokens, glue the commonest pair into one new token, and repeat. Each merge is one new vocabulary entry.

Watch it learn

Train a tokenizer, one merge at a time

Byte-pair encoding starts from single letters. Each round it counts every pair of neighbouring tokens (weighted by how often each word occurs), then glues the commonest pair into one new token. Step through and watch the words regroup and the vocabulary grow.

The words, as letters

79 tokens in the text

low
×5
lower
×2
newest
×6
widest
×3
Training text
8

The example from the BPE paper: four words with counts. Watch “est” form first, because it's shared by the two commonest words.

Step 0 / 18
1100 ms

Press Step to count the pairs for the first merge, or Start to watch it run.

Pair counts · round 1

  • e + s9
  • s + t9
  • w + e8
  • l + o7
  • o + w7
  • e + w6
  • n + e6
Vocabulary size
27
Merges learned
0

Learned tokens, in order (ids start at 27: 0 is [UNK], 1–26 are the letters):

none yet

Use it

Encode your own text

This tokenizer learned its merges from a short English paragraph (open it below). Type anything: each word starts as letters and the learned merges are replayed, earliest first. The small number under each token is its id: what a model would actually receive. Click a word to see its merges one by one.

60
The training paragraph

the dog sat on the mat and the cat sat on the hat. the dog and the cat are friends. the man and the woman went to the house. they sat at the table and ate the bread. the children played in the garden with the dog and the cat. the sun was warm and the water was cold. she said that the weather was better than the weather in the winter. they went to the market and the store.

Letters and symbols
26
Tokens
12

2.2 characters per token.

Click a word on the left to replay its merges.

The trade-off

Shorter text, bigger vocabulary

Every merge adds one token to the vocabulary and makes the training text shorter. Short sequences are cheaper for a model to read, but every vocabulary entry needs its own learned embedding. This paragraph runs out of pairs after 90 merges, when every word is a single token. Real models stop far earlier relative to their data, at vocabularies of tens of thousands to a few hundred thousand tokens.

0204060800100200
training text, in tokens vocabulary sizeacross: merges · amber line: the encoder's setting above (60)

Not every language costs the same

A tokenizer trained on English, reading German

The same tokenizer, the same meaning, two languages. Its merges came from English, so English words come out whole while German words fall apart into small pieces. A model pays for every token, in time, money and context space.

English

10 tokens

“the dog and the cat sat in the garden”

thedogandthecatsatinthegarden

German

26 tokens

“der hund und die katze sassen im garten”

derhundunddiekatzesassenimgarten

Uses the encoder's merge setting (60). Real tokenizers are trained on many languages, so the gap is smaller, but languages that are rare in the training data still tend to need more tokens for the same sentence.

No magic, just counting

The maths, step by step

The example from the BPE paper: “low” 5 times, “lower” twice, “newest” 6 times, “widest” 3 times. Two rounds of training by hand, then the learned merges cut up a word the tokenizer never saw. Every number comes from the same code as the stepper above.

  1. 1

    Start from letters

    Every word becomes a list of single letters. The counts say how often each word occurs.

    low ×5 → l o w

    lower ×2 → l o w e r

    newest ×6 → n e w e s t

    widest ×3 → w i d e s t

  2. 2

    Round 1: count every neighbouring pair

    A pair inside a word counts once per occurrence of that word.

    “e s”: newest ×6 + widest ×3 = 9

    “s t”: newest ×6 + widest ×3 = 9

    “w e”: lower ×2 + newest ×6 = 8

    “l o”: low ×5 + lower ×2 = 7

    “o w”: low ×5 + lower ×2 = 7

    “e s” and “s t” tie at 9. Ties go to the alphabetically first pair, so “e s” wins and becomes the token “es”.

  3. 3

    Merge, then count again

    Replace every “e s” with “es”. The pairs change, so count from scratch.

    low → l o w

    lower → l o w e r

    newest → n e w es t

    widest → w i d es t

    Round 2's winner: “es t”: newest ×6 + widest ×3 = 9

  4. 4

    Keep going

    Two more rounds give “lo” and then “low”. The merges, in order, are the whole tokenizer.

    merge 1: e + s → es (seen 9 times)

    merge 2: es + t → est (seen 9 times)

    merge 3: l + o → lo (seen 7 times)

    merge 4: lo + w → low (seen 7 times)

    Words now: low · low e r · n e w est · w i d est

  5. 5

    Encode a new word: “lowest”

    It never appeared in training. Start from letters and apply the earliest-learned merge that fits, again and again.

    l o w e s t

    merge 1 (e + s) → l o w es t

    merge 2 (es + t) → l o w est

    merge 3 (l + o) → lo w est

    merge 4 (lo + w) → low est

    “lowest” = “low” + “est”: built from pieces learned from other words.

Why it matters

Odd model behaviour that tokens explain

A language model never sees your letters, only token ids. That one fact explains several surprises.

Counting letters

Ask how many r's are in “strawberry” and the model sees a few tokens, not ten letters. Spelling and letter-counting are harder than they look.

Arithmetic on long numbers

Numbers are cut into chunks that don't line up with place value, so “carry the one” has no clean place to happen.

Spaces and capitals

In real tokenizers, “cat”, “Cat” and “ cat” (with a leading space) are usually three different tokens with three different ids.

Cost and context

Prices, rate limits and context windows are counted in tokens. A rough rule of thumb for English is about 4 characters per token.

Other languages

Text in languages that were rare in the tokenizer's training data splits into more tokens, so it costs more and fits less.

What comes next

Each token id is looked up in an embedding table to get a vector. Those vectors are what self-attention mixes. Open Self-Attention →

How it works: Tokenization (BPE)

Step by step

  • Single letter
  • The pair being merged
  • Learned tokens (each keeps its own colour)

Press Start or Step: the step the animation is on lights up below.

  1. 1

    Start from characters

    Every word becomes a list of single characters. The vocabulary is just the base alphabet.

    words = {tuple(w): count}
  2. 2

    Count neighbouring pairs

    Across all words, weighted by how often each word occurs. The most frequent pair wins; ties are broken by a fixed rule.

    best = argmax count(a, b)
  3. 3

    Merge the winner

    Replace every occurrence of the pair with one new token and add it to the vocabulary. The text gets shorter.

    tokens = merge(tokens, best)
  4. 4

    Stop at the vocabulary size

    After the chosen number of merges, or when no pairs are left. The ordered list of merges is the tokenizer.

    return merges

A language model works on numbers, so text is first cut into tokens, and each token is replaced by its id in a fixed vocabulary. Letters would make text very long; whole words would need an enormous vocabulary and still meet unknown words. Subword tokens sit in between.

Byte-pair encoding learns those subwords from data. Start from single characters, count every pair of neighbouring tokens across the training text, merge the most frequent pair into one new token, and repeat. Common words end up as single tokens; rare words split into familiar pieces.

To tokenize new text, replay the merges: start from characters and apply the earliest-learned merge that fits, until none do. Real tokenizers (GPT-2 onwards) start from the 256 byte values instead of letters, so every possible string can be encoded and nothing is ever unknown.

Where you'll meet it

  • Every large language model: prices, context windows and rate limits are all counted in tokens, not words.
  • Explaining odd model behaviour: counting letters, spelling, arithmetic on long numbers, and text in under-represented languages.
  • Interview favourites: “why BPE instead of words?”, “what happens to a word the tokenizer never saw?”, “why does the same sentence cost more tokens in another language?”.
time O(merges × corpus size) to trainspace O(vocabulary + corpus)

Each round recounts every pair in the corpus and rewrites the words. Production trainers update the counts incrementally instead of recounting, which is much faster on real corpora.