Explore LLM token sampling with interactive temperature, top-k and top-p controls. Follow logits, softmax probabilities, filtering and the final token draw.
1The model scores every next word
It doesn't know the next word. It ranks every possible one, and softmax turns those scores into probabilities.
2Temperature: focused or adventurous
Low temperature makes the favourite even more likely. High temperature flattens the odds, so rarer words get picked more.
3Cut the tail, then roll the dice
Top-p keeps the likeliest words until they cover, say, 90%. The rest are dropped, then one word is picked at random by its share.
The capital of France is ?
Final probabilities
- Paris95%
- a2.1%
- located1.3%
- the1.0%
- Lyon0.2%
- beautiful0.2%
- not0.1%
- Berlin0.0%
Draws will appear here
Low: focused. High: adventurous.
Keep only the k likeliest. 0 = off.
Keep words until they cover p. 1 = off.
Move the sliders and watch the bars change, or press Step to walk through how one word gets picked.
How it works: Next-Token Sampling
Step by step
- Probability
- Filtered out
- Picked word
Press Start or Step: the step the animation is on lights up below.
- 1
Score every candidate
The model's final layer outputs one raw score per token.
logits = model(prompt) - 2
Apply temperature
Divide every score by T. Small T stretches the gaps, large T squeezes them.
logits = logits / T - 3
Softmax to probabilities
Exponentiate and normalise, so every probability is positive and they add up to 1.
p = exp(logits) / Σ exp(logits) - 4
Top-k: keep the k likeliest
Everything below the k-th word is dropped. Off by default.
keep = top k of p - 5
Top-p: keep until the total reaches p
Take words from most to least likely until their probabilities add up to p. Also called nucleus sampling.
keep = smallest set with Σp ≥ P - 6
Rescale what's left
Divide the surviving probabilities by their total so they add up to 1 again.
p = p[keep] / Σ p[keep] - 7
Roll the dice
Pick one word at random, weighted by its probability. Then append it to the text and do it all again for the next word.
next = random.choice(p)
A language model writes one token (roughly, one word or word-piece) at a time. For each step it gives every token in its vocabulary a score, called a logit. It doesn't “know” the next word: it has a ranking of how likely each one is.
Softmax turns the scores into probabilities. Temperature reshapes them first: below 1 the favourite gets even more likely (focused, repetitive), above 1 the long tail gets a bigger share (varied, sometimes odd).
Top-k and top-p cut the unlikely tail before picking, so a high temperature can't pick nonsense. Top-k keeps a fixed number of words; top-p keeps however many it takes to cover p of the probability. Then the model rolls the dice over what's left.
The candidate words and their scores here are written by hand for each prompt; a real model scores tens of thousands of tokens. Everything after the scores works exactly as in real decoders.
Where you'll meet it
- The temperature, top_k and top_p settings in every LLM API and chat playground.
- Low temperature (0–0.3) for code and facts; around 0.7–1 for writing; higher only for brainstorming.
- Interview questions: “why does the same prompt give different answers?”, “what does temperature 0 do?”, “top-k vs top-p?”.
V is the vocabulary size. Softmax is O(V); top-k and top-p need the probabilities in order, which is a sort (or a partial sort for top-k).