How LLMs Actually Generate Text: From Tokens to the Next Word

LLMs don't think in words. I break down what happens between your prompt and the next token: tokenization, embeddings, attention, logits, and sampling. With the math, tables, and interactive demos, no math degree required.

How LLMs Actually Generate Text: From Tokens to the Next Word

Introduction

I used to assume LLMs planned their sentences. Somewhere inside the model, I figured, there was a little writer who decided what the paragraph should say and then typed it out.

Wrong. An LLM does exactly one thing. Given some text, it predicts the next token. Then it appends that token and asks the same question again. Over and over, until a stop condition fires.

Everything you have read from ChatGPT or Claude, the essays, the code, the weirdly confident tone, fell out of that loop. Nobody planned the paragraph. The paragraph is a few thousand coin flips with loaded dice.

The good news is that the whole pipeline is understandable without a math degree. There are five moving parts, and you can watch every one of them work in this post.

By the end you will know why models could never count the r's in "strawberry", why they hallucinate, and what temperature is really doing.

What a Token Actually Is

Models don't read text. They read integers.

Before your prompt reaches the model, a tokenizer splits it into chunks called tokens and maps each chunk to an ID from a fixed vocabulary. "The cat sat on the mat" becomes a handful of IDs, one per chunk. The IDs are arbitrary. The same phrase gets different IDs in different models' vocabularies.

What matters is that the vocab is finite. Every word you can type is either in that list or gets split into pieces that are.

The tokenizer, in one code block

import tiktoken
 
enc = tiktoken.get_encoding("o200k_base")  # GPT-4o's tokenizer
ids = enc.encode("The cat sat on the mat")
print(ids)
# a list of integers, one per token

Run it and you get a list of numbers. The model gets fed those numbers, not the string. Going the other way, enc.decode(ids) turns IDs back into text, which is how the model's output reaches your screen.

How big is a vocabulary?

Here are the GPT-2 family numbers, because they are public and well documented:

Two things to notice. The vocab stays fixed while the model grows, so bigger models read the same 50,257 chunks with more numbers behind each one. And the embedding dimension is not decorative, it is the width of the vector every token turns into, which matters when we get to attention.

How Tokenization Works

The tokenizer doesn't use a dictionary of whole words. Most models use byte pair encoding (BPE), which learns its merges from a training corpus.

Byte pair encoding, step by step

BPE starts with single characters. Then it repeatedly finds the most frequent adjacent pair in the corpus and merges it into a new token:

1. Count every adjacent character pair in the corpus
2. Merge the most frequent pair into a single token
3. Recount, repeat, until the vocab hits its target size

On English text, the early merges are predictable: "t" + "h" becomes "th", "th" + "e" becomes "the", and so on. Common words earn their own token. Rare words never do.

Why your prompts get chopped weirdly

The consequence is a vocab that is a frequency artifact, not a logic engine:

Names, URLs, and code get chopped into odd pieces because they are rare in the training corpus. This also makes the tokenizer inconsistent in ways that surprise people: "Unbelievable" might be two tokens while "unbelievably" is three, because each got merged separately during training.

Keep this in mind. It explains half of the weird LLM behaviors people complain about, and we will come back to it.

From Tokens to Vectors

An ID is just an address. The actual input to the model is the vector stored at that address in the embedding table.

The embedding table

The embedding table is a big matrix, one row per token in the vocab. Each row is a list of numbers, hundreds to thousands long, learned during training. GPT-2 small has 50,257 rows of 768 numbers each. That table alone is about 38 million parameters before the model has done anything else.

Why vectors and not IDs

Why bother with vectors? Because vectors can be similar or different, and IDs cannot. "Cat" and "dog" end up pointing in similar directions. "Cat" and "carburetor" don't. Training nudges these vectors so that tokens used in similar contexts land near each other.

That is the part people miss. The model doesn't know what a cat is. It knows what contexts "cat" appears in. That turns out to be enough to fake understanding most of the time, and it is also why the model gets things wrong in interesting ways.

Attention: Context Changes Everything

The same token means different things in different sentences. "Bank" in "river bank" is not "bank" in "bank account". A static vector can't capture that, because "bank" has exactly one row in the embedding table.

The attention layer fixes it. For each token, the model looks at the tokens before it and decides how much each one matters for the current position. Each token's vector gets blended with weighted amounts of the vectors before it, so the same word ends up with different representations in different contexts.

The math

This is the formula from the "Attention Is All You Need" paper:

In plain language:

The query of the current token gets compared against the key of every earlier token, the scores go through softmax, and the resulting weights blend the value vectors. That blend is the context-aware representation.

Watch it run

Here is a toy sentence where the disambiguation actually matters. Click a token to see what it attends to:

The interesting row is the last one. "bank" puts 45% of its attention on "cash" and 30% on "deposited", which is how the money meaning forms instead of the river meaning. In a real model those weights are learned, and every one of the model's layers runs this process once per token.

Early layers tend to capture low-level patterns like grammar. Later layers capture higher-level ones like topic and intent. Nobody designed that split. It fell out of training.

The Prediction: Logits and Softmax

After the last attention layer, the model holds one vector representing everything so far. The final step turns that vector into a prediction.

From vector to logits

A linear layer projects the context vector into logits: one raw score per token in the vocab. If the vocab has 100,000 tokens, you get 100,000 logits. Higher score means more likely.

The math: softmax

Softmax squashes those scores into a probability distribution that adds up to 1:

With temperature in the picture, the logits get divided first:

Worked example

import math
 
logits = [2.4, 1.1, 0.8, 0.7, 0.5, -0.2]  # scores for " mat", " couch", ...
 
def softmax(logits):
    m = max(logits)
    exps = [math.exp(l - m) for l in logits]
    total = sum(exps)
    return [e / total for e in exps]
 
probs = softmax(logits)
# roughly [0.53, 0.14, 0.11, 0.10, 0.08, 0.04]

After "The cat sat on the", " mat" wins with about 53%. " couch" gets 14%. " moon" gets 4%, because the model has read enough text to know that cats don't sit on moons very often.

One implementation detail worth knowing: the code subtracts the max logit before exponentiating. The math is identical, but it keeps e from overflowing into inf when logits get large. You will see this in every production softmax implementation.

That last example is the whole trick in miniature. The probabilities come from patterns in training data. Nothing looked up a fact. Nothing checked grammar. The model is doing statistics over everything it has ever read.

Sampling: How the Next Word Gets Picked

Here is where the API knobs enter. The probability distribution is not the answer. The model still has to pick one token from it, and how it picks changes the output.

The four strategies

Temperature

Greedy decoding always picks the top token. It's deterministic, and it's also repetitive: ask for three stories with the same prompt and you can get the same one three times.

Temperature divides the logits before softmax. Low values sharpen the distribution, so the top token wins more often. High values flatten it, so weird tokens get their chance. Using the worked example above:

Top-p

Top-p (nucleus sampling) cuts the tail. Keep the smallest set of tokens whose probabilities add up to p, renormalize, and sample from that set. With p = 0.9, tokens below the line get zero probability no matter what temperature did to them.

The difference from top-k is that top-p adapts. When the model is confident, the nucleus is small (maybe 3 tokens cover 90%). When the model is uncertain, the nucleus is large. Top-k cuts the same number of tokens every time regardless.

Try it

Here is a toy version you can play with. Six candidates, hardcoded logits, real softmax and sampling:

Slide temperature to 0.1 and watch the top candidate swallow the distribution. Slide it to 2 and " Bananas" becomes possible. That is the entire effect temperature has on a model. It never changes what the model knows. It only changes how the dice get rolled.

The Loop

One token is not an answer. The full generation loop:

def generate(prompt, max_tokens=200):
    tokens = tokenize(prompt)
    for _ in range(max_tokens):
        logits = model(tokens)          # one score per vocab entry
        probs = softmax(logits / temperature)
        probs = apply_top_p(probs)
        next_token = sample(probs)      # weighted random pick
        if next_token == STOP:
            break
        tokens.append(next_token)
    return detokenize(tokens)

The cost of a token

The output gets appended to the input and the whole thing runs again. This is why LLM APIs charge per token and why long generations take time: every new token requires a forward pass over the whole context.

The KV cache

The KV cache makes this less painful. It stores the attention results for old tokens, so each step only computes the new token's contribution. Without it, generation would redo the math for the entire prefix on every word. This is also why prompt length affects latency even before generation starts: the cache has to be filled for the whole prompt first.

Why the model can't revise

One more thing worth knowing: the model never goes back. Once " mat" is picked, it is locked in. If the sentence is heading somewhere dumb, the model can't revise it. It can only pick the best next token given the dumb sentence it just wrote. (This is why asking for an outline first and then the full text often works better. You are steering the dice rolls.)

Why This Explains the Weird Stuff

Why can't models count the r's in "strawberry"? Tokenization. The word gets split into chunks, and the chunks don't carry letter-level information. The model sees IDs and vectors, not spelling. For a long time models got this wrong constantly, and it was never a bug in the code. It was the tokenizer doing its job.

Why do LLMs hallucinate? The loop has no fact-check step. When the model doesn't know something, it doesn't stop or hedge. It picks a plausible next token and commits to it, then the next token continues the claim. A hallucination is the loop working exactly as designed, which is why you can't patch it away with a better prompt.

What does a confident answer mean? A steep probability distribution. The model found one continuation far more likely than the rest. Whether that continuation is true is a separate question the pipeline never asks.

Why does the same prompt give different answers? Sampling. Temperature 0 picks the top token every time. Anything above 0 rolls weighted dice.

Common Misconceptions

• The model plans the whole sentence first. It only ever picks one token at a time
• Assuming LLMs see letters. Tokens are chunks, and letter-level information gets lost in the split
• Trusting confidence. A confident answer just means the probability distribution is steep
• Thinking temperature changes what the model knows. It only changes how the next token gets picked
• Calling hallucinations a bug. They are the loop working exactly as designed
• Expecting the same answer twice. Sampling is random, so identical prompts can produce different replies

Mini FAQ

Q1. Do LLMs think in words?

No. They think in token IDs and vectors. Words only exist at the boundaries, in your input and the final output.

Q2. Why do LLMs fail at counting letters in a word?

Tokenization. Words get split into chunks that don't preserve spelling, and nothing in the ID-to-vector pipeline brings it back. The model never sees individual letters unless they happen to sit on chunk boundaries.

Q3. What is temperature, in one sentence?

A knob that reshapes the probability distribution before sampling. Low values sharpen it, high values flatten it, and zero means greedy decoding.

Q4. What is top-p?

Nucleus sampling. Keep the smallest set of tokens whose probabilities add up to p, then sample only from that set. It cuts off the long tail of improbable tokens.

Q5. Why do LLMs hallucinate?

Because the loop has no fact-check step. The model picks a likely-sounding continuation and commits to it. Confidence and truth are different things, and the pipeline only measures the first one.

Q6. Do I need to understand this to use LLMs?

No, but it makes every API parameter make sense. Temperature, top_p, max_tokens, and stop sequences all map to a step above. That is a better deal than memorizing what each knob "does".

My Honest Take

Understanding this pipeline changed how I use LLMs more than any prompt engineering trick did.

I stopped expecting the model to plan. When I want structure, I ask for an outline first, then the full version. When I want facts, I ask for sources and check them, because the model will happily invent convincing ones. When output feels too random, I reach for temperature before rephrasing the prompt five times.

I also stopped being impressed by the outputs in the mystical sense. Once you see the loop, "it wrote an essay" becomes "it rolled a few thousand weighted dice and this is what landed." The essays are still useful. I use them daily. I just stopped expecting the model to think.

Outro

The next time a model writes you something impressive, here is what happened: your text got sliced into tokens, the tokens became vectors, attention blended the context together, softmax produced a probability distribution over a hundred thousand options, and sampling picked one. Then the whole thing ran again for the next word.

That's it. That's the entire system. Five moving parts, running in a loop, at a scale nobody expected to work this well.

If this clicked, you'll probably enjoy How Modern Authentication Actually Works or Build Your Own MCP Server from Scratch.

Credible Sources

1. Attention Is All You Need. The transformer paper. Everything above builds on it.
2. OpenAI Tiktoken. The tokenizer behind GPT models. Tokenize your own prompts.
3. Andrej Karpathy's Zero to Hero. Builds GPT and the sampling loop from scratch, line by line.
4. Anthropic Messages API. Where temperature and top_p show up as real API parameters.

Subscribe to Our Newsletter