How LLMs Actually Generate Text: From Tokens to the Next Word

LLMs don't think in words. I break down what happens between your prompt and the next token: tokenization, embeddings, attention, logits, and sampling. No math degree required, and no hype.

How LLMs Actually Generate Text: From Tokens to the Next Word

Introduction

I used to assume LLMs planned their sentences. Somewhere inside the model, I figured, there was a little writer who decided what the paragraph should say and then typed it out.

Wrong. An LLM does exactly one thing. Given some text, it predicts the next token. Then it appends that token and asks the same question again. Over and over, until a stop condition fires.

Everything you have read from ChatGPT or Claude, the essays, the code, the weirdly confident tone, fell out of that loop. Nobody planned the paragraph. The paragraph is a few thousand coin flips with loaded dice.

The good news is that the whole pipeline is understandable without a math degree. Tokenizer, embeddings, attention, softmax, sampling. Five moving parts, and you can watch every one of them work.

By the end of this post you will know why models could never count the r's in "strawberry", why they hallucinate, and what temperature is really doing.

What a Token Actually Is

Models don't read text. They read integers.

Before your prompt reaches the model, a tokenizer splits it into chunks called tokens and maps each chunk to an ID from a fixed vocabulary. "The cat sat on the mat" becomes a handful of IDs, one per chunk. The IDs are arbitrary. The same phrase gets different IDs in different models' vocabularies.

What matters is that the vocab is finite. Common models carry somewhere between 50,000 and 200,000 tokens. Every word you can type is either in that list or gets split into pieces that are.

import tiktoken
 
enc = tiktoken.get_encoding("o200k_base")  # GPT-4o's tokenizer
ids = enc.encode("The cat sat on the mat")
print(ids)
# a list of integers, one per token

Run it and you get a list of numbers. The model gets fed those numbers, not the string.

How Tokenization Works

The tokenizer doesn't use a dictionary of whole words. Most models use byte pair encoding (BPE), which learns its merges from a training corpus.

It starts with single characters. Then it repeatedly merges the most frequent pair into a new token: "t" + "h" becomes "th", "th" + "e" becomes "the", and so on until the vocab hits its target size.

The result is a vocab where common words get their own token and rare words get sliced. "The" is one token. "Antidisestablishmentarianism" is several. Names, URLs, and code tend to get chopped into odd pieces because they are rare in the training corpus.

This also makes the tokenizer inconsistent in ways that surprise people. "Unbelievable" might be two tokens while "unbelievably" is three, because each got merged separately during training. The vocab is a frequency artifact, not a logic engine.

From Tokens to Vectors

An ID is just an address. The actual input to the model is the vector stored at that address in the embedding table.

The embedding table is a big matrix, one row per token in the vocab. Each row is a list of numbers, hundreds to thousands long, learned during training.

Why vectors? Because vectors can be similar or different. "Cat" and "dog" end up pointing in similar directions. "Cat" and "carburetor" don't. Training nudges these vectors so that tokens used in similar contexts land near each other.

That is the part people miss. The model doesn't know what a cat is. It knows what contexts "cat" appears in. That turns out to be enough to fake understanding most of the time, and it is also why the model gets things wrong in interesting ways.

Attention: Context Changes Everything

The same token means different things in different sentences. "Bank" in "river bank" is not "bank" in "bank account". A static vector can't capture that.

The attention layer fixes it. For each token, the model looks at the tokens before it and decides how much each one matters for the current position. Each token's vector gets blended with weighted amounts of the vectors before it, so the same word ends up with different representations in different contexts.

This is the core trick from the "Attention Is All You Need" paper, and every modern LLM is a stack of these layers. Early layers tend to capture low-level patterns like grammar. Later layers capture higher-level ones like topic and intent. Nobody designed that split. It fell out of training.

The Prediction: Logits and Softmax

After the last attention layer, the model holds one vector representing everything so far. The final step turns that vector into a prediction.

First, a linear layer projects the vector into logits: one raw score per token in the vocab. If the vocab has 100,000 tokens, you get 100,000 logits. Higher score means more likely.

Then softmax squashes those scores into a probability distribution that adds up to 1:

import math
 
logits = [2.4, 1.1, 0.8, 0.7, 0.5, -0.2]  # scores for " mat", " couch", ...
 
def softmax(logits):
    m = max(logits)
    exps = [math.exp(l - m) for l in logits]
    total = sum(exps)
    return [e / total for e in exps]
 
probs = softmax(logits)
# roughly [0.53, 0.14, 0.11, 0.10, 0.08, 0.04]

After "The cat sat on the", " mat" wins with about 53%. " couch" gets 14%. " moon" gets 4%, because the model has read enough text to know that cats don't sit on moons very often.

That last sentence is the whole trick in miniature. The probabilities come from patterns in training data. Nothing looked up a fact. Nothing checked grammar. The model is doing statistics over everything it has ever read.

Sampling: How the Next Word Gets Picked

Here is where the API knobs enter. The probability distribution is not the answer. The model still has to pick one token from it, and how it picks changes the output.

Greedy decoding always picks the top token. It's deterministic, and it's also repetitive: ask for three stories with the same prompt and you can get the same one three times.

Temperature reshapes the distribution before sampling. You divide the logits by the temperature. Low values sharpen the distribution, so the top token wins more often. High values flatten it, so weird tokens get their chance. Temperature 0.5 takes " mat" from 53% to about 85%. Temperature 2 drops it to 33% and gives " moon" a 9% shot.

Top-p (nucleus sampling) cuts the tail. Keep the smallest set of tokens whose probabilities add up to p, renormalize, and sample from that set. With p = 0.9, tokens below the line get zero probability no matter what temperature did to them.

Here is a toy version you can play with. Six candidates, hardcoded logits, real softmax and sampling:

Slide temperature to 0.1 and watch the top candidate swallow the distribution. Slide it to 2 and " Bananas" becomes possible. That is the entire effect temperature has on a model. It never changes what the model knows. It only changes how the dice get rolled.

The Loop

One token is not an answer. The full generation loop:

def generate(prompt, max_tokens=200):
    tokens = tokenize(prompt)
    for _ in range(max_tokens):
        logits = model(tokens)          # one score per vocab entry
        probs = softmax(logits / temperature)
        probs = apply_top_p(probs)
        next_token = sample(probs)      # weighted random pick
        if next_token == STOP:
            break
        tokens.append(next_token)
    return detokenize(tokens)

The output gets appended to the input and the whole thing runs again. This is why LLM APIs charge per token and why long generations take time: every new token requires a forward pass over the whole context.

The KV cache makes this less painful. It stores the attention results for old tokens, so each step only computes the new token's contribution. Without it, generation would redo the math for the entire prefix on every word.

One more thing worth knowing: the model never goes back. Once " mat" is picked, it is locked in. If the sentence is heading somewhere dumb, the model can't revise it. It can only pick the best next token given the dumb sentence it just wrote. (This is why asking for an outline first and then the full text often works better. You are steering the dice rolls.)

Why This Explains the Weird Stuff

Why can't models count the r's in "strawberry"? Tokenization. The word gets split into chunks, and the chunks don't carry letter-level information. The model sees IDs and vectors, not spelling. For a long time models got this wrong constantly, and it was never a bug in the code. It was the tokenizer doing its job.

Why do LLMs hallucinate? The loop has no fact-check step. When the model doesn't know something, it doesn't stop or hedge. It picks a plausible next token and commits to it, then the next token continues the claim. A hallucination is the loop working exactly as designed, which is why you can't patch it away with a better prompt.

What does a confident answer mean? A steep probability distribution. The model found one continuation far more likely than the rest. Whether that continuation is true is a separate question the pipeline never asks.

Why does the same prompt give different answers? Sampling. Temperature 0 picks the top token every time. Anything above 0 rolls weighted dice.

Common Misconceptions

• The model plans the whole sentence first. It only ever picks one token at a time
• Assuming LLMs see letters. Tokens are chunks, and letter-level information gets lost in the split
• Trusting confidence. A confident answer just means the probability distribution is steep
• Thinking temperature changes what the model knows. It only changes how the next token gets picked
• Calling hallucinations a bug. They are the loop working exactly as designed
• Expecting the same answer twice. Sampling is random, so identical prompts can produce different replies

Mini FAQ

Q1. Do LLMs think in words?

No. They think in token IDs and vectors. Words only exist at the boundaries, in your input and the final output.

Q2. Why do LLMs fail at counting letters in a word?

Tokenization. Words get split into chunks that don't preserve spelling, and nothing in the ID-to-vector pipeline brings it back. The model never sees individual letters unless they happen to sit on chunk boundaries.

Q3. What is temperature, in one sentence?

A knob that reshapes the probability distribution before sampling. Low values sharpen it, high values flatten it, and zero means greedy decoding.

Q4. What is top-p?

Nucleus sampling. Keep the smallest set of tokens whose probabilities add up to p, then sample only from that set. It cuts off the long tail of improbable tokens.

Q5. Why do LLMs hallucinate?

Because the loop has no fact-check step. The model picks a likely-sounding continuation and commits to it. Confidence and truth are different things, and the pipeline only measures the first one.

My Honest Take

Understanding this pipeline changed how I use LLMs more than any prompt engineering trick did.

I stopped expecting the model to plan. When I want structure, I ask for an outline first, then the full version. When I want facts, I ask for sources and check them, because the model will happily invent convincing ones. When output feels too random, I reach for temperature before rephrasing the prompt five times.

I also stopped being impressed by the outputs in the mystical sense. Once you see the loop, "it wrote an essay" becomes "it rolled a few thousand weighted dice and this is what landed." The essays are still useful. I use them daily. I just stopped expecting the model to think.

Outro

The next time a model writes you something impressive, here is what happened: your text got sliced into tokens, the tokens became vectors, attention blended the context together, softmax produced a probability distribution over a hundred thousand options, and sampling picked one. Then the whole thing ran again for the next word.

That's it. That's the entire system. Five moving parts, running in a loop, at a scale nobody expected to work this well.

If this clicked, you'll probably enjoy How Modern Authentication Actually Works or Build Your Own MCP Server from Scratch.

Credible Sources

1. Attention Is All You Need. The transformer paper. Everything above builds on it.
2. OpenAI Tiktoken. The tokenizer behind GPT models. Tokenize your own prompts.
3. Andrej Karpathy's Zero to Hero. Builds GPT and the sampling loop from scratch, line by line.
4. Anthropic Messages API. Where temperature and top_p show up as real API parameters.

Subscribe to Our Newsletter