LLM Inference: How Language Models Generate Text

From token IDs to useful representations

Token IDs are labels, not measurements. The model converts each ID into a numerical vector (often called an embedding). It also attaches position information to each token so the model can tell the difference between "the first word" and "the same word later."

From there, the vectors move through a stack of transformer layers. Each layer updates the representation at each position using two core operations:

• Attention: lets a token pull relevant information from other tokens in the sequence.
• A feed-forward network: transforms the representation at each position independently.

Real models also include things like normalization and residual connections. Those details vary, but the big idea stays the same: the token's representation changes as it moves through the network. By the time the model produces outputs, those representations reflect the surrounding context - not just the original token ID.

From a distribution to the next token

After the forward pass, the model produces a score (a logit) for each token in its vocabulary. Those scores are turned into a probability distribution, and the system samples a next token.

Sampling isn't always "pick the highest probability." Model settings may use temperature, top-p, and similar controls. They change which candidates are considered and how likely different outputs are. They don't give the model a human-style plan - they just adjust the selection process.

Once the model selects a token, it appends that token to the input sequence and runs another forward pass to get the next token. This continues until a stopping condition is met, such as an end-of-turn token or a maximum output length.

In short, the loop looks like this:

1. Process the current token sequence.
2. Calculate probabilities for the next token.
3. Select one token and append it.
4. Repeat.

The loop is easy to describe. The cost is where things get interesting: it depends on what happens inside each pass and on how much context the model has to consider.

Context changes what a token means

Take the word bank in river bank and bank account. The visible word might land on the same token ID in both phrases. But the surrounding tokens differ, so attention lets the representation incorporate different context. Later layers can then treat the two uses differently.

That's one reason an LLM can respond to the same token differently depending on what's around it. The token doesn't carry one fixed meaning through the whole model. Its representation is recomputed in context.

There's also a boundary: typical decoder-only LLMs use a causal mask. A token can attend to itself and earlier tokens, but not future tokens that haven't been generated yet. In other words, during generation the model can't "look ahead" to the rest of the sentence. At each step, the available input is the prompt plus whatever tokens the model has already produced.

That one-way constraint is what keeps the next-token process consistent. It also means generation is sequential: the model must produce a token before it can condition the next token on it. Some work inside a step can be parallelized, but a standard autoregressive model generally can't generate the entire response in one shot.

Attention is powerful, but it isn't free. Longer sequences increase the amount of context the model must relate to itself, and that shows up in production - especially across the two inference stages.

Prefill processes the prompt

During prefill, the model processes the prompt that's already available. That may include the latest user message, system instructions, retrieved documents, and earlier conversation history. The prompt tokens can be processed in parallel via large matrix operations, subject to causal masking.

Prefill is often compute-heavy, particularly for long prompts. The model also computes and stores attention keys and values for the prompt tokens at each layer. Those stored values form the key-value cache - usually called the KV cache.

Decode generates the response

During decode, the model generates output one token at a time. For each new token, it computes that token's keys and values, then reuses cached keys and values from earlier positions. It doesn't need to redo all the attention work for the entire history every step.

Without caching, generating a response would require rebuilding a lot of intermediate information repeatedly. The KV cache reduces that repeated work, which is why it's a core part of practical LLM inference.

The trade-off is memory. The cache grows as the sequence grows, and each active request needs its own cache. A common estimate is:

2 × layers × KV heads × head dimension × sequence length × bytes per value

The factor of two accounts for keys and values. The rest depends on the model architecture, context length, and the numerical precision used for the cache.

For example, consider Llama 3.1 70B with 80 layers, 8 key-value heads, a head dimension of 128, and a 128K-token context. If values are stored with 2 bytes each, the estimate is roughly 43 billion bytes (about 43 GB in decimal units, around 40 GiB). That's based on the stated assumptions, not a universal number for every deployment. Actual memory use varies with implementation details, batching, cache formats, and runtime behavior.

That's a lot of GPU memory for a single request's cache. If you're serving many long-context requests at once, you also need to account for the cache for each active sequence, plus model weights and other overhead. This is one reason long context can get expensive - and why providers may price or limit it differently.

The engineering trade-off behind context length

A bigger context window can help when the task genuinely needs more material: a long contract, a large codebase, or a detailed conversation. But a larger limit doesn't automatically produce better answers. It increases memory pressure, can reduce throughput, and raises serving costs. If an application sends the same irrelevant history on every turn, it's paying for context that doesn't add much.

Teams reduce the pressure in different ways. Grouped-query attention lowers the number of key-value heads compared with standard multi-head attention. Quantization stores cache values with fewer bits. Offloading moves part of the cache out of GPU memory (often trading speed for capacity). Each approach shifts quality and performance in its own way - there isn't a free lunch here.

So the real question isn't just "how do we get the biggest context window?" It's "how much context does this job actually need, and what latency and cost are acceptable?"

Conversation history is not automatically memory

What users perceive as "memory" still needs careful design. In a typical stateless API call, the model doesn't keep a private, persistent memory of earlier requests. If the application wants the model to use earlier conversation details, it generally includes those details again in the new input - either as the full history or as a selected summary.

So the model can seem like it remembers because relevant text is present in the current context. That's different from the model independently storing something that persists between requests. Some products add a "memory" feature, but that's an application-level system: it may store facts, summaries, or preferences and then retrieve them later. The model uses whatever the system supplies in the next request.

This distinction matters. Re-sending the full conversation can increase token usage and cache requirements. Summarizing or retrieving only the relevant pieces can reduce that load - but it introduces failure modes too: summaries can omit constraints, and retrieval can miss details that would have mattered. The right choice depends on the cost of forgetting versus the cost of carrying everything forward.

There are also privacy and governance implications. If you store conversation summaries or user facts, you need retention policies, access controls, and deletion workflows. Calling the feature "AI memory" doesn't remove those operational responsibilities.

A simple design checklist

Before you raise context limits or add a more complex memory layer, ask:

• Does this task really need the full conversation, or just a few earlier facts?
• Can the application retrieve relevant documents instead of resending a large fixed bundle every time?
• What are the typical and worst-case input and output lengths?
• How does latency change under realistic concurrency?
• What user information is stored outside the model, for how long, and how can it be removed?
• Does extra context measurably improve task quality enough to justify the cost?

These questions shift the conversation from "model features" to system behavior. In real products, that's where the trade-offs usually show up.