How Generative AI Works
Summary
Unveiling GenAI showed the full system architecture. What Is Generative AI? covered what models are and how they are trained. This page covers what happens at inference time — the moment you call the model. Tokenisation, the context window, the generation loop, and decoding strategies. These are the mechanics that determine cost, latency, and output quality.
Most engineers interact with language models as a black box: prompt in, text out. That works until something breaks. When a response is slow, inconsistent, truncated, or confidently wrong — the cause almost always lives in one of four places: how the prompt was tokenised, what fit inside the context window, how the generation loop ran, or which decoding strategy was applied.
Understanding these four mechanics gives you the tools to diagnose failures, tune quality, and predict behaviour before it reaches production.
| Mechanic | What it controls | Practical consequence |
|---|---|---|
| 🔤 Tokenisation | How text is split into token IDs and counted | Directly impacts billable tokens and fit. Example: adding 10 large log chunks can double token spend and force truncation of useful instructions. |
| 📦 Context window | How total token budget is allocated per request | Forces trade-offs between instructions, retrieval, history, and response length. Example: if retrieval consumes most tokens, conversation history may be dropped and answers lose continuity. |
| ⚡ Generation loop | How each output token is computed and streamed | Defines perceived responsiveness and completion behavior. Example: streaming can show first tokens in under a second, while missing stop sequences can produce overlong or runaway outputs. |
| 🎲 Decoding strategy | How token probabilities are filtered and sampled | Sets the stability-creativity balance. Example: lower temperature for claims decisions improves consistency, while higher temperature for marketing copy improves variety. |
A team building a contract review assistant noticed that responses were randomly inconsistent — same contract, same question, different answers on consecutive calls. Temperature was set to 0.9 by default in their SDK wrapper. Lowering it to 0.1 made responses deterministic and reproducible. The model had not changed. The decoding setting had.
The model never reads words. It reads token IDs — integers representing subword units. A word like "tokenization" may be 3–4 tokens. Cost, context window size, and latency are all measured in tokens, not words. Prompt design that ignores this burns context budget and increases costs at scale.
System instruction, retrieved documents, conversation history, and the current user query all share one token budget. When that budget fills, the oldest content is cut — silently. A 200K-token window sounds large until you count retrieved PDFs, multi-turn history, and a detailed system prompt all competing for the same space.
Temperature controls how the model samples the next token. Set too high, the model produces fluent variation — which looks creative in a demo and looks inconsistent in a compliance review. Enterprise applications that require reproducible, policy-anchored answers should run at temperature 0.1–0.2, not the default 0.7–1.0.
Why This Matters
The mechanics of inference directly determine whether your system is predictable in production — not just whether it works in a demo.
A difference of 1,000 tokens per call is invisible in a test environment. At 50,000 calls per day, it is 50 million tokens per day — the difference between a manageable inference budget and a cost overrun that appears in month-two financial reviews. Retrieval systems that inject 20 full-length documents into the context window when 3 targeted chunks would suffice are not just slow — they are expensive by design.
Research consistently shows that LLMs recall information near the beginning and end of long contexts more reliably than information buried in the middle — a phenomenon known as "lost in the middle." A RAG system that dumps 30 retrieved chunks in sequential order and puts the most relevant chunk at position 15 is not using the context window efficiently. Reranking and strategic placement matter as much as retrieval quality.
The default temperature in most SDKs is 0.7–1.0 — appropriate for creative generation where variety is desirable. For enterprise applications generating claim summaries, compliance drafts, or policy explanations, that same setting produces answers that vary on consecutive calls with identical inputs. One team auditing their customer-facing assistant found 12% of responses contained inconsistent policy language — tracing directly to temperature configuration, not model quality.
The model's capability is fixed once training ends. Everything you control at inference time — prompt assembly, retrieval strategy, temperature, top-p, max tokens — determines what fraction of that capability you actually capture in production.
Core Concepts
🔤 Tokenisation — Text Becomes Numbers
The model does not process words, characters, or sentences. It processes tokens — integer IDs representing subword units from a fixed vocabulary of roughly 100,000 entries. A tokeniser converts your text to this sequence of IDs before any model computation begins.
What this means in practice:
| Input Type | Example | Approximate Token Count | Why it matters |
|---|---|---|---|
| Simple Sentence | "Hello, how are you?" | ~5 tokens | Roughly 0.75 tokens per English word on average |
| Technical Term | "tokenization" | 3–4 tokens | Technical terms split into subwords — less efficient |
| Standard Text | A 1-page English document (~500 words) | ~650–700 tokens | Plan for 1.3–1.5× words when estimating context usage |
| Structured Data | A dense JSON or code block (~500 chars) | ~150–250 tokens | Code and syntax characters have specific tokenization patterns |
| Rich Document | A retrieved PDF page | ~800–1,200 tokens | Varies by formatting, whitespace, and layout |
Why tokenisation determines cost: Every API call is billed by input tokens (your prompt) plus output tokens (the generated response). A system that injects 3 full document pages when 2 targeted paragraphs would suffice pays 3–5× more per call — at every call, at every scale.
Why tokenisation determines context fit: The context window limit is in tokens. "200,000 tokens" sounds like a lot until you account for the system prompt (500–2,000 tokens), retrieved documents (5,000–20,000 tokens), conversation history (1,000–10,000 tokens), and the generated response reservation. Budget carefully.
📦 The Context Window — What the Model Can See
The context window is the total token budget for a single inference call. Everything the model processes and generates must fit within it.
What competes for context window space in a production RAG system:
| Component | Typical token range | Notes |
|---|---|---|
| System instruction | 200–1,000 tokens | Role, scope, output format, hard rules |
| Retrieved documents | 2,000–20,000 tokens | The biggest variable — drives most overruns |
| Conversation history | 500–8,000 tokens | Grows with turn count in multi-turn sessions |
| User query | 20–500 tokens | Usually small, but can be large for document upload |
| Response reservation | 500–2,000 tokens | Space held for the model's output |
Current context window sizes (2026):
- GPT-5.4: 128K tokens
- Claude Sonnet 4.6: 200K tokens
- Gemini 2.5 Pro: 1M tokens
Larger is not always better. Attention quality degrades at extreme context lengths for some tasks. Reranking retrieved content and injecting only the most relevant chunks produces more reliable outputs than filling the window with everything available.
When the window fills: Most APIs handle overflow by truncating from the oldest content. This happens silently — there is no error. A multi-turn support session where conversation history is not managed can gradually push the system instruction out of the window, causing the model to behave as if the rules were never set.
⚡ The Generation Loop — One Token at a Time
Language models are autoregressive: they generate one token at a time, each new token conditioned on all previous tokens in the context. This is the generation loop.
All four context components are assembled and tokenised together
The model does not receive the system instruction, documents, and query separately. They are concatenated into a single token sequence before the forward pass. The model has no structural awareness of which tokens came from which component — it sees one long sequence. This is why formatting and placement inside the prompt influence how reliably the model follows instructions.
The forward pass produces a probability distribution, not an answer
The transformer layers process the full token sequence and output a vector of logit scores — one score per token in the vocabulary (roughly 100,000 entries). A softmax function converts those scores into probabilities. At this point, the model has not yet "decided" what to say. It has computed how likely each possible next token is given the full context it received.
The decoding strategy selects one token from the distribution
Temperature and top-p are applied to the probability distribution before sampling. Temperature shrinks or expands the distribution. Top-p restricts which tokens are eligible to be sampled. One token is selected, appended to the growing output sequence, and the loop restarts — the new token becomes part of the context for the next forward pass.
The loop ends at a stop token or max_tokens — not at a sentence boundary
Generation halts when the model produces a designated stop token (like the end-of-sequence marker) or when the output hits the max_tokens limit. If max_tokens is set too low, responses are truncated mid-sentence with no warning. If it is not set, the model may run until it fills its output budget — which adds latency and cost on every call, not just the long ones.
🎲 Decoding Strategies — Temperature and Top-p
These two parameters are the primary controls over output variation. Both are applied at the decoding step — after the forward pass, before the token is selected.
Temperature divides the logit scores before softmax is applied:
| Temperature | Effect | Use case |
|---|---|---|
0.0 | Always selects the highest-probability token — fully deterministic | Automated classification, structured output, test assertions |
0.1–0.2 | Near-deterministic — slight variation, consistent policy language | Enterprise Q&A, claim summaries, compliance drafts |
0.5–0.7 | Balanced variation — coherent but not identical on repeats | Internal tools, moderate-stakes content |
0.9–1.0 | High variation — creative and diverse, inconsistent on repeats | Brainstorming, content generation, creative writing |
Top-p (nucleus sampling) restricts which tokens are eligible to be sampled. At top-p=0.9, the model samples only from the smallest set of tokens whose cumulative probability sums to 0.9 or more — eliminating very low-probability tokens that would produce unexpected or incoherent output. Most production deployments use top-p=0.9 alongside a low temperature.
The practical rule: set temperature first based on how much variation your use case tolerates. For anything customer-facing in a regulated environment, start at 0.1 and raise only if responses feel robotic. The default (0.7–1.0) is designed for demos — it makes responses feel natural and varied, which looks good on first impression and creates compliance problems on the hundredth.
Manager vs Engineer
👔 Manager Inference costs doubled between the pilot and production despite using the same model. What happened?
💻 Engineer Almost certainly prompt size. During the pilot, the application was likely calling the model with short, hand-crafted prompts. In production, the retrieval system often injects retrieved documents — potentially full pages rather than targeted chunks — alongside conversation history from multi-turn sessions and comprehensive system instructions. Each of these additions increases the token count. At 10,000 calls per day, a prompt growing from 800 to 3,000 tokens results in an extra 22 million input tokens daily. The recommended approach is to compare the average input token count per call in production against the pilot phase.
👔 Manager Can the context window simply be increased to avoid truncation issues?
💻 Engineer It is possible, but not without trade-offs. Larger context windows increase the cost per call and can introduce a different quality issue — the model's recall degrades for content buried in the middle of a very long context. A more effective solution is context management: summarizing older conversation turns rather than appending them indefinitely, and utilizing a reranker to inject only the 3–5 most relevant retrieved chunks instead of the top 20. This keeps the prompt well under the window limit while improving answer quality, because the model processes tighter, higher-relevance evidence rather than a long, noisy context.
👔 Manager Why do responses occasionally cut off mid-sentence?
💻 Engineer The max_tokens limit is likely set too low. The generation loop halts immediately upon hitting that limit — without warning or sentence boundary checks. If the system prompt is longer than expected, or if the model starts with excessive context, the output budget is exhausted before the response completes. The solution is to either increase the max_tokens threshold or restructure the prompt to consume fewer input tokens, leaving more capacity for the output. It is also advisable to verify whether a hard maximum is enforced at the API level, separate from the application code settings.
👔 Manager A customer-facing claims summary feature is about to go live. What temperature setting is appropriate?
💻 Engineer A temperature of 0.1 is recommended. Claims summaries are policy-anchored, requiring specific and deterministic answers. A setting of 0.1 ensures the model selects the highest-probability continuation almost consistently, meaning identical claims yield identical language. Higher temperatures introduce variation that may sound natural but appears inconsistent during compliance audits — describing the same policy clause differently across calls raises questions about which version is authoritative. The standard configuration is to set temperature to 0.1, top-p to 0.9, and include a requirement for citations in the output template to ensure every statement is grounded in a retrieved source.
👔 Manager How does streaming function? Does it provide a faster response, or just the perception of one?
💻 Engineer It provides the perception of speed — not a faster total response. The model generates one token at a time regardless of whether streaming is enabled. With streaming, each token is transmitted to the client as it is produced rather than being buffered until generation concludes. The total time to the final token remains unchanged. What improves is the time to the first token — the interface can display the beginning of the response almost immediately rather than waiting for the complete output. For customer-facing applications, streaming makes the system feel significantly more responsive. Conversely, for batch processing or downstream data pipelines that require the complete response before proceeding, streaming introduces unnecessary complexity.
References and Next Steps
- Previous: What Is Generative AI? →
- Next: Generative AI Model Architectures → — what is inside the model: transformers, self-attention, diffusion, and why architecture choices affect capability
- Then: Explain GenAI to Leadership — capstone exercise applying the full foundations track
- External — lost in the middle: Lost in the Middle: How Language Models Use Long Contexts (Liu et al., 2023) — the research behind context window placement strategy
- External — tokenisation: OpenAI Tokenizer — interactive tool to see exactly how your prompts are tokenised before calling the model