How LLMs Work

Course Information & Prerequisites

Use this sequence to get the most value from the full LLM Core section:

1. Start here with How LLMs Work for inference mechanics.
2. Continue to LLM Capabilities and Limitations for reliability boundaries.
3. Read Small Language Models (SLMs) to understand model-size trade-offs.
4. Move into Function Calling and Structured Output, Context Management for LLM Systems, and Agentic Patterns for system design.
5. Finish with LLM Evaluation and Observability and the Role Play and Quiz.

Recommended background before this module:

1. Foundations pages, especially Generative AI Ecosystem and How Generative AI Works.
2. Basic comfort with prompts, APIs, and application architecture.
3. Curiosity about how LLMs behave under production constraints like cost, latency, and context limits.

Summary

🧭 What this page covers

Generative AI Ecosystem and How Generative AI Works already covered the broad GenAI landscape. This page is the starting point for the LLM module and narrows the focus to one thing: what matters when an LLM call hits production traffic.

This is the operational view of inference. LLMs still predict the next token from prior tokens and context, but in production the engineering questions are more specific: how many tokens are we sending, what gets dropped when the prompt grows, how does output streaming behave, and which decoding settings change reliability.

If a live system is expensive, slow, truncated, or inconsistent, the root cause is usually one of four inference-time controls:

Mechanic	What it controls	Practical consequence
🔤 Tokenisation	How text maps to model inputs	Cost accounting and prompt fit
📦 Context window	How much the model can see per call	Trade-offs between instructions, retrieval, history, and output
⚡ Generation loop	How the model produces output token by token	Latency, streaming, and completion behavior
🌡️ Decoding settings	How the next token is sampled	Stability, variation, and output discipline

A claims operations team launched an internal assistant and spent two weeks blaming the model for inconsistent outputs. The issue turned out to be configuration drift: one service path used temperature: 0.2, another used 0.8. The model was stable. The wrapper was not.

🔤

Every request is a token budget decision

The model does not see words, pages, or paragraphs. It sees token IDs. That means prompt size, retrieved evidence, history, and output length all become budget choices that directly affect spend and fit.

📦

Context windows are shared working memory, not free space

Instructions, retrieved documents, user input, prior turns, and response allowance all compete for the same limited window. The failure mode is rarely dramatic. The model just sees less of what you thought it would see.

🌡️

Configuration quality is product quality

Many production incidents blamed on "the model" are really prompt assembly, retrieval placement, or decoding configuration issues. Inference settings are part of the application design, not just API parameters.

Why This Matters

💰

Inference design shows up on the bill

LLM cost is mostly a systems design problem

The difference between a compact prompt and an overstuffed one looks minor in development and painful at production volume. Retrieval scope, history growth, and output caps are often more important to cost than the model choice itself.

🧵

Prompt assembly changes output quality

The same model can look smart or sloppy depending on the request shape

High-quality retrieved chunks placed well, a sane history strategy, and explicit response limits produce materially better results than dumping raw context into the window and hoping the model sorts it out.

🛠️

This is where black-box thinking stops working

Once an LLM is part of a product, you need mechanical intuition

Support teams care about latency, finance teams care about token burn, and users care whether answers cut off or wander. Understanding inference mechanics gives you levers to fix those issues without treating every problem as model failure.

The key insight

An LLM call is not just "prompt in, answer out." It is a small runtime system with budgets, ordering effects, stop conditions, and tuning knobs. Reliable products come from treating it that way.

Core Concepts

🔤 Tokenisation — Measuring the Request

The model never processes words or characters. It processes tokens — integer IDs from a fixed vocabulary of roughly 50,000–100,000 entries, where each entry represents a subword unit. The tokeniser converts your prompt to this sequence before any model computation begins.

Input	Approximate token count	Note
"Hello, world!"	4	Common words are often single tokens
"tokenization"	3	Uncommon words split into subwords
"GPT-4o"	5	Compound terms split further
1,000-word document	~1,300–1,500 tokens	Rule of thumb: 1 word ≈ 1.3 tokens in English
1,000-word document (code)	~500–700 tokens	Code is tokenised more compactly

Why this matters for system design:

• Every component in your request — system prompt, retrieved content, conversation history, user message — consumes tokens from the same shared budget.
• Models are billed by token, not by word or character. Understanding token ratios directly translates to cost prediction.
• Non-English text tokenises less efficiently than English for most models, consuming more tokens per character.

📦 The Context Window — Allocating Shared Space

The context window is the total token capacity of a single model call — the sum of everything the model can see when it generates a response. As of 2026, production LLMs commonly offer 128K–2M token windows.

Context window management rules:

• Set explicit maximum lengths for each component: system prompt, retrieval, history, and user message.
• Implement a history truncation strategy — summarise or drop oldest turns before the window fills.
• Place the most relevant retrieved content near the beginning or end of the context, not buried in the middle.
• Monitor context utilisation per call in production — context budget exhaustion is a silent failure.

⚡ The Generation Loop — How Responses Emerge

The model generates output one token at a time. For each new token, the entire context — including all previously generated tokens — is processed through the network.

This is the part many teams underestimate: the model is not "thinking for a while and then answering." It is making thousands of tiny next-token decisions in sequence, which is why streaming, stop conditions, and max token limits have such a visible product impact.

Practical consequences:

• Streaming is essential for UX. Without streaming, the user sees nothing until the entire response is complete. With streaming, tokens appear as they are generated — reducing perceived latency dramatically for long responses.
• Stop sequences control output shape. A stop sequence is a string that, when generated, halts the loop immediately. Using </answer> as a stop sequence prevents the model from continuing beyond the structured response block.
• Longer outputs cost more. Every generated token adds latency and increases output cost. Set max_tokens explicitly to cap response length and prevent runaway output.

🌡️ Key Inference Settings — Controlling Behavior

These are the settings that most directly control output quality, consistency, and cost:

Setting	What it does	Enterprise guidance
Temperature	Controls randomness in token sampling. 0 = deterministic, 1.0 = high variation	Set 0.1–0.2 for factual, policy-anchored tasks; 0.7–0.9 for creative or exploratory tasks
Top P (nucleus sampling)	Restricts sampling to the smallest set of tokens whose cumulative probability ≥ P	Set 0.85–0.95; do not combine with very low temperature (redundant effect)
Max tokens	Hard cap on response length	Set explicitly; default maximums are wasteful on short-answer tasks
Stop sequences	Strings that halt generation when produced	Use to enforce output structure — e.g. ["</json>", "\n\n---"]
Frequency penalty	Reduces probability of tokens that have already appeared	Use 0.1–0.3 to reduce repetition in long-form output
Presence penalty	Penalises tokens that have appeared at all, regardless of frequency	Use sparingly; can cause vocabulary drift in structured tasks

Temperature and Top P interact

Setting both temperature very low (0.0–0.1) and Top P very low (0.5) simultaneously can cause degenerate output — the model collapses to a single predictable token at every step. In practice: set temperature low for consistency, leave Top P at 0.9 unless you have a specific reason to lower it.

📊 Configuration Reference for Common Use Cases

# Policy-anchored QA (compliance, legal, finance)
temperature: 0.1
top_p: 0.9
max_tokens: 800
stop: ["</answer>"]
frequency_penalty: 0.0

# Customer-facing chat assistant
temperature: 0.4
top_p: 0.9
max_tokens: 600
frequency_penalty: 0.2

# Creative content generation
temperature: 0.8
top_p: 0.95
max_tokens: 2000
frequency_penalty: 0.3

# Structured data extraction (JSON output)
temperature: 0.0
top_p: 1.0
max_tokens: 1000
stop: ["}"]

Realistic Example

Scenario

A financial services firm runs an internal policy Q&A assistant. An employee asks: "Can I expense a client dinner over $200 without pre-approval?"

What happens at inference time

System prompt:     ~800 tokens   (instructions + tone + format rules)
Retrieved policy:  ~3,200 tokens (top 4 policy document chunks, reranked)
Conversation hist: ~400 tokens   (last 3 turns)
User question:     ~25 tokens
─────────────────────────────────────────────────────
Total input:       ~4,425 tokens

1. Tokeniser converts the assembled prompt to 4,425 token IDs.
2. Context window check — 4,425 tokens against a 128K budget. Well within limits.
3. Generation loop begins. The model computes probability distributions over its vocabulary (~50K tokens), samples the first output token using temperature: 0.1, then the next, and so on.
4. At </answer>, the stop sequence halts generation. Response length: 320 tokens.
5. Total call cost: 4,425 + 320 = 4,745 tokens.

What controlled configuration delivers

At temperature 0.1, the same question produces the same policy-aligned answer on every call. The retrieved policy chunks are placed first in the context after the system prompt — not buried after conversation history — ensuring the model uses them reliably.

At temperature 0.9 (the prior default), the same system produced 7 distinct phrasings of the same answer on consecutive identical calls — including one that omitted the pre-approval threshold entirely.

Senior Tech vs Dev Conversation

Senior Tech: The model says temperature 0 makes output deterministic. Is that actually true?

Dev: Not exactly. Temperature 0 makes the model always select the highest-probability token — it becomes greedy decoding. In practice this is very stable, but floating-point hardware differences across data centres can occasionally produce small variations. For all practical purposes, treat it as deterministic. The real issue is that temperature 0 can be over-confident — it commits to the first token without exploring alternatives. For tasks where the "best" answer is ambiguous, 0.1–0.2 is safer.

Senior Tech: We have a retrieval system injecting 15 document chunks. What's the risk?

Dev: Volume and placement. First: 15 chunks could be 15,000–20,000 tokens. That is fine for a 128K window but expensive. Evaluate whether 5 high-quality reranked chunks outperform 15 raw ones — they usually do. Second: do not dump all 15 chunks between the system prompt and the user question, because content in the middle of long contexts is recalled less reliably than content near the start or end. Place the top-ranked chunk first.

Senior Tech: When do we hit context window limits in real systems?

Dev: Faster than teams expect. A detailed system prompt (1,500 tokens), five retrieved PDFs at 2,000 tokens each (10,000 tokens), a 30-turn conversation (6,000 tokens), and a user message (100 tokens) is already 17,600 tokens. A GPT-4-32k window fills in under 2 calls of heavy retrieval. The design answer is: set component-level token budgets and implement history summarisation after 10 turns.

Senior Tech: Give me a production-safe settings template.

Dev:

{
  "model": "gpt-4.1",
  "temperature": 0.1,
  "top_p": 0.9,
  "max_tokens": 800,
  "stop": ["</answer>"],
  "frequency_penalty": 0.1,
  "stream": true
}

Common Pitfalls

Pitfall	What goes wrong	Prevention
Using SDK default temperature	Inconsistent output on identical inputs	Always set temperature explicitly; default to 0.1 for factual tasks
Not setting max_tokens	Runaway responses increase cost and latency	Set max_tokens per use case and monitor P95 response length
Injecting full documents as context	Token budget exhaustion and "lost in the middle" degradation	Use targeted retrieval: 3–5 chunks, reranked, placed strategically
Combining very low temperature with very low Top P	Degenerate repetitive output	Set temperature low; leave Top P at 0.85–0.95
Not streaming for interactive use cases	User sees nothing until response is complete	Enable streaming for any human-facing interface
Accumulating conversation history without truncation	Context fills; early policy instructions are cut	Summarise or drop oldest turns; keep history budget explicit

References and Next Steps

• Next: LLM Capabilities and Limitations →
• Previous module: Foundations of Generative AI →
• Tokeniser tools: OpenAI Tokeniser, Hugging Face Tokeniser
• Context window research: Lost in the Middle (Stanford, 2023)
• OpenAI API reference: Chat Completions