Context Management for LLM Systems
Summary
How LLMs Work introduced the context window as a shared token budget. This page covers the engineering discipline of managing that budget in production: chunking strategies for large documents, history management for multi-turn conversations, long-context trade-offs, and the retrieval patterns that keep context relevant rather than just large.
The first version of many RAG systems has the same vibe: "we gave the model everything, so surely it will be smarter." Then the bills arrive, latency spikes, and the best paragraph is hiding in the middle of 18 pages of context like it joined a witness-protection program.
The context window is the most consequential resource in any LLM system. Manage it poorly and you pay for tokens that do not improve quality, lose instructions to silent truncation, or retrieve irrelevant content that confuses the model. Manage it well and you get reliable, cost-predictable responses even in complex, multi-document, multi-turn workflows.
| Component | Token footprint | Management strategy |
|---|---|---|
| System prompt | Fixed: 500–3,000 tokens | Keep concise; version-control changes |
| Retrieved context | Variable: 1,000–20,000 tokens | Chunk precisely; rerank aggressively |
| Conversation history | Grows with turns: 0–50,000+ | Truncate, summarise, or session-reset |
| User message | Typically 20–500 tokens | No management needed |
| Model response (output) | 200–4,000 tokens | Set max_tokens per use case |
A legal document platform ingested 200-page contracts into the context window — the full document, every call. With a 128K token window, it fit. But response quality was lower than expected, latency was consistently high, and cost per call was 8× what it needed to be. After implementing targeted chunking + reranking, they used 3–5 chunks per call instead of the full document. Quality improved because the model was no longer "lost in the middle" of 40,000 tokens of contract text.
A chunk is the smallest unit of text that is meaningful by itself and retrievable in isolation. Too large: the chunk contains irrelevant content that dilutes the signal. Too small: the chunk loses the surrounding context that makes it interpretable. Good chunking matches the retrieval grain to the query grain — section-level for document Q&A, paragraph-level for dense technical docs.
Every turn adds tokens to the history. Without a management strategy, a 20-turn conversation accumulates 10,000–20,000 tokens of history — most of which is no longer relevant to the current question. Truncation (drop oldest), summarisation (compress old turns), and session windowing (reset at natural break points) are all valid strategies. None is universally correct; the right choice depends on task continuity requirements.
A 1M-token context window does not mean the model uses all 1M tokens equally well. Research consistently shows quality degradation for content buried in the middle of very long contexts. A 200K-token context is not 4× better than a 50K-token context for most tasks — targeted retrieval of the most relevant 5,000 tokens frequently outperforms undifferentiated stuffing of 200,000 tokens.
Why This Matters
API pricing is per token, not per "useful token." A system that injects 20,000 tokens of context when 4,000 targeted tokens would achieve equivalent quality is paying 5× the necessary input cost. At 50,000 calls per day, a 16,000-token reduction per call saves 800 million tokens per day in input cost. Context management is cost engineering.
LLMs recall information near the beginning and end of long contexts more reliably than content in the middle. A contract clause at position 150 of 200 retrieved paragraphs may be used less reliably than if it were positioned first. Reranking and strategic placement — not just retrieval volume — determine whether retrieved context actually improves quality.
When the total token count exceeds the context window limit, the API does not return an error — it silently truncates the oldest or lowest-priority content. A system prompt instruction at position 20,000 in a 16K window is simply gone. The model has no record of it. The first symptom is unexpected model behaviour — not an exception in the application logs.
Before deploying any LLM feature, define a per-call token budget: how many tokens for the system prompt, how many for retrieved context, how many for history, and how many for the response. Measure actual usage on a representative sample. Adjust before the system reaches production scale.
Core Concepts
✂️ Chunking Strategies
Chunking is the process of dividing source documents into units that can be retrieved and injected into the context window. The goal is to match the retrieval grain to the query grain.
The important detail here is that chunking is not just splitting text. It is defining the unit your retrieval system is allowed to remember.
Chunking strategy comparison:
| Strategy | How it works | Best for | Trade-offs | Mini case |
|---|---|---|---|---|
| Fixed-size | Split every N tokens with overlap. | Uniform documents, quick setup. | Splits mid-sentence; loses semantic boundaries. | Rapid prototype indexing of internal wiki pages before advanced retrieval tuning. |
| Sentence / paragraph | Split on natural boundaries. | Prose documents, policies. | Variable chunk size; may lose section context. | HR policy Q&A where preserving paragraph meaning improves answer quality. |
| Semantic | Group sentences by topic coherence (embedding similarity). | Heterogeneous documents. | Higher compute cost at indexing time. | Product manuals with mixed specs and troubleshooting sections benefit from topic-aware grouping. |
| Hierarchical | Store section + paragraph chunks; retrieve section, inject paragraph. | Long technical docs with structure. | More complex retrieval; more storage. | API documentation where section-level retrieval narrows scope, then paragraph-level evidence is injected. |
| Document-level | Embed and retrieve full documents. | Short reference documents (< 1,000 tokens). | Not suitable for long documents. | Short SOP one-pagers where full-document retrieval is simpler and sufficient. |
Practical default for enterprise documents:
from langchain.text_splitter import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(
chunk_size=512, # tokens per chunk
chunk_overlap=64, # overlap to preserve boundary context
separators=["\n\n", "\n", ". ", " "], # semantic boundary preference
)
chunks = splitter.split_text(document_text)
Chunk size guidance:
256–512 tokens— Precise extraction tasks (contracts, policies, forms)512–1,024 tokens— General Q&A, document understanding1,024–2,048 tokens— Summary and synthesis tasks where full paragraph context matters
Always add chunk metadata:
{
"text": "...chunk content...",
"source": "policy-v4.pdf",
"page": 12,
"section": "3.2 Approval Thresholds",
"chunk_index": 47,
"document_id": "pol-2026-04"
}
Metadata enables citation generation and makes chunk provenance traceable.
🔍 Retrieval and Reranking
Chunking produces the index. Retrieval selects the chunks to inject. The two-stage pattern — retrieve broadly, then rerank precisely — consistently outperforms single-stage retrieval.
If your retrieval stack does only one stage, it usually confuses "similar enough to retrieve" with "good enough to show the model."
Two-stage retrieval:
# Stage 1 — vector recall (fast, approximate)
candidates = vector_store.similarity_search(query, k=20)
# Stage 2 — cross-encoder rerank (slower, precise)
from sentence_transformers import CrossEncoder
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
scores = reranker.predict([(query, c.page_content) for c in candidates])
top_chunks = [candidates[i] for i in sorted(range(len(scores)), key=lambda x: scores[x], reverse=True)[:5]]
Placement rule: inject the highest-ranked chunk first, not last. The model pays more attention to content near the beginning of retrieved context.
📜 Conversation History Management
In multi-turn conversations, history grows with every turn. The three management strategies:
This is a product choice as much as a technical choice. A legal-review workspace and a casual support chat should not manage memory the same way.
Summarisation pattern (best for long conversations):
HISTORY_SUMMARY_THRESHOLD = 10 # turns before summarisation
if len(conversation_history) > HISTORY_SUMMARY_THRESHOLD:
old_turns = conversation_history[:-4] # keep last 4 turns verbatim
summary = llm.chat([
{"role": "system", "content": "Summarise this conversation history concisely, preserving key facts and decisions."},
{"role": "user", "content": format_turns(old_turns)}
])
conversation_history = [
{"role": "system", "content": f"[Conversation summary: {summary.content}]"}
] + conversation_history[-4:]
Token budget allocation for a 128K window:
System prompt: 2,000 tokens (fixed; version-controlled)
Retrieved context: 12,000 tokens (top-5 chunks at ~2,400 avg)
Conversation history: 6,000 tokens (max ~15 recent turns)
User message: 500 tokens (generous cap)
───────────────────────────────────────
Total input: 20,500 tokens (well within 128K limit)
Max response: 4,000 tokens (set via max_tokens)
🔭 Long-Context Models — Trade-offs
As of 2026, several models offer 1M+ token context windows. The common assumption — "longer context = better results" — is not reliable.
| Context size | Retrieval strategy | Quality characteristic |
|---|---|---|
| Short (4K–32K) | Aggressive chunking + retrieval required | High quality on retrieved content; misses non-retrieved context |
| Mid (32K–128K) | Targeted retrieval + selective history | Good balance for most enterprise workflows |
| Long (128K–1M) | Selective retrieval still recommended | Quality degrades mid-document; higher cost per call |
| Very long (1M+) | Experimental in most use cases | Useful for code repository analysis; unreliable for dense Q&A |
The practical guidance (2026): Use a 128K model with targeted 5-chunk retrieval before using a 1M model with full-document injection. Benchmark both on your task. Full-document injection frequently underperforms targeted retrieval on factual precision tasks.
Realistic Example
Scenario
A contract analysis platform. Users upload 80-page service agreements and ask questions like "What is the auto-renewal clause?" or "What are our termination rights?"
Context management design
class ContractContextManager:
def __init__(self):
self.system_prompt_tokens = 1_200 # cap
self.retrieval_budget = 10_000 # top-5 chunks
self.history_budget = 4_000 # 8–10 turns
self.response_budget = 2_000 # max_tokens
def build_context(self, query: str, contract_id: str, history: list) -> dict:
# 1. Retrieve top-5 relevant chunks from the contract
chunks = self.retrieve(query, contract_id, top_k=20)
ranked_chunks = self.rerank(query, chunks, top_n=5)
# 2. Manage history — summarise if over budget
managed_history = self.manage_history(history, budget=self.history_budget)
# 3. Assemble context — highest-ranked chunk first
context_text = "\n\n---\n\n".join(
f"[Section: {c.metadata['section']}]\n{c.text}"
for c in ranked_chunks
)
messages = [
{"role": "system", "content": SYSTEM_PROMPT},
*managed_history,
{"role": "user", "content": f"Contract context:\n{context_text}\n\nQuestion: {query}"}
]
total_tokens = self.count_tokens(messages)
assert total_tokens < 128_000, f"Context overflow: {total_tokens} tokens"
return messages
Result: Average call cost reduced from 28,000 tokens (full document) to 14,000 tokens (targeted retrieval). Response quality on factual Q&A improved by 11% on the evaluation set — because targeted chunks were more precisely relevant than the full document context.
Senior Tech vs Dev Conversation
Senior Tech: Our RAG system retrieves 20 chunks per query. Is that too many?
Dev: Probably, yes. The research consistently shows diminishing returns past 5–7 top-ranked chunks for most Q&A tasks. Twenty chunks might be 20,000–30,000 tokens — that is a third of a 128K window used on context that may be mostly irrelevant. The better approach: retrieve 20 candidates for high recall, then rerank and keep the top 5. You spend retrieval compute to get recall, then spend reranking compute to get precision. What lands in the context window should be the best 5, not the first 20.
Senior Tech: We're building a customer support chat. Users have long conversations. What's our history strategy?
Dev: Depends on task continuity. For support chat, I'd use rolling summarisation: keep the last 6 turns verbatim (the active context), and after every 8 turns, compress the older history into a 200-token summary. That summary becomes the first message in the history list. You preserve continuity — the model knows the customer mentioned a billing issue 15 turns ago — without accumulating 10,000 tokens of history per session. Test it by checking whether the model correctly references earlier facts after the summarisation runs.
Senior Tech: Does the chunk overlap setting actually matter?
Dev: Yes, meaningfully. Chunks with zero overlap can split a sentence — or a key phrase — right at the boundary, so the sentence about a penalty clause ends on chunk 12 and the penalty amount starts on chunk 13. Neither chunk retrieves correctly for a query about penalties. An overlap of 10–15% of the chunk size (e.g. 50–100 tokens for a 512-token chunk) prevents boundary splits from losing relevant content. Too much overlap — say 50% — creates near-duplicate chunks that add noise to vector similarity search. 10–20% overlap is the practical sweet spot.
Common Pitfalls
| Pitfall | What goes wrong | Prevention |
|---|---|---|
| Injecting full documents instead of chunks | High cost, "lost in the middle" quality degradation | Always chunk + retrieve; only use full-doc for docs < 1,000 tokens |
| No chunk overlap | Boundary splits lose key phrases | Use 10–20% overlap relative to chunk size |
| Retrieving by vector similarity alone | Keyword-exact matches missed (e.g. clause numbers, codes) | Add BM25 or keyword search alongside vector; hybrid retrieval |
| History grows unbounded | Context overflow; system prompt silently truncated | Implement history summarisation trigger at 8–10 turns |
| Placing retrieved chunks in the middle of the context | "Lost in the middle" — less reliable model recall | Highest-relevance chunk first, second-highest last |
| No per-call token budget definition | Usage surprises in billing; silent overflow | Define and enforce component-level token budgets before deployment |
References and Next Steps
- Previous: Function Calling and Structured Output →
- Next: Agentic Patterns →
- Research: Lost in the Middle (Stanford, 2023)
- Chunking library: LangChain Text Splitters
- Reranking: Cohere Rerank API, sentence-transformers CrossEncoder
- Vector stores: Azure AI Search, Qdrant, pgvector