LLM Evaluation and Observability
Summary
Context Management for LLM Systems and Agentic Patterns covered how to build LLM-powered systems that can retrieve, reason, and act. This page covers how to know whether they are working — evaluation frameworks for pre-deployment quality, and observability patterns for post-deployment monitoring. Without measurement, you are operating blind.
Friday's demo looked great. Tuesday's production traffic looks less great. Nobody changed the code path on purpose, but answers are now shorter, citations are weaker, and costs are somehow up 18%. Welcome to the part of the job where "it felt good in testing" stops being a strategy.
You cannot improve what you cannot measure. LLM systems have unique evaluation challenges: there is rarely a single "correct" answer, quality degrades gradually rather than failing hard, and the failure modes (hallucination, relevance drift, context window truncation) are invisible without instrumentation. Evaluation and observability are not post-launch concerns — they must be designed in from the first iteration.
| Evaluation type | When | What it measures | Primary tools |
|---|---|---|---|
| Unit evaluation | During development | Single input → expected output | pytest + custom eval harness |
| Dataset evaluation | Pre-deployment | Quality across representative sample | Azure AI Evaluation, RAGAS, DeepEval |
| LLM-as-judge | Pre/post deployment | Criteria-based quality at scale | GPT-4.1 as evaluator, custom rubrics |
| Production tracing | Post-deployment | Latency, token usage, cost, error rate | Azure Monitor, OpenTelemetry, Langfuse |
| Continuous evaluation | Ongoing | Quality drift detection on live traffic | Sample of production calls → re-evaluate |
A customer service AI was live for 3 weeks before anyone noticed response quality had degraded. The cause: a system prompt change 2 weeks earlier had shifted the model's tone toward terse, unhelpful answers on edge cases. No alert fired — there were no quality metrics in production. After implementing continuous evaluation (10% sample of live calls evaluated by GPT-4.1 with a quality rubric), similar regressions are now caught within 24 hours.
A golden dataset is a curated set of inputs with expected outputs — or at minimum, expected quality criteria. Without it, evaluation is subjective and unrepeatable. Build the golden dataset during development, ideally with domain expert review. 50–200 examples covering the core use cases, edge cases, and known failure modes is sufficient for most enterprise deployments. More is better; zero is not acceptable.
Every production LLM call should emit a trace: the full prompt, the response, token counts, latency, model version, and a trace ID. Traces enable debugging, cost analysis, quality sampling, and regression investigation. The cost of tracing is negligible compared to the cost of debugging a production issue without it. OpenTelemetry with a backend (Azure Monitor, Langfuse, Grafana) is the standard integration pattern.
Human evaluation is the gold standard but does not scale. LLM-as-judge — using a capable model (GPT-4.1, Claude Sonnet) with a structured rubric to evaluate other model outputs — scales to thousands of examples at low cost. Correlation with human judgement is high (0.7–0.9) for well-designed rubrics. LLM-as-judge is not a replacement for human review of high-stakes outputs, but it enables continuous quality monitoring at production scale.
Why This Matters
When a database query fails, an exception is raised. When an LLM response becomes less accurate over time — due to a model update, a prompt change, or a distribution shift in incoming queries — no exception is raised. The application continues functioning, returning worse answers quietly. Only a measurement pipeline that continuously evaluates response quality will detect this. Without it, quality degradation can go unnoticed for weeks.
At 10,000 calls per day with an average of 5,000 tokens per call, a 1,000-token context bloat per call costs 10 million extra tokens per day. At $0.01/1K tokens (input), that is $100/day in preventable cost — $3,000/month. Token usage monitoring with per-feature and per-endpoint breakdowns is the only way to identify which features are over-consuming context budget and which prompt changes reduced or increased cost.
LLM prompts are not deterministic programs. A prompt refinement that improves response quality on the primary use case may reduce quality on edge cases that were not included in the developer's informal testing. An evaluation pipeline that runs the golden dataset on every prompt change — before it reaches production — is the equivalent of a test suite for LLM quality. Without it, prompt engineering is guesswork at scale.
Define your quality metrics before you build. "Good response" is not a metric. "Groundedness ≥ 0.85, relevance ≥ 0.80, no hallucinated citations" is a metric. Define it, measure it, and set a deployment gate against it.
Core Concepts
📏 Evaluation Metrics
For RAG (Retrieval-Augmented Generation) systems:
| Metric | What it measures | Good score | How to compute |
|---|---|---|---|
| Groundedness | Does the response only use information from the retrieved context? | ≥ 0.85 | LLM-as-judge: compare response claims to context |
| Relevance | Does the response answer the actual question? | ≥ 0.80 | LLM-as-judge: rubric on question-answer alignment |
| Retrieval recall | Does the retrieved context contain the answer? | ≥ 0.90 | Check if gold answer appears in retrieved chunks |
| Faithfulness (RAGAS) | Are all response claims supported by context? | ≥ 0.80 | RAGAS library |
| Answer correctness | Does the answer match the expected answer? | Varies by task | Exact match, semantic similarity, or LLM judge |
For general chat and generation:
| Metric | What it measures | Tool |
|---|---|---|
| Coherence | Is the response logically structured and readable? | LLM-as-judge |
| Fluency | Is the language natural and grammatically correct? | LLM-as-judge |
| Harm / Safety | Does the response contain harmful content? | Azure Content Safety, LLM-as-judge |
| Task completion | Did the response fulfil the user's request? | LLM-as-judge with task-specific rubric |
⚖️ LLM-as-Judge Pattern
The judge model is not there to sound wise. It is there to turn "seems good" into a score you can graph, threshold, and compare over time.
Groundedness evaluation prompt template:
You are an expert evaluator for AI responses.
QUESTION: {question}
CONTEXT (retrieved documents): {context}
RESPONSE: {response}
Rate the response on GROUNDEDNESS (1–5):
- 5: Every claim in the response is directly supported by the context
- 4: Almost all claims supported; minor extension beyond context
- 3: Most claims supported; some statements not in context
- 2: Significant claims not supported by context
- 1: Response primarily contradicts or ignores context
Output JSON only:
{"groundedness": <int>, "reasoning": "<one sentence>", "unsupported_claims": ["<claim1>", ...]}
Running at scale with Azure AI Evaluation (Python):
from azure.ai.evaluation import GroundednessEvaluator, RelevanceEvaluator, evaluate
groundedness_eval = GroundednessEvaluator(model_config={
"azure_deployment": "gpt-4.1",
"azure_endpoint": os.environ["AZURE_OPENAI_ENDPOINT"],
"api_key": os.environ["AZURE_OPENAI_API_KEY"],
})
results = evaluate(
data="golden_dataset.jsonl", # {"question": ..., "context": ..., "response": ..., "expected": ...}
evaluators={"groundedness": groundedness_eval, "relevance": RelevanceEvaluator(...)},
output_path="eval_results.json",
)
print(f"Mean groundedness: {results['metrics']['groundedness.groundedness']:.2f}")
🔭 Production Tracing
Every LLM call in production must emit a trace. The OpenTelemetry standard is the recommended approach — it is vendor-neutral and supported by all major observability platforms.
If you only log the final answer, you miss the plot. Good traces tell you what the model saw, what it cost, how long it took, and which version did the talking.
Minimal tracing implementation (Python, OpenTelemetry):
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from azure.monitor.opentelemetry.exporter import AzureMonitorTraceExporter
provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(
AzureMonitorTraceExporter(connection_string=os.environ["APPINSIGHTS_CONN_STRING"])
))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("llm.service")
def call_llm(prompt: str, user_id: str) -> str:
with tracer.start_as_current_span("llm.call") as span:
span.set_attribute("user_id", user_id)
span.set_attribute("prompt_preview", prompt[:200]) # never log full PII prompt
start = time.monotonic()
response = client.chat.completions.create(
model="gpt-4.1",
messages=[{"role": "user", "content": prompt}],
max_tokens=1000,
)
latency_ms = (time.monotonic() - start) * 1000
span.set_attribute("prompt_tokens", response.usage.prompt_tokens)
span.set_attribute("completion_tokens", response.usage.completion_tokens)
span.set_attribute("latency_ms", round(latency_ms, 1))
span.set_attribute("model", response.model)
return response.choices[0].message.content
📈 Continuous Evaluation in Production
Sample production calls, evaluate them with LLM-as-judge, and alert on quality metric drift.
This is the quiet superpower of mature teams: they find regressions before users start writing annoyed emails.
Recommended production dashboards (Azure Monitor KQL):
// Token usage by endpoint per day
traces
| where message == "llm.call"
| extend endpoint = tostring(customDimensions["endpoint"]),
prompt_tokens = toint(customDimensions["prompt_tokens"]),
completion_tokens = toint(customDimensions["completion_tokens"])
| summarize total_tokens = sum(prompt_tokens + completion_tokens) by endpoint, bin(timestamp, 1d)
| render timechart
// P99 latency by model
traces
| where message == "llm.call"
| extend latency_ms = todouble(customDimensions["latency_ms"]),
model = tostring(customDimensions["model"])
| summarize p99_latency = percentile(latency_ms, 99) by model, bin(timestamp, 1h)
| render timechart
🧪 Evaluation Pipeline for Pre-Deployment Gates
GitHub Actions integration:
name: LLM Evaluation Gate
on:
pull_request:
paths: ['prompts/**', 'config/**']
jobs:
evaluate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Run evaluation suite
run: python scripts/evaluate.py --dataset golden_dataset.jsonl --baseline baseline_scores.json
env:
AZURE_OPENAI_ENDPOINT: ${{ secrets.AZURE_OPENAI_ENDPOINT }}
AZURE_OPENAI_API_KEY: ${{ secrets.AZURE_OPENAI_API_KEY }}
- name: Check evaluation gate
run: python scripts/check_gate.py --min-groundedness 0.85 --min-relevance 0.80
Realistic Example
Scenario
A HR policy Q&A chatbot built on internal policy documents. 1,200 calls per day. The team needs to know: are responses grounded in policy? Are they relevant? Is cost under control? Is quality stable week-over-week?
Implementation architecture
# Evaluation configuration
EVAL_CONFIG = {
"sample_rate": 0.10, # 10% of live calls → ~120/day evaluated
"metrics": ["groundedness", "relevance", "safety"],
"thresholds": {
"groundedness": 0.82, # alert if daily average drops below
"relevance": 0.78,
"safety": 0.99, # near-zero tolerance for unsafe content
},
"alert_channel": "teams-llm-alerts",
"dashboard": "azure-monitor-llm-quality",
}
# Daily quality report (automated)
daily_scores = {
"date": "2026-07-15",
"calls_sampled": 118,
"groundedness": 0.87, # ✅ above threshold
"relevance": 0.83, # ✅ above threshold
"safety": 1.00, # ✅ no unsafe responses detected
"avg_prompt_tokens": 4_820,
"avg_completion_tokens": 312,
"avg_latency_ms": 1_240,
"total_daily_cost_usd": 6.73,
}
Week-over-week quality trend dashboard entries:
Week 1: groundedness 0.88, relevance 0.84 — baseline
Week 2: groundedness 0.87, relevance 0.83 — stable
Week 3: groundedness 0.79 ← ALERT FIRED — investigation reveals system prompt truncation bug
Week 3 (fixed): groundedness 0.87, relevance 0.84 — restored
The alert at week 3 identified a truncation bug that had caused the system prompt to be cut at 4,096 tokens, silently dropping the grounding instruction. Without continuous evaluation, this would have been invisible.
Senior Tech vs Dev Conversation
Senior Tech: We already do unit testing. Why do we need a separate evaluation pipeline?
Dev: Unit tests check deterministic behaviour — does the function return the right type, does the API return 200, does the JSON parse correctly. Evaluation checks probabilistic quality — is the response accurate, grounded, relevant, safe. These are fundamentally different things. A unit test that asserts response != "" will pass even if the response is wrong. You need both: unit tests for infrastructure correctness, evaluation metrics for output quality. They serve different purposes and neither replaces the other.
Senior Tech: How many examples do we need in a golden dataset?
Dev: For initial deployment, 50–100 well-curated examples is a usable baseline. Aim for 150–200 before you rely on it for deployment gating. The composition matters more than the count: 60% core happy-path queries, 20% edge cases you discovered during development, 20% known failure modes you want to protect against (prompt injection attempts, ambiguous queries, queries with no good answer in the docs). Domain expert review of the expected answers is non-negotiable for high-stakes systems. Ten wrong expected answers in your golden set will produce ten wrong baseline scores.
Senior Tech: LLM-as-judge feels circular — using an LLM to evaluate an LLM. How reliable is it?
Dev: The research shows 0.75–0.85 correlation with human expert judgement for well-designed rubrics — which is comparable to inter-human annotation agreement on the same tasks. The key conditions: use a higher-capability model as judge than the target model (GPT-4.1 judging GPT-4o is more reliable than the reverse), use structured rubrics with defined score levels rather than open-ended questions, and calibrate by having human raters score 20–30 examples and checking that the LLM judge agrees. It is not perfect. For regulatory-grade quality validation, human review remains required. For continuous production monitoring at scale, LLM-as-judge is the only economically viable approach.
Common Pitfalls
| Pitfall | What goes wrong | Prevention |
|---|---|---|
| No golden dataset before deployment | Cannot establish a quality baseline; regressions invisible | Build golden dataset during development, before first prod deployment |
| Evaluating only on happy-path examples | Good scores mask failure on edge cases | Include 20% edge cases and 20% known failure modes in golden set |
| No production tracing | Cannot debug quality issues; no cost visibility | Instrument every LLM call with OpenTelemetry before launch |
| LLM-as-judge with vague rubrics | Low correlation with human judgement; inconsistent scores | Define score levels explicitly (1–5 with descriptions per level) |
| One-time evaluation, no continuous pipeline | Quality regressions from prompt changes go undetected | Run evaluation on every prompt/config change in CI |
| Sampling too low in production | Quality drift detected too late | Minimum 10% sample rate; 25%+ for high-stakes applications |
| Logging full prompt content | PII in traces; compliance violation | Log prompt previews (first 200 chars) or hash identifiers only |
References and Next Steps
- Next: LLM Role-Play and Quiz →
- Previous: Agentic Patterns →
- Azure AI Evaluation SDK: learn.microsoft.com/azure/ai-studio/how-to/develop/evaluate-sdk
- RAGAS (RAG evaluation): docs.ragas.io
- Langfuse (open-source LLM tracing): langfuse.com
- OpenTelemetry for LLM: opentelemetry.io/docs/concepts/signals/traces
- LLM-as-Judge paper (Zheng et al., 2023): arxiv.org/abs/2306.05685