Skip to main content

Operational Excellence: Evaluation, Safety & Governance

Foundations trackPart 3 of 3

Production discipline

Operate AI with evidence, layered controls, and clear ownership.

Evaluation tells you whether quality is acceptable, monitoring shows whether the system is healthy, and tracing explains how a run behaved.

Evaluate

Prove quality

Use representative datasets and release gates whenever models, prompts, tools, or retrieval change.
Protect

Layer controls

Combine input checks, scoped tools, deterministic rules, approvals, and output validation.
Observe

Trace outcomes

Connect quality, latency, cost, tool calls, and failures to the individual run that produced them.
Read the diagram

Every architecture diagram should make a control boundary visible.

In this article, read the flow from user input through authorization and business rules to see where AI guidance ends and deterministic execution begins.

Where this fits in the learning path

This is the third article in the Foundations track and assumes you already understand the platform and core building blocks.

If needed, go back to:

  1. What is Microsoft Foundry: Foundations & Architecture
  2. Core Capabilities: Models, Agents, Data & Tools
Freshness note

This article was refreshed on September 28, 2026 using current Microsoft Foundry documentation. Feature maturity can still vary by capability. For example, current tracing documentation states that tracing is generally available for prompt and hosted agents, while workflow and external agents remain in preview.

This article covers the final three capability areas:

  1. Evaluation: proving quality instead of guessing
  2. Safety: reducing harmful or unauthorized behavior
  3. Governance and observability: running AI systems with visibility, control, and accountability
When to use which

Choose Microsoft 365 Copilot when you want Microsoft to provide most of the user-facing productivity experience and much of the surrounding Microsoft 365 operating model.

Choose Microsoft Foundry when your organization needs custom evaluation pipelines, custom observability, custom tool governance, or operational ownership of bespoke AI applications and agents.

5. Evaluation: Quality Assurance for Probabilistic Systems​

The difference between a prototype and a production AI system is often not the prompt.

It is the evaluation discipline.

Traditional software tests often ask:

Given input X,
the system must return output Y.

Generative AI systems need a broader question:

Was the answer accurate?
Was it grounded?
Did it follow instructions?
Was the tool call correct?
Was the response safe?
Was latency acceptable?
Did anything regress after a model or prompt change?

Evaluation should be continuous, not one-time​

Foundry's current observability guidance emphasizes evaluation across the whole lifecycle:

  • Base model selection
  • Pre-production validation
  • Production monitoring

What should be evaluated​

The exact metrics depend on the workload, but common ones include:

  • Accuracy
  • Relevance
  • Groundedness
  • Coherence and fluency
  • Tool-call accuracy
  • Task completion
  • Safety outcomes
  • Latency
  • Token usage
  • Cost

For agentic systems, evaluation should also consider:

  • Whether the right tool was selected
  • Whether the sequence of actions made sense
  • Whether the final outcome matched the user goal
  • Whether retries or loops stayed within acceptable limits

CI/CD trigger points for evaluation​

Run evaluation whenever any meaningful variable changes:

  • Prompt changed
  • Model changed
  • Deployment type changed
  • Tool schema changed
  • Retrieval pipeline changed
  • Chunking or reranking changed
  • Business policy changed
  • Safety settings changed

The rule is simple:

If the change could affect output quality, safety, latency, or cost, it belongs behind evaluation gates.

6. Safety: Defense in Depth​

AI safety is not a single content filter.

It is a layered system.

Common risks include:

  • Prompt injection
  • Jailbreak attempts
  • Sensitive data leakage
  • Unauthorized tool use
  • Harmful or policy-violating output
  • Incorrect business actions
  • Overconfident unsupported answers
  • Excessive autonomy

Layered control model​

Practical safety controls​

Typical enterprise controls include:

  • Input validation
  • Content filtering
  • Prompt injection defenses
  • Identity and authentication
  • Role-based authorization
  • Tool allowlists and scoped permissions
  • Deterministic policy enforcement
  • Output validation
  • Rate limits
  • Spend limits
  • Human approval
  • Audit logging
  • Red teaming
  • Production monitoring

No single control is enough on its own.

The stronger principle is:

Safety comes from multiple layers that assume some other layer may fail.

Human approval is a safety control, not a weakness​

For higher-risk actions, human review is often the correct design choice.

Examples that often justify approval:

  • High-value payments
  • Destructive data changes
  • Customer-affecting account actions
  • Regulated workflow steps
  • Security-sensitive access changes

Well-designed human-in-the-loop checkpoints often improve both safety and organizational trust.

7. Governance and Observability: Operating AI Systems at Scale​

Once an organization moves from a few prototypes to dozens or hundreds of AI workloads, the problem changes.

You are no longer only asking:

Can this agent work?

You are asking:

Who owns this agent?
Which model version is it using?
What tools can it call?
What data can it access?
How much does it cost?
Why did this run fail?
Which changes caused regression?
Can we trace and audit what happened?

Observability is now a first-class capability​

Current Microsoft Foundry documentation groups observability into three connected capabilities:

  • Evaluation
  • Monitoring
  • Tracing

That framing is useful because it separates three different jobs:

  • Evaluation tells you whether the system is good enough
  • Monitoring tells you whether the system is healthy right now
  • Tracing tells you why a specific run behaved the way it did

Observability map​

What tracing gives you​

Tracing is especially important for agentic systems because a failure can come from many places:

  • The prompt or instructions
  • The model choice
  • Retrieval quality
  • Tool selection
  • Tool latency
  • Tool failure
  • State or memory handling
  • Business-rule rejection

Current Foundry tracing guidance highlights signals such as:

  • Inputs and outputs
  • Tool usage
  • Retries
  • Latencies
  • Costs

And it integrates with Azure Monitor / Application Insights using OpenTelemetry-aligned patterns.

Governance goes beyond the AI service itself​

Foundry governance is strongest when treated as part of wider enterprise governance:

How this differs from Microsoft 365 Copilot in operations​

This distinction becomes especially important during rollout.

With Microsoft 365 Copilot, much of the end-user experience is already productized inside Microsoft 365 apps, so operational questions often focus on:

  • Tenant readiness
  • Microsoft 365 permissions and content exposure
  • Adoption and change management
  • Compliance and retention controls in the Microsoft 365 estate

With Microsoft Foundry, the operational burden is broader because you are running a custom AI system. That usually means you own more of:

  • Architecture choices
  • Agent behavior
  • Tool boundaries
  • Evaluation gates
  • Observability design
  • Release management
  • Runtime governance

So while both need governance, they operate at different layers:

Microsoft 365 Copilot is mainly governed as a user-facing Microsoft 365 capability. Microsoft Foundry is governed as a custom AI platform and application estate.

RBAC is part of governance, and naming changed​

One easy way for docs to go stale is role naming.

As of September 28, 2026, Microsoft documentation notes that Foundry RBAC role names were recently renamed:

  • Foundry User
  • Foundry Owner
  • Foundry Account Owner
  • Foundry Project Manager

Older content may still reference the previous Azure AI role names during rollout.

That matters because teams often copy old tutorials and end up assigning the wrong roles.

Questions every production team should answer​

Before scale, a team should be able to answer:

  • Who owns each agent or AI application?
  • Which models and deployment types are approved?
  • Which tools are allowed, and under what permissions?
  • Which datasets are allowed for grounding?
  • Which evaluations must pass before release?
  • Which actions require human approval?
  • Which telemetry and traces must be retained?
  • Which alerts indicate quality, cost, or safety drift?

Example: Claims Investigation Agent​

Consider an insurance use case:

"Investigate claim CLM-83921 and determine whether it should be escalated."

Reference architecture​

Safe decision flow​

This pattern preserves the value of the agent while keeping high-risk control points deterministic and auditable.

Final Takeaways​

The operational lesson is straightforward:

  1. Evaluation turns AI from a demo into an engineering discipline.
  2. Safety requires layered controls, not a single guardrail.
  3. Observability means evaluation, monitoring, and tracing working together.
  4. Governance works best when Foundry is integrated into the wider Azure and organizational control model.
  5. Human approval remains a core design tool for high-risk actions.

Series Wrap-Up​

You now have the full Foundations view:

  1. Why Microsoft Foundry exists
  2. How models, agents, data, and tools fit together
  3. How to evaluate, secure, observe, and govern the system in production

From here, the next strong step is usually one of these:

  • Build a small reference architecture for a real use case
  • Design a model-evaluation matrix for a workload
  • Define tool boundaries and approval paths for an agent
  • Set up tracing, monitoring, and release gates before production rollout