Function Calling and Structured Output

Summary

🧭 What this page covers

Small Language Models (SLMs) covered the model-sizing decision. This page now shifts from model choice to system integration: how any chosen model becomes production-reliable through function calling (deterministic tool invocation from model reasoning) and structured output (JSON schema enforcement for downstream processing).

A support lead asks, "Can the assistant check refund eligibility, pull the order status, and return a clean JSON verdict our workflow engine can trust?" That question is really about system contracts, not prompt cleverness.

Under the hood, that request crosses three different boundaries. The model has to decide whether it needs live order data, whether refund policy should be checked through a deterministic rule path, and whether the final answer is for a human reader or a workflow engine expecting typed fields. If you leave all of that inside unconstrained free text, the system becomes brittle fast: one model version adds a friendly preamble, another renames a field, and suddenly your "automation" is a regex held together by optimism.

This is why function calling and structured output matter so much in production. They turn "please do the right thing" into explicit contracts: which tool gets called, what arguments are passed, what schema comes back, and what downstream code can trust without guessing.

Function calling and structured output solve the core integration problem: LLMs produce free text, but software systems require structured, typed data. These two features replace fragile string parsing with typed contracts — making LLM output as predictable and reliable as an API response.

Feature	What it does	Why it matters
Function calling	Model decides to call a tool and returns structured arguments	Connects LLM reasoning to real system actions without free-text parsing
Structured output	Model response is constrained to a JSON schema	Eliminates parsing errors; survives model version upgrades
Parallel tool calls	Model calls multiple tools in a single response	Reduces round-trips; enables concurrent data fetching
Tool choice control	Caller specifies which tool to use, or forces a specific tool	Enables deterministic routing for predictable paths

An expense management system used to parse LLM responses with regex to extract expense amounts, categories, and approval flags. After three model version upgrades, the parsing broke twice due to format variations. After migrating to structured output with a fixed JSON schema, the system survived four model upgrades without a single parsing change. The schema is the contract.

🔌

Function calling bridges language and logic

The model reads natural language, reasons about what action is needed, and returns a structured tool call with typed arguments. The application executes the action deterministically. The model never executes code — it produces the specification that your code executes. This separation keeps LLM reasoning and deterministic execution cleanly separated.

📋

Structured output is a schema contract

When you define a JSON schema for the model's response, the model is constrained to produce output that validates against that schema. No extra text, no format variations, no "Here is the JSON you requested:" preamble. The output is the JSON. Downstream code parses a typed object — not a string that may or may not contain JSON somewhere inside it.

🔄

Both features require disciplined schema design

Poorly designed schemas — too broad, with optional fields everywhere, or requiring the model to infer implicit information — defeat the purpose. The schema should reflect exactly what downstream code needs: explicit required fields, typed values, and enumerations where the set of valid values is finite.

Why This Matters

🔧

Free-text parsing breaks at scale

String parsing is the wrong abstraction for LLM integration

Early LLM integrations parse model output with string manipulation — extracting values with regex or substring matching. This works until the model uses slightly different phrasing, adds an explanation, or outputs in a different order. At scale, format variations become incidents. One enterprise team tracked 34 distinct format variations across 6 months of model output for a task they thought was deterministic. Structured output replaces all 34 variations with one schema.

⚡

Tool calls reduce hallucination for factual tasks

The model retrieves — it does not remember

A model without tool access answering "What is the current balance on account #12345?" will hallucinate a plausible-sounding number. A model with a get_account_balance tool calls it, gets the real value, and reports it accurately. Function calling is the architecture that makes LLMs reliable for tasks requiring current, precise, or system-specific data — by ensuring the model fetches rather than fabricates.

🏗️

Tool architecture enables agent systems

Function calling is the foundation of autonomous agents

When a model can call tools, observe results, and decide on the next action based on those results, you have the foundation of an agent loop. The LLM becomes the reasoning engine; tools are its capabilities. Multi-step tasks — research, plan, write, validate, send — are executable by the model with appropriate tool definitions. The quality of tool definition directly determines agent reliability.

The production principle

Function calling and structured output shift the integration contract from "parse whatever the model produces" to "the model produces exactly what the schema requires." That shift makes the system testable, monitorable, and maintainable.

Core Concepts

🔌 Function Calling — How It Works

Function calling follows a multi-turn exchange between the caller and the model:

What the flow should make you notice:

1. The model never executes code directly. It proposes a tool call.
2. Your application stays in charge of permissions, retries, and error handling.
3. The tool result becomes evidence for the next model step, which is why clean JSON beats free-text tool output every time.

Three-step pattern:

1. Define tools as typed schemas — name, description, parameters with types and descriptions.
2. Send user message with tool definitions attached. The model decides whether to call a tool and which one.
3. Execute the tool call, return the result to the model, let the model produce the final response.

Tool definition structure (OpenAI format):

{
  "type": "function",
  "function": {
    "name": "get_account_balance",
    "description": "Retrieve the current balance for a customer account. Use when the user asks about their account balance or available funds.",
    "parameters": {
      "type": "object",
      "properties": {
        "account_id": {
          "type": "string",
          "description": "The customer's account identifier, e.g. 'ACC-12345'"
        },
        "currency": {
          "type": "string",
          "enum": ["USD", "EUR", "GBP"],
          "description": "The currency for the balance. Defaults to USD."
        }
      },
      "required": ["account_id"]
    }
  }
}

Critical design rules for tool descriptions:

• The description tells the model when to call the tool — be specific about the triggering conditions.
• Parameter descriptions tell the model how to extract the value from the user's message.
• Vague descriptions lead to incorrect tool selection and incorrect parameter extraction.

📋 Structured Output — Schema Enforcement

Structured output constrains the model's entire response to a JSON schema. Unlike function calling (which is about taking actions), structured output is about shaping the model's reasoning output into a typed format.

When to use each:

Use case	Function calling fit	Structured output fit
Retrieve data from an external system	Primary fit: invoke a tool such as get_customer_profile or fetch_invoice_status and return live data.	Optional for post-processing if you need the final answer in a strict schema.
Perform an action (send email, update record)	Primary fit: execute side-effect actions through controlled, permission-checked tools.	Not the execution mechanism; use only to shape a confirmation payload.
Extract typed entities from a document	Usually not needed unless extraction requires an external validation tool.	Primary fit: enforce fields such as invoice_id, amount, due_date with schema validation.
Generate a report with required fields	Useful when report data must be gathered from APIs before synthesis.	Primary fit: require deterministic report JSON with mandatory sections and types.
Route a request based on intent classification	Optional if routing logic depends on external systems.	Primary fit: force stable intent labels and confidence fields for downstream routing.
Multi-step agent with tool access	Essential for tool orchestration across steps (lookup, compute, update).	Strong companion for final step output so each run returns parse-safe structured results.

Structured output example — contract review:

{
  "schema": {
    "type": "object",
    "properties": {
      "parties": {
        "type": "array",
        "items": { "type": "string" },
        "description": "Names of all contracting parties"
      },
      "effective_date": {
        "type": "string",
        "format": "date",
        "description": "Contract effective date in ISO 8601 format"
      },
      "risk_flags": {
        "type": "array",
        "items": {
          "type": "object",
          "properties": {
            "clause": { "type": "string" },
            "risk_level": { "type": "string", "enum": ["low", "medium", "high"] },
            "rationale": { "type": "string" }
          },
          "required": ["clause", "risk_level", "rationale"]
        }
      },
      "requires_legal_review": {
        "type": "boolean"
      }
    },
    "required": ["parties", "effective_date", "risk_flags", "requires_legal_review"]
  }
}

The model's output is this JSON object — nothing else. Downstream code deserialises it directly into a typed data structure without any parsing logic.

🔄 Parallel Tool Calls

When a task requires multiple independent data fetches, the model can call multiple tools in a single response — reducing latency by parallelising the tool executions.

This is the nice version of concurrency: two independent facts, one model turn, one cleaner final answer.

Realistic Example

Scenario

A corporate travel booking assistant. An employee asks: "Book me on the cheapest available direct flight from London to New York on June 15th, under company policy limits."

Implementation

Step 1 — Tools defined:

[
  {
    "name": "search_flights",
    "description": "Search for available flights on a route and date. Use when the user requests flight options.",
    "parameters": {
      "origin": "string (IATA code)",
      "destination": "string (IATA code)",
      "date": "string (ISO 8601)",
      "direct_only": "boolean"
    }
  },
  {
    "name": "get_travel_policy",
    "description": "Retrieve the company travel policy limits for a given route class.",
    "parameters": {
      "route_class": "string: 'domestic' | 'international_short' | 'international_long'"
    }
  },
  {
    "name": "book_flight",
    "description": "Book a specific flight for the requesting employee.",
    "parameters": {
      "flight_id": "string",
      "employee_id": "string",
      "payment_method": "string: 'corporate_card' | 'expense_claim'"
    }
  }
]

Step 2 — First model response (parallel tool calls):

[
  { "tool": "search_flights", "args": { "origin": "LHR", "destination": "JFK", "date": "2026-06-15", "direct_only": true } },
  { "tool": "get_travel_policy", "args": { "route_class": "international_long" } }
]

Step 3 — Tool results returned:

• search_flights returns 5 flights (cheapest: £620 direct, BA0117)
• get_travel_policy returns: economy class required, max £800 for LHR-JFK

Step 4 — Second model response (book the flight):

{ "tool": "book_flight", "args": { "flight_id": "BA0117-20260615", "employee_id": "EMP-4821", "payment_method": "corporate_card" } }

Step 5 — Final natural language response:

"I've booked you on BA0117 on June 15th, departing LHR at 11:25, arriving JFK at 14:30 — direct, economy class, £620. This is within your travel policy limit of £800. Booking confirmation has been sent to your email."

What made this reliable:

• No arithmetic by the LLM (policy comparison is a schema lookup, not a calculation)
• No hallucinated flight data (search results come from a live tool)
• Final response grounded in tool results, not model memory

Senior Tech vs Dev Conversation

Senior Tech: When should we use function calling vs just asking the model to produce JSON in the prompt?

Dev: Use native function calling (or the structured output API parameter) whenever possible. Asking the model to "return JSON" in a prompt instruction produces JSON most of the time — but the model can prepend explanation text, wrap the JSON in markdown code fences, or add a trailing comment. Native structured output guarantees the response is exactly the schema and nothing else. It also survives model version upgrades because the constraint is enforced at the API level, not via prompt phrasing.

Senior Tech: We have 12 tools. The team is concerned about the model picking the wrong one.

Dev: Tool description quality is the primary lever. The description must specify the triggering condition precisely — not just what the tool does, but when to call it vs the alternatives. If two tools have overlapping descriptions, add a disambiguation clause: "Use this tool when the user explicitly asks about account balance. Do not use this tool for transaction history — use get_transactions for that." Also: benchmark your tool selection accuracy on a held-out test set before going to production. A 5% wrong-tool rate at 10,000 calls per day is 500 incidents per day.

Senior Tech: The team wants to give the agent the ability to update database records. Concerns?

Dev: Significant. Treat write-access tools with the same care as database write operations in any other system. Apply: (1) confirmation step before executing writes — the model presents what it will do, the user confirms; (2) least-privilege scope — the tool should only update fields relevant to its purpose; (3) audit logging for every tool call; (4) rate limiting to prevent runaway loops; (5) test with adversarial inputs — can a user craft a message that causes the agent to overwrite data it should not touch?

Senior Tech: Give me a minimal production config for structured output.

Dev:

response = client.chat.completions.create(
    model="gpt-4.1",
    messages=[
        {"role": "system", "content": "Extract the contract fields from the provided text."},
        {"role": "user", "content": contract_text}
    ],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "contract_extraction",
            "strict": True,  # enforces schema compliance
            "schema": CONTRACT_SCHEMA
        }
    },
    temperature=0.0  # deterministic extraction
)
result = json.loads(response.choices[0].message.content)

Common Pitfalls

Pitfall	What goes wrong	Prevention
Vague tool descriptions	Model calls wrong tool or wrong parameters	Write explicit triggering conditions and extraction guidance in every description
Using prompt instructions instead of native schema	Format inconsistency across model versions	Use response_format: json_schema with strict: true
Giving agents write access without confirmation	Agent overwrites data on mis-parsed intent	Require explicit user confirmation before any write-action tool executes
Optional fields everywhere in the schema	Model omits required data; downstream code breaks	Mark all fields the downstream code needs as required
No audit log for tool calls	Cannot debug incorrect agent behaviour	Log every tool call: input arguments, tool result, and model decision
No rate limiting on agent tool calls	Runaway loops exhaust API budget	Set per-call tool invocation limits; add loop detection

References and Next Steps

• Previous: Small Language Models →
• Next: Context Management for LLM Systems →
• Previous: LLM Capabilities and Limitations →
• OpenAI function calling docs: Function Calling Guide
• Structured output: OpenAI Structured Outputs
• Agent architecture patterns: OpenAI Agents SDK