Structured Outputs: Making LLMs Reliable in Production

Every LLM system I've shipped has hit the same wall, and it's not the one people warn you about. Not reasoning, not accuracy, not latency. Format.

The prototype works in a notebook. The model picks out entities, classifies sentiment, summarizes documents that read well. Then you wire it into a real system and the whole thing falls apart. The model wraps the JSON in a markdown code fence. It prepends "Sure, here is the JSON you asked for:" before the actual payload. It returns "true" as a string when the schema asked for a boolean. It invents a field that isn't in your schema. I've debugged every one of these in production, and most of them blew up at 4 PM on a Friday.

This is the structured output problem. Getting it right is one of the most underrated parts of running LLMs in production, and the tooling has gotten better than most people realize.

The naive approach: prompt engineering and regex

The first instinct is to ask politely. You append "Return valid JSON with no additional text" to the prompt and ship. When that fails 5% of the time, you wrap the parse in try/except and retry. When that still isn't enough, you reach for regex.

import re
import json

def extract_json(text: str) -> dict:
    # Try to find JSON in the response
    match = re.search(r'\{.*\}', text, re.DOTALL)
    if match:
        return json.loads(match.group())
    raise ValueError("No JSON found in response")

This works until it doesn't. Nested braces break the regex. Missing commas pass the regex and blow up json.loads. The model occasionally returns an array when you asked for an object. You end up maintaining a brittle parser for a problem that shouldn't exist at all.

The real issue is that LLM decoding is unconstrained by default. At every token the model assigns probability across its full vocabulary, and nothing stops it from picking one that violates your schema. The fix is to constrain the decoder itself, not the prompt.

Constrained decoding: enforcing structure at the token level

Constrained decoding changes the sampling step. Before each token, the system filters the vocabulary down to tokens that keep the output consistent with a schema. Some teams call it grammar-guided generation, or logit masking. Same idea, different name.

The mechanics:

You define a schema. JSON Schema, a Pydantic model, a context-free grammar, a regex.
Before each sample, the system checks which tokens in the vocabulary keep the output schema-valid.
Every other token has its logit set to negative infinity, which zeros its probability.
The model samples from the remaining valid tokens.

If the schema says "age" is an integer and the model just emitted "age":, only tokens that begin a valid integer have non-zero probability. The model literally can't emit "twenty-five" or null because those tokens aren't on the table.

Schema:  { "age": integer, "name": string }

Step 1: Generate `{` → only `{` is valid (start of object)
Step 2: Generate `"age"` → only valid field names allowed
Step 3: Generate `:` → required after field name
Step 4: Generate `25` → only digit tokens allowed (integer type)
Step 5: Generate `,` → comma or `}` allowed (more fields remain)
...

The output is guaranteed to parse. Not probably. Not usually. Guaranteed. The model still picks the content — which integer, which string — but the structure is locked by the schema.

Here it is on real, messy input. Below are two recorded runs of Gemini Flash with a schema attached (an order object: customer, items with quantities, priority, optional total and date). Feed it a rambling email or a chat scribble and watch structured JSON come out. These are actual captured runs, replayed:

No prose, no markdown fences to strip — the schema constraint means the model can only emit a valid order object. Watch it map "rush us 4 bags" to a quantity and priority "high", and "no rush whenever" to priority "low", while omitting the fields that aren't in the message rather than hallucinating a total. The green block at the end is the parsed, schema-valid result.

Modern solutions in practice

The ecosystem has settled into a handful of approaches, each with its own trade-offs.

API-level JSON mode and structured outputs

OpenAI, Google, and Anthropic all support some form of structured output. OpenAI takes a response_format: { type: "json_schema", json_schema: {...} } and guarantees the response matches it. Gemini takes response_mime_type: "application/json" with an optional schema. These are the easiest to reach for: pass a schema, get valid output back. The catch is that you're tied to a provider, and JSON Schema feature coverage varies — some constructs that work on OpenAI silently fail on Gemini and the other way around.

Function calling and tool use

Function calling is structured output wearing a different hat. You define a tool with a parameter schema and the model emits a call that matches it. This is the default pattern for agents, and it fits when the output is naturally an action: a search, a database write, an API request.

tools = [{
    "type": "function",
    "function": {
        "name": "extract_contact",
        "parameters": {
            "type": "object",
            "properties": {
                "name": {"type": "string"},
                "email": {"type": "string", "format": "email"},
                "phone": {"type": "string"}
            },
            "required": ["name", "email"]
        }
    }
}]

A concrete example: Gemini structured output and a function call

Talking about schemas in the abstract gets old fast. Here's the same idea end to end with Gemini and a tiny calendar-event domain — first as data extraction, then as a tool call. Two postures of the same constrained decoder.

Step 1: define the schema with Pydantic

from pydantic import BaseModel, Field

class MeetingRequest(BaseModel):
    title: str = Field(description="Short meeting title")
    attendees: list[str] = Field(description="Names or emails of attendees")
    start_iso: str = Field(description="ISO 8601 start, e.g. 2026-05-20T12:00:00")
    duration_minutes: int = Field(ge=5, le=480)

Pydantic is the source of truth for the shape. The description strings ship to the model alongside the field names and act like inline hints. The ge and le constraints become JSON Schema bounds, so the decoder masks any token that would push duration_minutes to 0 or 999. The model can't pick an out-of-range value because the tokens aren't on the table at sampling time.

Step 2: call Gemini with `response_schema`

from google import genai

client = genai.Client()  # reads GEMINI_API_KEY

response = client.models.generate_content(
    model="gemini-2.5-flash",
    contents="Schedule lunch with Maria next Friday at noon for 1 hour",
    config={
        "response_mime_type": "application/json",
        "response_schema": MeetingRequest,
    },
)

meeting: MeetingRequest = response.parsed
print(meeting.title, meeting.start_iso, meeting.duration_minutes)

Three things are doing the work here:

response_mime_type="application/json" flips Gemini into JSON mode. No prose, no code fences, no friendly preamble.
response_schema=MeetingRequest is the actual constraint. The SDK serializes the Pydantic class to JSON Schema and ships it with the request. The decoder uses it to mask any token that would violate the shape, the types, or the bounds.
response.parsed hands you back a populated MeetingRequest instance. No json.loads, no try/except, no defensive string surgery. If the model would have produced invalid JSON the call fails loudly instead of returning garbage that breaks something downstream.

That's the full structured-output pattern. Schema in, typed object out.

Step 3: define the tool

Same domain, exposed as something the model can call instead of something it fills in:

def create_calendar_event(
    title: str,
    attendees: list[str],
    start_iso: str,
    duration_minutes: int,
) -> dict:
    """Create a calendar event and return its ID and status."""
    # In real code this would hit Google Calendar or your scheduler.
    event_id = "evt_" + start_iso.replace(":", "").replace("-", "")
    return {"event_id": event_id, "status": "created"}

Plain Python. No separate schema file, no decorator. The Gemini SDK reads the function signature and docstring and builds the tool schema from them. The argument names, type hints, and docstring are the contract — rename attendees to participants and the contract the model sees changes too.

Step 4: let Gemini call the tool

from google.genai import types

response = client.models.generate_content(
    model="gemini-2.5-flash",
    contents="Book a 30 minute design review with Ana and Luis tomorrow at 10am",
    config=types.GenerateContentConfig(
        tools=[create_calendar_event],
    ),
)

# Gemini decided to call the tool. The SDK does not auto-execute by default.
for part in response.candidates[0].content.parts:
    if part.function_call:
        fc = part.function_call
        result = create_calendar_event(**fc.args)
        print(fc.name, "->", result)

A few details that matter:

tools=[create_calendar_event] tells the model what's available. The model decides whether to call it and with which arguments. The decoder applies the same constraint logic as before: any tool call it emits has the right argument names and the right types.
The response carries a function_call part with a name and an args dict. The SDK validates args against the function signature before handing it back, so splatting with **fc.args is safe.
Auto-execution is opt-in. The SDK has an automatic_function_calling mode that runs the function for you and feeds the result back in a loop. In production I usually want the keys. Manual dispatch lets me log the call, check a rate limit, refuse a side-effectful operation when something looks off, or swap in a dry-run handler. The model proposes; my code decides.

If you want the model to react to the result — say, to confirm the booking back to the user in natural language — feed the return value back as a function_response in a follow-up generate_content call. That's the agent loop: parse, call, respond.

Same mechanism, two postures

Structured output and function calling run on the same constrained decoder. Structured output says "the next thing you emit is JSON of this shape." Function calling says "the next thing you emit is a call to this function with these arguments, which is also JSON of a known shape." Pick the framing that matches what the output represents. If it's data you'll process, use response_schema. If it's an action the system should take, use tools.

Instructor: Pydantic-native structured outputs

Instructor is a Python library that patches the OpenAI client (and friends) so calls return Pydantic models instead of strings. You define your output type and Instructor handles schema generation, validation, and retries.

import instructor
from pydantic import BaseModel
from openai import OpenAI

client = instructor.from_openai(OpenAI())

class ContactInfo(BaseModel):
    name: str
    email: str
    phone: str | None = None
    company: str | None = None

contact = client.chat.completions.create(
    model="gpt-4o",
    response_model=ContactInfo,
    messages=[{
        "role": "user",
        "content": "Extract: John Smith, [email protected], works at Acme Corp"
    }]
)

print(contact.name)   # "John Smith"
print(contact.email)  # "[email protected]"

Your output is a real Python object with validation, defaults, and type hints. When the model returns invalid data, Instructor retries with the validation error in the next turn, so the model has a fair shot at fixing itself. This is the path I reach for in Python codebases when I'm not already on the Gemini SDK.

Outlines: grammar-level control for open models

Outlines does constrained decoding for open-weight models you host yourself. It compiles a JSON Schema or regex into a finite-state machine that walks the model through generation token by token. Real logit masking, no retries — the invalid tokens are never sampled in the first place.

import outlines

model = outlines.models.transformers("mistralai/Mistral-7B-v0.1")

schema = '''{
    "type": "object",
    "properties": {
        "sentiment": {"type": "string", "enum": ["positive", "negative", "neutral"]},
        "confidence": {"type": "number", "minimum": 0, "maximum": 1}
    },
    "required": ["sentiment", "confidence"]
}'''

generator = outlines.generate.json(model, schema)
result = generator("Review: The food was excellent but the service was slow.")
# result is guaranteed to have sentiment in [positive, negative, neutral]
# and confidence as a float between 0 and 1

Trade-offs

No single approach wins everywhere.

API-level structured outputs are the easiest to reach for and the latency is usually close to plain generation. The trade is that you're locked into a provider, and schema coverage varies between them.

Function calling crosses providers and fits naturally into agent code. The catch is that framing every output as a "tool call" feels forced when you only want to pull a few fields out of a string.

Instructor wins on developer experience for Python teams. The retry loop handles edge cases so I'm not writing the same try/except again. The cost is some extra latency on retries and a hard dependency on the underlying JSON mode.

Outlines gives the strongest guarantee: zero retries, masking at the token level. The price is that you host the model yourself, and complex schemas pay a compilation cost plus a small per-token slowdown.

When it actually matters

Structured outputs aren't always necessary. If you're streaming prose into a chat UI, free-form text is fine. The moment LLM output feeds something else — a database write, an API call, a UI component, a downstream model — you need structure.

In production the LLM is almost never the last stop. It's a processing node in a pipeline, and pipelines need contracts. Structured outputs are how you write those contracts in a way the model can't violate.

The tooling is mature enough now that regex-parsing LLM output in production is a choice, not a necessity. Pick one of the methods above, define your schemas, and let the model spend its capacity on the actual content. Format is a solved problem. There's no good reason to keep solving it in your application code.