The Rise of Structured Generation: From JSON Mode to Grammar-Constrained Decoding

Asking nicely for JSON works most of the time. Grammar-constrained decoding works all of the time.

I've spent a lot of cycles cleaning up "almost-JSON" from LLMs. Trailing commas. Missing brackets. A stray null where there should be a string. The fix isn't a better prompt. The fix is making the model physically incapable of emitting invalid output. That's what structured generation does.

The spectrum of "structured output"

Four levels of guarantee, from weakest to strongest:

Prompt engineering — "Reply only with JSON, no other text." Works around 80% of the time on a good model. Fails in interesting ways: the model wraps its JSON in a code fence, or apologizes first, or hits a token limit and truncates mid-object.

JSON mode — a flag on the API (response_format={"type": "json_object"} on OpenAI, equivalent on others) that biases the model toward JSON. The output is parseable JSON, but the schema is up to the prompt. You still don't know if it returned the keys you asked for.

Function calling / tool use — the API takes a JSON schema and the model returns arguments matching that schema. The keys are guaranteed. The types are mostly guaranteed. Edge cases (unioned types, deeply nested schemas) still slip through depending on the provider.

Grammar-constrained decoding — at every generation step, the model can only sample tokens that keep the output valid against a formal grammar. Invalid tokens are masked out before sampling. The output is provably correct by construction.

Each level is more constrained and more reliable. Each one shifts work from "validate after the fact" to "make invalid impossible." The fourth level is where I want production code to live.

How grammar-constrained decoding works

The mental model is simple. The model produces a probability distribution over the vocabulary at each step. Without constraints, it samples from that distribution directly. With constraints, you mask out every token that would make the output invalid, then renormalize and sample from what's left.

The grammar tells you what's valid. It can be regex, BNF, JSON Schema, or any formalism that can be compiled into a finite-state machine (FSM). At each generation step, the FSM tracks which states are reachable. Tokens that would lead to an unreachable state get masked.

Concretely: if the grammar says the next token must be a digit, and the vocabulary has 50,000 tokens, you set the logits of all non-digit tokens to negative infinity before sampling. The model can only pick a digit. There's no failure mode where it picks something else, because something else is mathematically unreachable.

# Pseudo-code for one step of constrained generation
logits = model.forward(input_ids)               # raw token logits
allowed = grammar_fsm.allowed_tokens(state)     # which tokens keep the output valid
mask = build_mask(vocabulary, allowed)          # -inf for disallowed
masked_logits = logits + mask                   # disallowed → -inf
next_token = sample(softmax(masked_logits))     # sample from valid tokens only
state = grammar_fsm.advance(state, next_token)  # update state for next step

That's the whole idea. The math is rejection sampling done at the token level, before sampling instead of after. The model never gets a chance to be wrong.

The libraries that implement it

Three open-source libraries are doing serious work here.

Outlines (Python) is the one I reach for first. It compiles regex patterns and JSON schemas to FSMs, then runs them against any HuggingFace model or compatible inference engine. The API is small:

import outlines
from pydantic import BaseModel

class Person(BaseModel):
    name: str
    age: int
    email: str

model = outlines.models.transformers("meta-llama/Llama-3-8B-Instruct")
generator = outlines.generate.json(model, Person)

result = generator("Generate a person with realistic data.")
# result is a Person instance, guaranteed

Guidance (Microsoft) takes a more declarative approach. You write a template with constrained slots, and the library handles the FSM construction. Useful when you want to interleave fixed text with constrained generation.

LMQL (Language Model Query Language) is closer to SQL for prompts. You write a query with type constraints, and it compiles to a constrained generation plan. Heavier abstraction, but powerful for complex multi-step prompts.

For most cases, Outlines is enough. The other two are worth knowing exist.

What it costs you

The conventional wisdom used to be that constraining a model would cripple its quality. The model "wants" to say something else, and forcing it into a grammar would produce stilted, low-quality output.

That turned out to mostly be wrong. Recent research (and my own benchmarks) shows that grammar-constrained decoding has minimal quality impact on tasks where the grammar is the right shape. JSON output gets slightly better because the model isn't fighting itself on syntax. Free-form natural language gets worse if you over-constrain it. Match the grammar to the task and you're fine.

The performance cost is real but smaller than you'd expect. The mask construction adds maybe 5-15% latency depending on the FSM complexity. For complex JSON Schemas, that can creep up to 25%. Still nothing close to the cost of regenerating a malformed response.

The integration cost is the bigger thing. You need a library that hooks into the model's decoding loop. That's easy with vLLM, llama.cpp, and HuggingFace Transformers. It's harder (sometimes impossible) with hosted APIs that don't expose the logits.

JSON Schema as the lingua franca

JSON Schema has won as the way developers describe what they want. Every major library can compile a JSON Schema into an FSM. The schema works as documentation, validation, and constraint, all from one source of truth.

schema = {
    "type": "object",
    "properties": {
        "title": {"type": "string", "minLength": 1},
        "tags": {
            "type": "array",
            "items": {"type": "string"},
            "maxItems": 5
        },
        "publish_date": {
            "type": "string",
            "format": "date"
        }
    },
    "required": ["title", "tags"]
}

generator = outlines.generate.json(model, schema)
result = generator("Suggest a blog post about distributed databases.")
# Output: {"title": "...", "tags": [...], "publish_date": "..."}

This composes nicely with everything else in a typical Python project. Pydantic models compile to JSON Schema. FastAPI uses the same schemas. You can take a Pydantic model from your API layer, hand it to Outlines, and get LLM output that drops directly back into your API response.

Beyond JSON

Once you have FSM-based decoding, JSON is just one application. Anything you can describe with a grammar, you can constrain.

Valid SQL — define a SQL grammar, get queries that always parse. Useful for text-to-SQL systems where invalid queries cost you a database round trip.

Valid Python — define Python's grammar, get code that always parses. Doesn't guarantee correctness, but eliminates an entire failure class.

Valid YAML, XML, HTML — same pattern, different grammars.

Domain-specific languages — if you have a custom config language, an internal DSL, or a structured query language, you can write its grammar and constrain the LLM to produce only valid expressions.

I've seen teams use this for RegEx generation, where the model was constrained to produce only valid regular expressions. The error rate dropped from "occasional invalid output" to zero.

Inference engine integration

vLLM ships with structured output support out of the box. You pass a JSON schema or regex and it handles the rest:

from vllm import LLM, SamplingParams
from pydantic import BaseModel

class Movie(BaseModel):
    title: str
    year: int

llm = LLM(model="meta-llama/Llama-3-8B-Instruct")
params = SamplingParams(
    guided_json=Movie.model_json_schema(),
    max_tokens=200
)

output = llm.generate("Recommend a movie about space.", params)
# output is guaranteed to parse as Movie

llama.cpp has GBNF grammar files for the same purpose. You write a grammar and the engine enforces it during generation.

OpenAI added "Structured Outputs" mode in 2024 that's roughly equivalent — pass a schema, get guaranteed-conforming output. It's not labeled grammar-constrained decoding, but functionally that's what they're doing under the hood.

When NOT to constrain

Grammar-constrained decoding isn't the right tool for everything.

Creative writing. If you're generating prose, poetry, or anything where free-form expression matters, don't constrain it. The grammar would have nothing useful to say about good prose.

Open-ended generation. Brainstorming, summarization, exploratory research — tasks where you don't know the shape of the output up front. You can't write a grammar for "interesting ideas."

Conversational AI. Chat responses are where you want the model's full expressiveness. Constrain the tool calls and tool arguments, leave the conversation free.

The pattern I follow: constrain at the boundaries where structure matters (tool calls, structured data, code generation). Leave the model free in the middle where reasoning and language quality matter.

The bigger shift

Structured generation is part of a broader move from "LLMs are unpredictable, build robust validation around them" to "LLMs can be made deterministic in shape, even if non-deterministic in content." The shape gets locked. The content stays creative. That separation is what makes LLMs production-ready for things they couldn't do before.

If you're building a tool-using agent, a code-generating assistant, or anything that needs LLM output to drop into a downstream system, this should be on by default. The cost is minimal. The reliability gain is enormous. There's no longer a good reason to ship a system that hopes the model returns valid JSON.