Imagine a source says a person is 34, and an extraction model returns {"name": "Alice", "age": 43, "email": "[email protected]"}. The JSON parses. The fields have the expected types. The age is wrong. This is an illustrative example, but it captures a distinction that matters whenever a model feeds another program: valid structure and correct information are separate achievements.

Grammar-constrained decoding gives us control over the first. It prevents the model from choosing continuations that violate an allowed output structure. That's a useful improvement over cleaning up trailing commas and missing brackets after generation. Understanding exactly how it works also explains why it can't repair Alice's age.

Stop the invalid choice before it happens

A language model assigns scores, called logits, to possible next tokens. A token may contain one character, part of a word, or several characters. The decoder turns those scores into probabilities and chooses what comes next.

Constrained decoding adds a check before that choice. Given the text already generated, which tokens could still lead to an output that satisfies the grammar? The decoder masks the other tokens by assigning them negative infinity, giving them zero probability after normalization. The model then chooses among the remaining options.

Consider the partial output {"age": when the schema requires an integer. A token containing 43 could be allowed. A token containing "unknown" could not. Both 34 and 43 satisfy the integer requirement, so the grammar has no reason to prefer the correct age. It knows what an integer looks like; it doesn't know Alice.

This pseudocode shows one decoding step. grammar stands for the constraint engine, whose implementation depends on the grammar being enforced:

# Pseudo-code for one step of constrained generation logits = model.forward(input_ids) # raw token logits allowed = grammar.allowed_tokens(state) # keep a valid completion possible mask = build_mask(vocabulary, allowed) # -inf for disallowed masked_logits = logits + mask # disallowed → -inf next_token = sample(softmax(masked_logits)) # sample from valid tokens only state = grammar.advance(state, next_token) # update state for next step

The distinction between a valid prefix and a complete answer matters here. {"age": is a permissible beginning, although it isn't a JSON document yet. Generation must reach an allowed end state. A token budget can interrupt the process before that happens.

This is masking before sampling. Calling it rejection sampling would suggest drawing an invalid token and trying again. Here, the invalid choice is removed before the draw. The remaining probabilities still belong to the model, which is why constraining structure doesn't make the content deterministic.

A grammar needs the right machinery

A finite-state machine, or FSM, tracks a position in a set of permitted transitions. It's a good fit for regular patterns such as a restricted identifier. General nested structures need more expressive machinery. An arbitrarily nested object can contain another object, which can contain another; the decoder needs to remember which structures remain open.

Context-free grammars describe that kind of nesting. A grammar engine can use parsing state or a stack to track it. BNF is a notation for writing grammar rules, not a promise that every grammar becomes an FSM. Outlines documents context-free grammar output, and llama.cpp's GBNF guide describes its rule-based grammar format.

The token boundary adds another complication. The checker must consider all the characters a token contributes, including tokens that cross a punctuation boundary. Restricting output to digits isn't generally as simple as identifying ten vocabulary entries. This is why a decoder integration is useful: it connects the grammar's rules to the model's actual tokenizer.

What each output option promises

Prompting, JSON mode, tool calling, and constrained decoding describe different parts of the system. They aren't four universal reliability levels.

A prompt such as "reply only with JSON" asks the model to follow a convention. It doesn't itself remove any token choices. How often it succeeds depends on the task and model.

JSON mode targets JSON syntax. A schema describes further requirements, such as an integer age and a required name. Tool calling describes how a model proposes a function and its arguments. Whether those arguments are strictly constrained depends on the provider and configuration.

For example, OpenAI distinguishes JSON mode from strict Structured Outputs and documents both refusals and incomplete responses. Its supported schema features also have limits. A caller must handle those outcomes instead of assuming every request returns a usable object. OpenAI's Structured Outputs guide explains the contract.

Grammar-constrained decoding is a mechanism for enforcing supported structure during generation. It can sit underneath a structured response or a tool call. The guarantee concerns a completed output under the implemented constraints. It doesn't cover facts that those constraints never expressed.

Give the decoder a useful schema

JSON Schema is useful because the same description can inform generation and validation. The example below asks for a blog-post suggestion: a nonempty title, up to five string tags, and an optional publication date.

With an initialized Outlines model, pass the schema as an output type. The companion example below shows model initialization. The Outlines migration guide documents this interface.

from outlines.types import JsonSchema schema = { "type": "object", "properties": { "title": {"type": "string", "minLength": 1}, "tags": { "type": "array", "items": {"type": "string"}, "maxItems": 5 }, "publish_date": { "type": "string", "format": "date" } }, "required": ["title", "tags"] } result = model( "Suggest a blog post about distributed databases.", output_type=JsonSchema(schema), max_new_tokens=200, ) # result contains JSON text; validate it before using it.

Read the schema as carefully as the output. publish_date isn't in required, so an answer may omit it. Declaring format: "date" also requires checking whether the chosen backend enforces that keyword. A generated date can satisfy a format without being the right date to publish anything.

Schema support varies. llama.cpp explicitly supports conversion of a subset of JSON Schema to GBNF. Treat portability as something to test with the constraints you actually use. A schema can document a requirement that a particular generation backend doesn't enforce.

The same idea works with a Pydantic model. This companion example initializes the model and validates the returned text:

import outlines from pydantic import BaseModel from transformers import AutoModelForCausalLM, AutoTokenizer class Person(BaseModel): name: str age: int email: str model_id = "meta-llama/Llama-3-8B-Instruct" model = outlines.from_transformers( AutoModelForCausalLM.from_pretrained(model_id), AutoTokenizer.from_pretrained(model_id), ) result = model( "Generate a fictional person with realistic data.", output_type=Person, max_new_tokens=200, ) person = Person.model_validate_json(result)

Here, email: str accepts a string; it doesn't establish that the address exists. And "realistic data" asks for a plausible fictional person, so it would be a mistake to treat the result as a verified identity.

The Transformers integration returns JSON text; Person.model_validate_json turns a valid result into a Python object. These examples require model access, compatible dependencies, and enough memory, and haven't been executed for this article. Reusing a Pydantic model across generation and an API layer is convenient, but validation must still handle incomplete or otherwise unusable responses.

Put the constraint inside the inference engine

For a local deployment, the engine can own the decoding loop. The following movie-extraction example uses the documented StructuredOutputsParams interface. The older guided_json field was removed in vLLM 0.12.0. vLLM's structured-output documentation covers the current configuration.

from vllm import LLM, SamplingParams from vllm.sampling_params import StructuredOutputsParams from pydantic import BaseModel class Movie(BaseModel): title: str year: int llm = LLM(model="meta-llama/Llama-3-8B-Instruct") params = SamplingParams( structured_outputs=StructuredOutputsParams(json=Movie.model_json_schema()), max_tokens=200 ) output = llm.generate("Recommend a movie about space.", params) # Inspect output[0].outputs[0].finish_reason before accepting the result. movie = Movie.model_validate_json(output[0].outputs[0].text)

This is a documentation-aligned example, not a recorded benchmark or a claim that this model was run for the article. The token limit remains significant: a partial completion can still fail validation. Even a completed Movie object establishes only a title string and an integer year. Checking that the film exists, has that release year, and concerns space requires additional evidence.

Hosted APIs move the integration boundary. You usually cannot install your own token mask inside a provider's decoder; you select the constraints that its API exposes. With local engines, grammar support, tokenizer compatibility, and model setup become part of your deployment work.

What the constraint costs

Masking adds work, and the shape of that work depends on the implementation. Schema compilation, repeated token checks, caching, output length, and batching can all affect the comparison. Even equivalent grammar expressions can have different costs: llama.cpp warns that some ways of writing optional repetitions make sampling extremely slow.

The useful comparison is a complete task under the same acceptance rules. Measure time to a valid, usable result, including retries, and evaluate whether the content still answers the question. Constraining a classification field to three allowed labels prevents a fourth label; it cannot make an unsuitable set of labels describe the input well. If the schema has no way to express missing information, the model may fill a required field with something plausible.

I'd use constrained generation when a downstream program needs a predictable shape, then test the actual schema and workload. I wouldn't budget around a universal overhead percentage.

Beyond JSON, with the same boundary

A SQL grammar can restrict output to statements that parse under that grammar. It won't establish that a column exists in your database or that a query should be permitted to run. A Python grammar can constrain syntax without proving the program's behavior. YAML, XML, HTML, and internal configuration languages offer similar opportunities when an appropriate grammar and compatible engine are available.

Regular-expression generation is another example: grammar rules can prevent malformed expressions, but a valid expression can still match the wrong text. Reaching zero syntax errors in a completed sample would still leave that matching behavior to test.

For prose, decide what structure would help the consumer. A summary can live inside a JSON field without constraining every sentence's wording. A conversation can remain free text while tool arguments follow a schema. A grammar won't tell you whether a joke is funny or an explanation is clear, so those aren't useful promises to demand from it.

The library options are worth knowing without turning the mechanism into a shopping list. Outlines provides typed and grammar-based generation interfaces. Guidance combines fixed text with constrained generation. LMQL expresses constraints within a query language for prompts. Their interfaces and supported backends differ; choose around the constraints you need to enforce.

Know which error you have prevented

Return to Alice's age. A grammar can prevent "unknown" from appearing where an integer is required. It can keep punctuation and fields within the supported schema. That saves downstream software from handling malformed answers as ordinary results. It leaves the choice between 34 and 43 to a model that still needs evidence.

I separate the final checks accordingly: syntax and schema validation establish the shape; semantic checks establish whether the values answer the task; authorization checks establish whether a proposed action may run. The production structured-output article develops the application side of that boundary.

Use the decoder to eliminate structural choices your program cannot accept. Then verify the information and permissions that the grammar cannot know. You get a more dependable interface, with a clear explanation of which failures it prevents and which still belong to the application.