Ask an LLM to generate a JSON object and it will eventually emit a trailing comma, an unescaped newline inside a string, or a bare word where a quoted key should be. The error is not a bug in the model. It is a consequence of how autoregressive sampling works: at every step, the model sees every token in its vocabulary as a candidate, including tokens that would make the partial output syntactically invalid.
You can fix this by constraining the token vocabulary itself. Instead of letting the model choose from thirty-two thousand tokens, give it only the subset that keeps the partial parse in a valid state. The technique is called grammar-constrained decoding, and it turns an LLM from a probabilistic text generator into a syntax-aware code synthesizer.
Why prompt engineering cannot solve syntax validity
Most developers start with the obvious approach: add “Output valid JSON only” to the system prompt, repeat the schema in the user message, and hope. This helps, but it is not a guarantee.
The model does not parse its own output while generating it. It predicts the next token based on the probability distribution over the vocabulary, conditioned on the prompt and everything generated so far. A token like , might have high probability after a closing brace because commas often appear in that position in the training data. Whether the comma is syntactically legal at this exact point in this specific JSON object is not part of the model’s decision process.
Even fine-tuned models with Reinforcement Learning from Human Feedback (RLHF) only shift the probability mass toward generally correct patterns. They do not eliminate invalid paths. If you need a guarantee, not a high probability, you need to change the sampling mechanism itself.
How grammar-constrained decoding works
The core idea is simple: maintain a parser state alongside the LLM’s generation, and at each token position, mask out every token that would transition the parser into an error state.
At step t, the model produces a logits vector over the entire vocabulary. Normally you apply temperature scaling and sample. With constrained decoding, you first ask a grammar engine: given the tokens generated so far, which tokens are legal next? The engine returns a bitmask over the vocabulary. You set the logits of illegal tokens to negative infinity, run softmax over the remainder, and sample from the filtered distribution.
The grammar engine does not need to parse the full output after the fact. It parses incrementally. After each token is accepted, it updates its internal state machine. When the parser reaches an accepting state, generation can stop. When it is in a non-accepting state, generation continues.
This means the model can still choose freely among all syntactically valid continuations. It retains creativity inside the grammar. It just cannot step outside it.
What the implementation looks like
Here is a simplified Python sketch using lark, a modern parsing library, to build a grammar mask. In production you would use a faster engine like outlines, guidance, or llama.cpp’s built-in GBNF support, but the principle is identical.
from lark import Lark, Token
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
# Define a tiny grammar for a simple config DSL.
grammar = r"""
start: pair+
pair: KEY "=" VALUE
KEY: /[a-z_]+/
VALUE: /"[^"]*"/
"""
parser = Lark(grammar, parser="lalr")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")
def legal_next_tokens(partial_text: str) -> set[int]:
"""Return token IDs that do not cause a parse error."""
legal = set()
for token_id in range(tokenizer.vocab_size):
candidate = tokenizer.decode([token_id], skip_special_tokens=True)
try:
# Try parsing partial_text + candidate as a prefix.
parser.parse(partial_text + candidate)
legal.add(token_id)
except Exception:
# Some exceptions are expected mid-parse.
# A production engine tracks parser state, not exceptions.
pass
return legal
# Generate one token at a time with grammar masking.
prompt = "Generate a config: "
input_ids = tokenizer.encode(prompt, return_tensors="pt")
generated = input_ids.clone()
for _ in range(50):
outputs = model(generated)
logits = outputs.logits[:, -1, :]
partial = tokenizer.decode(generated[0], skip_special_tokens=True)
legal_ids = legal_next_tokens(partial)
mask = torch.full_like(logits, float("-inf"))
for tid in legal_ids:
mask[0, tid] = logits[0, tid]
next_token = torch.argmax(mask, dim=-1).unsqueeze(0)
generated = torch.cat([generated, next_token], dim=-1)
# Stop if the parser is in an accepting state.
if parser.parse(partial):
break
print(tokenizer.decode(generated[0], skip_special_tokens=True))This is a naïve implementation. A real system does not iterate over the entire vocabulary on every step. It builds a finite-state automaton from the grammar ahead of time and precomputes which vocabulary tokens are valid from each automaton state. At generation time, the mask is a single table lookup.
Where constrained decoding fits in a bounded context
Bounded contexts in Domain-Driven Design are defined by their ubiquitous language. That language is often formalized as a small DSL: a query syntax, a rule grammar, a config format, or an expression language. When you ask an LLM to generate or edit artifacts inside that context, you want it to speak the language correctly.
Grammar constraints enforce the boundary. The LLM cannot invent fields that do not exist in your DSL, cannot emit mismatched brackets, and cannot produce values outside the allowed enum set. The syntax becomes a compile-time guarantee, not a runtime prayer.
This is especially valuable when the LLM output feeds directly into a parser or interpreter. If your DSL is parsed by a strict recursive descent parser, a single unexpected token aborts the entire pipeline. Constrained decoding removes that failure mode.
The trade-offs you should know about
Constrained decoding is not free. The overhead depends on how you implement the grammar engine.
Throughput. Building the mask on every forward pass adds latency. Fast engines like outlines or llama.cpp with GBNF precompute token-to-state mappings, reducing the per-step cost to roughly 5-10% overhead. Naïve implementations that reparse on every candidate token can slow generation by orders of magnitude.
Grammar expressiveness. Not every syntax is easy to express in a grammar that maps cleanly to token masks. Context-sensitive rules, like “this identifier must be declared earlier in the scope,” are not captured by standard context-free grammars. You can constrain syntax with a grammar. You cannot constrain semantics without additional machinery.
Model compatibility. Some inference APIs, especially hosted cloud APIs, do not expose logits or allow custom masking. OpenAI’s API offers JSON mode, which is a hardcoded grammar constraint for JSON specifically, but you cannot bring your own grammar. For custom DSLs, you typically need to run inference locally or use a framework that exposes the logits.
Partial token problems. A token boundary does not always align with a grammar boundary. A grammar might expect a quoted string, but the next token could be "hel followed by lo". The grammar engine must reason about partial token matches, not just whole tokens. Production libraries handle this by mapping tokens to character prefixes and checking prefix validity against the grammar.
How to actually implement this today
You do not need to write a grammar engine from scratch. Several libraries handle the hard parts.
outlines is the most approachable for Python users. You define a Pydantic model or a regular expression, and it compiles the constraint into an efficient finite-state automaton that integrates with Hugging Face transformers and vLLM.
from outlines import models, generate
model = models.transformers("microsoft/Phi-3-mini-4k-instruct")
generator = generate.regex(model, r'\{[a-z_]+\}')
result = generator("Extract the key: ")guidance from Microsoft offers a richer templating system. You interleave grammar constraints with prompt templates, and it handles the masking internally.
llama.cpp supports GBNF (GGML BNF), a BNF-like grammar format. You pass a .gbnf file at inference time, and the engine enforces it at the C++ level. This is the fastest option for local inference.
jsonformer and instructor are lighter alternatives specifically for JSON. They use grammar-like constraints but are restricted to JSON schema. If your bounded context DSL happens to be JSON-shaped, they are the easiest starting point.
What constrained decoding does not fix
A grammar-constrained LLM can still generate semantically wrong output. It can emit a valid SQL query that references a nonexistent table, or a valid config that sets an invalid port number. Syntax is a necessary guardrail, not a sufficient one.
You still need validation layers below the LLM. Parse the output, type-check it against your domain model, and reject or retry if the semantics are wrong. Constrained decoding reduces the failure modes from “syntax error” to “logic error.” That is a big improvement, but it is not a free pass.
Also, grammar constraints do not make the model smarter. If the grammar is too permissive, the model can still wander into nonsensical but syntactically valid territory. If the grammar is too restrictive, the model has no valid path to express a correct answer, and you get low-probability garbage or repetitive loops. Designing the grammar is part of the engineering work.
Start with the output format, not the model
Before you tune the prompt or swap to a larger model, ask whether the problem is really about reasoning or about syntax. If the LLM understands what you want but occasionally formats it wrong, grammar constraints are the right fix. They are cheaper than fine-tuning, more reliable than prompt engineering, and they give you a guarantee that sampling alone cannot.
Define the grammar for your bounded context DSL, plug it into a constrained decoder, and let the model generate inside the lines. The output will still surprise you occasionally, but it will never be a syntax error.
FAQ
What is grammar-constrained decoding?
Grammar-constrained decoding is a technique that filters an LLM’s vocabulary at each generation step using a formal grammar. Only tokens that keep the partial output syntactically valid are considered, guaranteeing that the final output conforms to the grammar.
Does constrained decoding reduce output quality?
Not for syntax-bound tasks. The model still chooses freely among all grammatically valid continuations. For open-ended creative writing, constraints would harm quality. For code, config, or DSL generation, they improve both correctness and reliability.
Can I use this with OpenAI or Claude APIs?
OpenAI offers JSON mode, which is a built-in grammar constraint for JSON. You cannot bring your own grammar. Anthropic does not currently expose grammar constraints. For custom DSLs, you typically need local inference with vLLM, llama.cpp, or a similar engine.
What grammar formats are supported?
Common formats include EBNF, BNF, PEG, and GBNF. Libraries like outlines accept regular expressions and Pydantic models. llama.cpp uses GBNF. Choose the format that your inference engine supports.