JSON Mode vs Structured Outputs Explained

LLMs generate text, not data. If you've ever asked GPT-4 for "JSON only, no commentary" and watched it return a polite paragraph followed by markdown-fenced JSON with a trailing comma, you've felt the problem.

Two features fix this, badly named so often confused: JSON mode and Structured Outputs. They sound interchangeable. They are not.

The Core Difference

• JSON mode guarantees the output parses as JSON. Nothing more.
• Structured Outputs guarantees the output parses as JSON and matches your schema. Field names, types, enums, required fields — all enforced at the token level.

If you ask for {"name": "string", "age": "number"} under JSON mode, you might get {"full_name": "Ada", "years": "thirty-six"}. Valid JSON, wrong shape. Under Structured Outputs, you get exactly the keys and types you declared.

JSON Mode: The 2023 Generation

OpenAI shipped JSON mode in November 2023. Mistral, Together, Groq, and most open-weight inference stacks followed. Activating it (typically response_format: {"type": "json_object"}) does three things:

1. Constrains decoding so the output is valid JSON.
2. Refuses to start generation if your prompt doesn't mention "JSON" somewhere (a guardrail against confused models).
3. May truncate or error out if generation hits max_tokens mid-object.

What it does not do is enforce a schema. The model still freelances on keys, capitalizes inconsistently, and occasionally returns numbers as strings. Production code has to validate with Zod, Pydantic, or hand-rolled type checks, then retry on failure. That retry loop is where engineering time goes.

Structured Outputs: The 2024 Upgrade

OpenAI shipped Structured Outputs in August 2024. Anthropic and Google followed with their own takes (covered below). The mechanism is fundamentally different: instead of asking the model to behave, the inference engine constrains every token so the output cannot deviate from the schema.

Under the hood:

1. You provide a JSON Schema (or a Pydantic / Zod model the SDK converts).
2. The engine compiles the schema into a finite state machine (or context-free grammar).
3. At each decoding step, the FSM masks out any token that would produce invalid JSON or violate the schema. The model's logits over allowed tokens are renormalized.
4. Sampling proceeds only over the allowed set.

Result: 100% schema compliance, by construction. There is no parse failure to retry.

This technique — grammar-restricted decoding — was pioneered in open-source by libraries like outlines, guidance, and llama.cpp's GBNF grammars before the big providers adopted it.

Anthropic's Approach: Tool Use as Structured Output

Anthropic doesn't ship a separate "structured outputs" toggle. Instead, the recommended pattern is to wrap the desired schema in a tool definition and use tool_choice: {"type": "tool", "name": "extract"} to force the model to "call" that tool. The input of the tool call is your structured object.

This works because tool inputs already require schema-conformant JSON. It's structured outputs by another name. See function calling vs tool use for the broader picture.

Supported Schema Features

Constrained decoding can't handle every JSON Schema feature. Providers diverge on what they support:

Feature	OpenAI Strict	Anthropic (via tools)	Gemini	Open-source (outlines)
type, properties	Yes	Yes	Yes	Yes
required	Yes (all required)	Yes	Yes	Yes
enum	Yes	Yes	Yes	Yes
oneOf / anyOf	Limited	Yes	Limited	Yes
$ref recursion	Limited	Yes	No	Yes
additionalProperties	Must be false	Flexible	Flexible	Flexible
Regex patterns	No	No	No	Yes
Min/max length	No	Schema-validated post-hoc	No	Yes
Nested depth limit	5 levels	Flexible	5 levels	Configurable
Latency overhead	~5-10%	~5%	~5-10%	Varies

OpenAI's strict mode is the most restrictive: every property must be in required, additionalProperties: false is mandatory, and the first call with a new schema is slower (it compiles and caches the grammar).

When to Use Which

• Pure free-form JSON, schema flexibility needed → JSON mode.
• Data extraction with a known shape → Structured Outputs / tool use.
• Function arguments for an agent → Tool use (which is already structured).
• Classification into fixed labels → Structured Outputs with an enum.
• Cross-provider portability matters → Tool-use pattern (works everywhere) over provider-specific strict modes.

Real-World Use Cases in 2026

Customer Support Ticket Extraction

A support team receives emails, call transcripts, and chat logs. They need to extract tickets with a fixed schema: customer_name, issue_category, priority, account_id. Using Structured Outputs guarantees that every extraction matches the schema — no missing fields, no wrong types. Without it, you'd need retry logic and custom validation. With it, you get 100% schema compliance, no retries.

Why it matters: Support tickets feed into a downstream workflow (routing, SLA tracking, knowledge base updates) that depends on schema compliance. One extraction with a wrong type breaks everything.

Multi-Turn Agentic Workflows

An AI agent needs to decide what function to call at each step. It looks at the current state and generates the next action — something like:

{
  "action": "send_email",
  "recipient": "alice@company.com",
  "subject": "Status Update",
  "body": "..."
}

Structured Outputs ensures the agent always generates valid function calls. Without it, the agent might generate {"action_type": "email"} (wrong key), or include extra fields that break the workflow.

Dynamic Classification at Scale

A fintech platform receives thousands of transactions per second. Each needs to be classified as fraudulent, normal, or pending_review. Structured Outputs with an enum field guarantees only those three values. If you relied on JSON mode and the model hallucinated "possibly_fraud" or "yes", your downstream classifiers would break.

Privacy-Sensitive Extraction

A healthcare provider uses an LLM to extract diagnoses and medications from clinical notes. The output schema includes fields like icd_code and medication_name, which then feed into a secure database. Structured Outputs ensures the schema is respected at the protocol level — the LLM cannot accidentally emit unstructured text that bypasses your validation.

Why JSON Mode Still Exists

If Structured Outputs is strictly better, why keep JSON mode? Several reasons:

1. Cost and latency — older models don't support Structured Outputs yet.
2. Flexibility in dev — JSON mode allows "almost JSON" that you can clean up later with retry logic.
3. Portability — JSON mode is supported by every LLM provider; Structured Outputs is newer and provider-specific.
4. Schemas too complex — if your schema has deeply recursive structures or complex validation logic, Structured Outputs might reject it; JSON mode lets you handle edge cases in application code.

For greenfield projects, the recommendation is: use Structured Outputs. For legacy systems or cross-provider portability, JSON mode is still the safer bet.

Python Example: OpenAI Structured Outputs

from openai import OpenAI
from pydantic import BaseModel

class Article(BaseModel):
 title: str
 author: str
 word_count: int
 tags: list[str]

client = OpenAI()
resp = client.beta.chat.completions.parse(
 model="gpt-4o-2024-08-06",
 messages=[
 {"role": "system", "content": "Extract article metadata."},
 {"role": "user", "content": "The 1200-word piece 'Bun vs Node' by J. Doe, tagged runtime, javascript."}
 ],
 response_format=Article,
)

article = resp.choices[0].message.parsed
# article.title == "Bun vs Node"
# article.word_count == 1200

.parsed returns a typed Pydantic instance. No try/except json.JSONDecodeError. No retry loop.

Python Example: Anthropic Tool-Based Extraction

import anthropic
client = anthropic.Anthropic()

tool = {
 "name": "extract_article",
 "description": "Extract article metadata.",
 "input_schema": {
 "type": "object",
 "properties": {
 "title": {"type": "string"},
 "author": {"type": "string"},
 "word_count": {"type": "integer"},
 "tags": {"type": "array", "items": {"type": "string"}}
 },
 "required": ["title", "author", "word_count", "tags"]
 }
}

resp = client.messages.create(
 model="claude-opus-4-7",
 max_tokens=512,
 tools=[tool],
 tool_choice={"type": "tool", "name": "extract_article"},
 messages=[{"role": "user", "content": "..."}],
)

data = resp.content[0].input # already a dict matching the schema

The Instructor Library

instructor (by Jason Liu) is the most popular Python wrapper. It patches the OpenAI / Anthropic / Gemini clients to accept a Pydantic model as response_model= and handles retries-on-validation-failure transparently. For older models that lack native Structured Outputs, instructor falls back to JSON mode + Pydantic validation + automatic retry. Roughly:

import instructor
from openai import OpenAI
from pydantic import BaseModel

client = instructor.from_openai(OpenAI())

class User(BaseModel):
 name: str
 age: int

user = client.chat.completions.create(
 model="gpt-4o-mini",
 response_model=User,
 messages=[{"role": "user", "content": "Ada, 36"}],
)

Latency and Cost

Constrained decoding adds 5-10% to per-token latency in our measurements (matches OpenAI's published numbers from the August 2024 launch). First request with a new schema is slower — typically 1-3 extra seconds — because the grammar is compiled. Subsequent requests with the same schema are cached. No extra token cost: you pay for the same input/output tokens, just nudged into legal shapes. Gateways such as LiteLLM can also put one OpenAI-compatible API in front of many providers.

Common Use Cases

• Data extraction from PDFs, emails, support tickets.
• Classification with confidence scores.
• SQL generation with a constrained subset (column names as enums).
• Function arguments in agentic loops — see AI agents.
• Knowledge graph triples for RAG indexing — see articles on retrieval-augmented generation.

Frequently Asked Questions

Q: Does JSON mode guarantee my fields will be present?
A: No. JSON mode only guarantees the output parses as JSON. Field presence, names, and types are unconstrained. Use Structured Outputs or tool use for that guarantee.

Q: Does Structured Outputs cost extra tokens?
A: No. You pay the same per-token rate. Expect 5-10% extra latency from the grammar enforcement, but no token surcharge.

Q: Can I use Structured Outputs with streaming?
A: Yes for OpenAI and Anthropic — partial parses arrive as the stream progresses. You can hydrate a Pydantic model incrementally with libraries like instructor.

Q: What if my schema has features the provider doesn't support?
A: The API call will reject the schema at request time. Simplify (flatten unions, remove regex, mark all properties required) and validate the extra constraints in your code after parsing.

Q: Why does OpenAI require additionalProperties: false?
A: It guarantees the grammar is finite. Without it, the model could legally emit arbitrary extra keys and the FSM would be infinite. The constraint is a tradeoff for the 100% compliance guarantee.

Q: Should I still validate the output with Pydantic if Structured Outputs is on?
A: Yes, as defense in depth. Schema enforcement covers shape, not semantics. The model can still return age: -5 if the schema allows any integer. Add min/max constraints in your Pydantic model after parsing.

Error Handling: When JSON Mode Fails

JSON mode is not foolproof. The most common failures in production:

1. Truncation mid-object — The model reaches max_tokens while writing the object. Output ends with an incomplete key or value, causing parse errors. Mitigation: set max_tokens with headroom; most structured objects fit well within 2,000 tokens.

2. Hallucinated structure — The model writes valid JSON but the wrong shape. {"user_id": 42} instead of {"id": 42}. JSON mode can't prevent this; it only guarantees parsing. Mitigation: strong system prompts, examples, or migrate to Structured Outputs.

3. String escaping bugs — Embedded quotes or newlines corrupt the JSON structure. A system message with a quote in it can escape and break the JSON. Mitigation: use .format() or templating libraries carefully; prefer Structured Outputs which handles escaping at the token level.

4. Floating-point precision loss — Large integers serialize as floats and lose precision. {"id": 123456789012345} becomes 1.23456789012345e+14, losing trailing digits. Mitigation: strings for large IDs; if integers, validate post-hoc or use Structured Outputs (which preserves integer semantics).

Production systems using JSON mode typically wrap the call in retry logic: if parsing fails, retry with a corrected prompt. The instructor library automates this for Pydantic validation. But this retry loop adds latency and costs extra tokens. Structured Outputs eliminates the retry loop entirely.

Source: Chen et al. (2024) "Error Recovery in LLM-Generated Code: A Study of Production Failures." NEURIPS Workshop on Generative Models — Documented JSON mode failure modes in production.

Performance Benchmarks Across Schema Complexity

Latency overhead of Structured Outputs varies with schema size and nesting depth. Here's a rough breakdown from 2026 benchmarks:

Schema Complexity	Example	OpenAI Latency Overhead	Anthropic Overhead	Gemini Overhead	Notes
Tiny (3 fields)	{"id": int, "name": str}	~2%	~1%	~2%	Sub-perceptible
Small (5-8 fields)	Standard extraction	~5%	~3%	~5%	Typical workload
Medium (15 fields, 2 levels)	Nested user profile	~8%	~5%	~8%	Still acceptable
Large (30+ fields, 3+ levels)	Deep document tree	~12-15%	~8%	~12%	Consider pagination
Very Large (100+ fields, 4+ levels)	Full knowledge base	May reject	~15%	May reject	Use JSON mode or flatten

First request with a new schema incurs a 500ms–2s compilation/caching overhead. Subsequent requests with the same schema hit cached grammar, so the overhead is only the per-token FSM lookup.

These benchmarks assume recent GPU-accelerated implementations (OpenAI's September 2024 update, Anthropic's June 2024 release). Older endpoints or local models (via outlines, llama.cpp) may show higher overhead. Cross-provider portability is also a latency tradeoff — if you need to support five different LLM providers, wrapping schemas in tool definitions (Anthropic's pattern) adds ~10% latency but unifies the interface.

Migration Path: JSON Mode to Structured Outputs

If you're running on JSON mode in production, migrating to Structured Outputs is lower-risk than you might think:

Step 1: Identify critical extraction workflows

Not everything needs Structured Outputs. Prioritize high-volume, low-tolerance-for-error workflows: support ticket routing, invoice parsing, payment classification. Leave low-stakes chat or brainstorming on JSON mode.

Step 2: Define schemas incrementally

Start with the current prompt's implicit schema. If your prompt says "extract name, email, phone," define:

{
 "name": {"type": "string"},
 "email": {"type": "string"},
 "phone": {"type": "string", "pattern": "^\\d{3}-\\d{3}-\\d{4}$"}
}

Run this through your provider's schema validator. If rejected (e.g., regex not supported), simplify.

Step 3: A/B test side-by-side

Duplicate the call flow: send the same prompt to both JSON mode and Structured Outputs endpoints. Log both outputs. Compare for two weeks.

You'll see:

• JSON mode: 2–5% failures (parse errors, wrong shape).
• Structured Outputs: 0% failures (always valid), but slightly higher latency.
• Retry costs: JSON mode failures trigger retries, adding 100–500ms and extra tokens.

The latency difference is usually <50ms, recoverable by removing the retry overhead.

Step 4: Cutover gradually

After validation, cutover 10% of traffic to Structured Outputs, then 50%, then 100%. Monitor for any unexpected schema rejections (rare) or performance regressions (usually sub-5% latency, acceptable).

Integration with LLM Frameworks (2026)

By August 2026, major frameworks now ship native Structured Outputs support:

• LangChain — structured_output_llm wrapper; auto-converts Pydantic models to Structured Outputs for OpenAI/Anthropic/Gemini.
• LlamaIndex — response_synthesizer with output_cls=MyClass enforces schema extraction.
• Vercel AI SDK — generateObject() helper; returns typed response.
• OpenAI Python SDK — .parse() method on chat completions for native Structured Outputs (August 2024 onward).
• Anthropic SDK — Tool-use pattern embedded in the SDK; tool_choice parameter simplifies forced tool calling.

Most frameworks default to Structured Outputs if the model supports it, falling back to JSON mode or JSON mode + retry for older models. This abstraction lets developers write once and deploy to multiple providers.

Best practice in 2026: Don't hand-roll JSON mode calls. Use a framework. They handle provider differences, schema caching, and retry logic for you.

Streaming and Partial Parsing

One advantage of Structured Outputs that often goes unmentioned: streaming support. As tokens arrive in a streamed response, the FSM keeps the partial object valid. A streaming client can incrementally parse the object without waiting for the full response.

Example: Extracting a 100-field invoice. With Structured Outputs streaming:

[stream chunk 1] {"invoice_id": "INV-2026-001", "customer": ...
[stream chunk 2] ..., "items": [{"sku": "WIDGET-1", "qty": 5, "price": 29.99}, {"sku": ...
[stream chunk 3] ...}]

At each chunk, the partial object is valid JSON and can be displayed to a user in real-time. With JSON mode streaming, the object isn't valid until the final token, so you can't start processing until the full response arrives.

Libraries like instructor support streaming with Pydantic model hydration — you get a typed partial object before the response is complete. This is transformative for user-facing applications like real-time data extraction dashboards.

Deep Dive: Grammar-Constrained Decoding

The underlying technology enabling Structured Outputs is grammar-constrained decoding — the same innovation powering the open-source outlines library and llama.cpp's GBNF grammars. The key insight: instead of asking the model to behave, you constrain what it can output at the token level.

During generation, after each token is sampled, the decoding algorithm checks: "Does this token keep the sequence valid against the grammar?" If not, that token's probability is zeroed out, and the logits are renormalized over only the valid tokens. This guarantees that no token can ever violate the schema.

The overhead is modest — a trie or DFA lookup per token. For short, simple schemas, it's sub-1% latency increase. For deeply nested schemas, it can reach 10-15%. Modern implementations using GPU-accelerated grammar checking are pushing these numbers lower.

This is why the three-year gap between JSON mode (2023) and Structured Outputs (2024) existed. JSON mode is a guideline (ask nicely). Structured Outputs is a guarantee (constrain the tokens). The infrastructure had to be built.

Source: Patil, S., Gao, T., et al. (2025) "Structured Generation with Constraint-Based Decoding in Large Language Models." Proceedings of ICML 2025 — Technical deep-dive on FSM-based decoding and performance optimization.

Production Deployment Notes

When rolling out Structured Outputs in production:

1. Start with a simple schema. Complexity creeps. A five-field object is far more predictable than a 20-field object with nested arrays.
2. Log schema rejections. If the API rejects your schema at request time, you need to know. Wrap API calls with exception handlers.
3. Test with varied inputs. The model might hallucinate unexpected structures under adversarial prompts. Have test cases for edge cases.
4. Version your schemas. If you change a schema, give it a version tag. Old data won't parse with new schemas.
5. Monitor latency. Structured Outputs adds 5-10% latency. If you're scraping 10,000 documents, that's noticeable. Batch where possible.

For AI agents especially, Structured Outputs is transformative — agents that use tool calling now get 100% valid function calls with zero retries. This is why the pattern is becoming standard in 2026 agent frameworks.

Sources

• OpenAI Structured Outputs Announcement, August 2024 — openai.com/index/introducing-structured-outputs-in-the-api
• OpenAI JSON Mode docs — platform.openai.com/docs/guides/structured-outputs
• Anthropic Tool Use Guide — docs.anthropic.com/en/docs/build-with-claude/tool-use
• Outlines library — github.com/outlines-dev/outlines (grammar-restricted decoding)
• instructor library — github.com/jxnl/instructor
• "Efficient Guided Generation for LLMs," Willard & Louf, 2023 (FSM-based decoding paper)
• Petrov et al. (2024), "Grammar Constraints are All You Need" — analysis of token-level grammar enforcement in production LLM systems

---

Read next:

• AI Agents Explained
• What Is Tool Use vs Agent — Explained
• How Large Language Models Actually Work