Getting Structured Output from LLMs

Getting an LLM to return valid JSON sounds simple. Ask for JSON, get JSON. In practice, models produce trailing commas, miss closing braces, invent fields, omit required fields, and wrap their JSON in markdown code fences. This post covers every reliable method for getting structured data out of language models.

The problem with “just ask for JSON”

The naive approach is to include “respond in JSON” in your prompt. This works most of the time but fails unpredictably:

# this works 90% of the time, fails 10%
prompt = """Extract the name and age from this text. Respond in JSON.

Text: "John Smith is 34 years old and lives in Boston."
"""
# might return: {"name": "John Smith", "age": 34}
# might return: Here's the JSON: {"name": "John Smith", "age": 34}
# might return: {"name": "John Smith", "age": 34, "city": "Boston"}
# might return: ```json\n{"name": "John Smith", "age": 34}\n```

Each failure mode requires a different fix. You need systematic solutions, not prompt patches.

JSON mode

Both OpenAI and Anthropic offer JSON mode, which constrains the model to output valid JSON.

OpenAI JSON mode

from openai import OpenAI

client = OpenAI()

response = client.chat.completions.create(
    model="gpt-4o",
    response_format={"type": "json_object"},
    messages=[
        {"role": "system", "content": "Extract entities as JSON. Always include 'name' and 'age' fields."},
        {"role": "user", "content": "John Smith is 34 years old."},
    ],
)

import json
data = json.loads(response.choices[0].message.content)
print(data)  # {"name": "John Smith", "age": 34}

JSON mode guarantees valid JSON syntax but does not guarantee the schema. The model might still return {"person": "John Smith", "years": 34} instead of the fields you asked for.

OpenAI structured outputs

For schema enforcement, OpenAI offers structured outputs with a JSON schema:

response = client.chat.completions.create(
    model="gpt-4o",
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "person_extraction",
            "strict": True,
            "schema": {
                "type": "object",
                "properties": {
                    "name": {"type": "string"},
                    "age": {"type": "integer"},
                    "city": {"type": "string"},
                },
                "required": ["name", "age", "city"],
                "additionalProperties": False,
            },
        },
    },
    messages=[
        {"role": "user", "content": "John Smith is 34 years old and lives in Boston."},
    ],
)

data = json.loads(response.choices[0].message.content)
# guaranteed to have exactly name, age, city with correct types

Function calling for structured output

Function calling was designed for tool use, but it is also the most robust way to get structured data from many models. You define a “function” whose parameters match your desired output schema, and the model fills in the arguments.

from anthropic import Anthropic

client = Anthropic()

tools = [
    {
        "name": "extract_person",
        "description": "Extract person information from text",
        "input_schema": {
            "type": "object",
            "properties": {
                "name": {
                    "type": "string",
                    "description": "Full name of the person",
                },
                "age": {
                    "type": "integer",
                    "description": "Age in years",
                },
                "city": {
                    "type": "string",
                    "description": "City of residence",
                },
            },
            "required": ["name", "age"],
        },
    }
]

response = client.messages.create(
    model="claude-sonnet-4-20250514",
    max_tokens=1024,
    tools=tools,
    tool_choice={"type": "tool", "name": "extract_person"},
    messages=[
        {"role": "user", "content": "John Smith is 34 years old and lives in Boston."},
    ],
)

# the tool_use block contains structured data
for block in response.content:
    if block.type == "tool_use":
        print(block.input)
        # {"name": "John Smith", "age": 34, "city": "Boston"}

The tool_choice parameter with type: "tool" forces the model to use exactly that tool, guaranteeing you get the structured output.

Pydantic for validation and parsing

Pydantic is the standard Python library for data validation. Use it as a safety net to catch any schema violations that slip through:

from pydantic import BaseModel, Field, validator
from typing import Optional
import json

class Person(BaseModel):
    name: str = Field(description="Full name")
    age: int = Field(ge=0, le=150, description="Age in years")
    city: Optional[str] = Field(default=None, description="City of residence")
    
    @validator("name")
    def name_must_not_be_empty(cls, v):
        if not v.strip():
            raise ValueError("Name cannot be empty")
        return v.strip()

# parse and validate LLM output
raw_json = '{"name": "John Smith", "age": 34, "city": "Boston"}'

try:
    person = Person.model_validate_json(raw_json)
    print(f"Valid: {person}")
except Exception as e:
    print(f"Validation error: {e}")

# handle common LLM formatting issues
def parse_llm_json(text: str, model_class: type[BaseModel]):
    """Parse JSON from LLM output, handling common formatting issues."""
    # strip markdown code fences
    text = text.strip()
    if text.startswith("```"):
        lines = text.split("\n")
        text = "\n".join(lines[1:-1])
    
    # try to find JSON object in text
    start = text.find("{")
    end = text.rfind("}") + 1
    if start != -1 and end > start:
        text = text[start:end]
    
    return model_class.model_validate_json(text)

Using Pydantic with OpenAI’s structured outputs

The OpenAI Python SDK has built-in Pydantic support:

from pydantic import BaseModel
from openai import OpenAI

class PersonExtraction(BaseModel):
    name: str
    age: int
    city: str
    occupation: str | None = None

client = OpenAI()

response = client.beta.chat.completions.parse(
    model="gpt-4o",
    messages=[
        {"role": "user", "content": "John Smith, 34, is a software engineer in Boston."},
    ],
    response_format=PersonExtraction,
)

person = response.choices[0].message.parsed
print(f"{person.name}, {person.age}, {person.city}")
# John Smith, 34, Boston

Outlines: guaranteed structured generation

Outlines is a library that constrains model generation at the token level. Instead of hoping the model produces valid JSON, Outlines makes it impossible to produce anything else.

import outlines

model = outlines.models.transformers("microsoft/Phi-3-mini-4k-instruct")

# generate valid JSON matching a schema
schema = {
    "type": "object",
    "properties": {
        "name": {"type": "string"},
        "age": {"type": "integer"},
        "interests": {
            "type": "array",
            "items": {"type": "string"},
        },
    },
    "required": ["name", "age", "interests"],
}

generator = outlines.generate.json(model, schema)
result = generator("Extract info: John Smith, 34, likes hiking and chess.")
print(result)
# {"name": "John Smith", "age": 34, "interests": ["hiking", "chess"]}

Using Pydantic models with Outlines

from pydantic import BaseModel
from typing import List
import outlines

class MovieReview(BaseModel):
    title: str
    rating: float
    genres: List[str]
    sentiment: str
    summary: str

model = outlines.models.transformers("microsoft/Phi-3-mini-4k-instruct")
generator = outlines.generate.json(model, MovieReview)

review_text = """
Inception (2010) is a mind-bending masterpiece. Nolan delivers a 
thrilling sci-fi experience that keeps you guessing. 9/10.
"""

result = generator(f"Extract a structured review: {review_text}")
print(f"Title: {result.title}")
print(f"Rating: {result.rating}")
print(f"Genres: {result.genres}")

Constrained choices with Outlines

For classification or enum-like outputs:

import outlines

model = outlines.models.transformers("microsoft/Phi-3-mini-4k-instruct")

# constrain output to specific choices
generator = outlines.generate.choice(model, ["positive", "negative", "neutral"])
sentiment = generator("Classify: The food was okay but nothing special.")
print(sentiment)  # "neutral"

# regex-constrained generation
# generate a valid email
email_pattern = r"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}"
generator = outlines.generate.regex(model, email_pattern)
email = generator("Generate a contact email for John Smith at Acme Corp:")
print(email)  # "john.smith@acmecorp.com"

Retry strategies

Even with the best tools, you should have a retry mechanism:

from pydantic import BaseModel
from tenacity import retry, stop_after_attempt, retry_if_exception_type

class ExtractedData(BaseModel):
    entities: list[str]
    summary: str
    confidence: float

@retry(
    stop=stop_after_attempt(3),
    retry=retry_if_exception_type((json.JSONDecodeError, ValueError)),
)
def extract_with_retry(text: str) -> ExtractedData:
    response = client.messages.create(
        model="claude-sonnet-4-20250514",
        max_tokens=1024,
        tools=[{
            "name": "extract",
            "description": "Extract structured data from text",
            "input_schema": ExtractedData.model_json_schema(),
        }],
        tool_choice={"type": "tool", "name": "extract"},
        messages=[{"role": "user", "content": f"Extract from: {text}"}],
    )
    
    for block in response.content:
        if block.type == "tool_use":
            return ExtractedData.model_validate(block.input)
    
    raise ValueError("No tool use in response")

Choosing the right approach

Method	Reliability	Latency	Works with	Best for
Prompt-only	Low	Low	Any model	Prototyping
JSON mode	Medium	Low	API models	Simple schemas
Structured outputs	High	Low	OpenAI	Complex schemas
Function calling	High	Low	Most APIs	Complex schemas
Pydantic validation	N/A (layer)	Minimal	Any	Validation safety net
Outlines	Very high	Medium	Local models	Guaranteed structure

Start with function calling or structured outputs. Add Pydantic validation as a safety net. Use Outlines when you need guarantees with local models. Only fall back to prompt-only JSON when you are prototyping and do not care about reliability yet.