Getting Structured Output from LLMs

The Problem with Free Text

LLMs generate free-form text by default. Ask “extract the name and email from this paragraph” and you might get:

• The name is John Smith and the email is john@example.com
• Name: John Smith\nEmail: john@example.com
• {"name": "John Smith", "email": "john@example.com"}

Three valid answers, three different formats. Your downstream code needs to handle all of them, or it breaks. Structured output prompting eliminates this problem by forcing the model into a consistent, parseable format.

JSON Output: The Basics

The simplest approach is to ask for JSON explicitly and show the schema.

import openai
import json

client = openai.OpenAI()

prompt = """Extract contact information from the text below.
Return a JSON object with exactly these keys:
- "name": full name (string)
- "email": email address (string or null)
- "phone": phone number (string or null)
- "company": company name (string or null)

Return ONLY the JSON object, no other text.

Text: "Hi, I'm Sarah Chen from Acme Corp. Reach me at sarah.chen@acme.io
or call 555-0142."
"""

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": prompt}],
    temperature=0
)

result = json.loads(response.choices[0].message.content)
# {"name": "Sarah Chen", "email": "sarah.chen@acme.io",
#  "phone": "555-0142", "company": "Acme Corp"}

This works most of the time, but “most of the time” is not good enough for production. The model might add markdown code fences, explanatory text, or slightly different keys.

API-Level Format Enforcement

Modern LLM APIs offer built-in JSON mode that guarantees valid JSON output.

OpenAI JSON Mode

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": "You extract contact info as JSON."},
        {"role": "user", "content": "Extract from: 'Call Mike at 555-0199'"}
    ],
    response_format={"type": "json_object"},
    temperature=0
)
# Guaranteed valid JSON, no code fences, no extra text
data = json.loads(response.choices[0].message.content)

OpenAI Structured Outputs with Schema

For even stricter control, pass a JSON schema.

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": "Extract contact information from the text."},
        {"role": "user", "content": "Email jane@corp.com, she's the CTO of DataFlow Inc."}
    ],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "contact_info",
            "strict": True,
            "schema": {
                "type": "object",
                "properties": {
                    "name": {"type": "string"},
                    "email": {"type": ["string", "null"]},
                    "phone": {"type": ["string", "null"]},
                    "company": {"type": ["string", "null"]},
                    "role": {"type": ["string", "null"]}
                },
                "required": ["name", "email", "phone", "company", "role"],
                "additionalProperties": False
            }
        }
    }
)

This approach is the gold standard. The API constrains the model’s token generation to match your schema exactly. Keys cannot be missing, types cannot be wrong.

Pydantic Validation

When using models or APIs without schema enforcement, validate on your side with Pydantic.

from pydantic import BaseModel, ValidationError
from typing import Optional

class ContactInfo(BaseModel):
    name: str
    email: Optional[str] = None
    phone: Optional[str] = None
    company: Optional[str] = None

def extract_contact(text: str) -> ContactInfo:
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": f"""Extract contact info as JSON matching this schema:
{ContactInfo.model_json_schema()}

Return ONLY valid JSON."""},
            {"role": "user", "content": text}
        ],
        response_format={"type": "json_object"},
        temperature=0
    )

    raw = json.loads(response.choices[0].message.content)

    try:
        return ContactInfo(**raw)
    except ValidationError as e:
        # Retry once with the error message
        retry_response = client.chat.completions.create(
            model="gpt-4o",
            messages=[
                {"role": "system", "content": "Fix the JSON to match the required schema."},
                {"role": "user", "content": f"JSON: {json.dumps(raw)}\nErrors: {str(e)}\n\nReturn corrected JSON only."}
            ],
            response_format={"type": "json_object"},
            temperature=0
        )
        fixed = json.loads(retry_response.choices[0].message.content)
        return ContactInfo(**fixed)

The retry pattern is important. Instead of failing, you send the validation error back to the model and ask it to fix the output. This handles the 1-2% of cases where the first attempt has a minor schema violation.

XML Output

XML works well when you need nested, hierarchical data or when your downstream system expects XML.

XML_PROMPT = """Analyze the following product review and return your analysis as XML.

Use this exact structure:
<review_analysis>
  <sentiment>positive|negative|neutral</sentiment>
  <confidence>0.0 to 1.0</confidence>
  <topics>
    <topic name="topic_name" sentiment="positive|negative|neutral" />
  </topics>
  <summary>One sentence summary</summary>
</review_analysis>

Review: "{review}"
"""

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": XML_PROMPT.format(
        review="The camera is incredible but the battery barely lasts 4 hours. Screen is gorgeous though."
    )}],
    temperature=0
)

# Output:
# <review_analysis>
#   <sentiment>positive</sentiment>
#   <confidence>0.65</confidence>
#   <topics>
#     <topic name="camera" sentiment="positive" />
#     <topic name="battery" sentiment="negative" />
#     <topic name="screen" sentiment="positive" />
#   </topics>
#   <summary>Mixed review praising camera and screen quality but criticizing short battery life.</summary>
# </review_analysis>

XML has one advantage over JSON for LLMs: closing tags give the model a clear signal for when to stop generating a section. </topics> is unambiguous, while JSON’s ] could be any array ending.

Table Output

For tabular data, markdown tables are the most reliable format.

TABLE_PROMPT = """Compare the following programming languages on the given criteria.
Return a markdown table with columns: Language, Typing, Speed, Learning Curve, Best For.

Languages: Python, Rust, Go, JavaScript
"""

# Output:
# | Language   | Typing  | Speed     | Learning Curve | Best For              |
# |------------|---------|-----------|----------------|-----------------------|
# | Python     | Dynamic | Slow      | Easy           | Data science, scripts |
# | Rust       | Static  | Very fast | Hard           | Systems, performance  |
# | Go         | Static  | Fast      | Moderate       | Backend services      |
# | JavaScript | Dynamic | Moderate  | Easy           | Web, full-stack       |

For machine-parseable tables, CSV is more reliable.

CSV_PROMPT = """Extract all transactions from the following bank statement.
Return as CSV with headers: date,description,amount,type

Rules:
- date format: YYYY-MM-DD
- amount: numeric, no currency symbols
- type: debit or credit
- No extra text, just the CSV

Statement: "{statement}"
"""

Nested and Complex Structures

For deeply nested data, provide a complete example rather than just a schema description.

COMPLEX_PROMPT = """Parse this job posting into structured data.

Example output:
{
  "title": "Senior Backend Engineer",
  "company": "TechCorp",
  "location": {
    "city": "San Francisco",
    "state": "CA",
    "remote": true
  },
  "compensation": {
    "salary_min": 150000,
    "salary_max": 200000,
    "currency": "USD",
    "equity": true
  },
  "requirements": {
    "years_experience": 5,
    "required_skills": ["Python", "PostgreSQL", "AWS"],
    "preferred_skills": ["Kubernetes", "GraphQL"]
  }
}

Now parse this posting:
"{posting}"

Return ONLY the JSON object."""

Showing a complete example of the nested structure works better than describing it in words. The model copies the structure exactly and fills in different values.

Handling Arrays of Objects

When extracting multiple items, explicitly instruct the model about array handling.

MULTI_EXTRACT_PROMPT = """Extract ALL people mentioned in the text.
Return a JSON object with a single key "people" containing an array.

Each person object has:
- "name": string
- "role": string or null
- "mentioned_context": one sentence about why they were mentioned

If no people are found, return {"people": []}.

Text: "{text}"
"""

# This explicit instruction about empty arrays prevents the model from
# returning null, "none found", or omitting the key entirely.

The empty-array instruction is crucial. Without it, models handle zero-result cases inconsistently.

Reliable Patterns Summary

Most reliable ──────────────────────────────> Least reliable

API schema       API JSON mode    Prompt + examples    Prompt only
enforcement      + Pydantic       + validation         "return JSON"
(guaranteed)     (99%+ reliable)  (95%+ reliable)      (80-90%)

For production systems:

1. Use API-level schema enforcement when available
2. Fall back to JSON mode + Pydantic validation
3. Add retry logic with error feedback
4. Always handle parsing failures gracefully

def safe_extract(text: str, max_retries: int = 2) -> dict | None:
    """Extract with retries and graceful failure."""
    for attempt in range(max_retries + 1):
        try:
            response = client.chat.completions.create(
                model="gpt-4o",
                messages=[{"role": "user", "content": f"Extract data from: {text}"}],
                response_format={"type": "json_object"},
                temperature=0
            )
            data = json.loads(response.choices[0].message.content)
            return ContactInfo(**data).model_dump()
        except (json.JSONDecodeError, ValidationError) as e:
            if attempt == max_retries:
                return None  # Log and handle gracefully
            continue

Key Takeaways

Structured output is not optional for production LLM applications. Use API-level schema enforcement as your first choice. When that is not available, combine JSON mode with Pydantic validation and retry logic. Show complete examples of your desired structure rather than describing it abstractly. Handle edge cases explicitly, especially empty results and null fields. And always, always validate before passing LLM output to the rest of your system.