Synthetic Data Generation with LLMs

Training data is the bottleneck for most AI projects. Collecting, labeling, and cleaning real-world data is expensive and slow. Privacy regulations restrict access to sensitive datasets. Edge cases are rare by definition, making them hard to collect naturally. Synthetic data generation with LLMs addresses all three problems by creating realistic, diverse datasets on demand.

This is not about replacing real data entirely. It is about augmenting limited datasets, bootstrapping new projects, and generating the edge cases that real collection misses.

When Synthetic Data Makes Sense

Synthetic data works well in specific scenarios. Understanding when to use it prevents wasted effort.

Good use cases:

• Bootstrapping a classifier when you have fewer than 100 labeled examples per class
• Generating edge cases for testing (malformed inputs, adversarial examples, boundary conditions)
• Creating evaluation datasets for LLM applications
• Augmenting imbalanced datasets where minority classes have few samples
• Prototyping a pipeline before real data collection begins

Poor use cases:

• Replacing real data when you already have thousands of labeled examples
• Generating data for domains the LLM has not been trained on
• Creating data that requires factual accuracy about specific real-world entities

Basic Generation Pattern

The simplest approach prompts an LLM to generate examples matching a specification. The key is providing clear structure and enough examples to establish the pattern.

import json
from openai import OpenAI

client = OpenAI()

def generate_examples(
    task_description: str,
    seed_examples: list[dict],
    count: int = 20,
) -> list[dict]:
    seed_text = json.dumps(seed_examples, indent=2)

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {
                "role": "system",
                "content": (
                    f"Generate {count} new examples for the following task. "
                    f"Task: {task_description}\n\n"
                    f"Follow the exact same JSON structure as the seed "
                    f"examples. Make examples diverse and realistic. "
                    f"Return a JSON array only."
                ),
            },
            {
                "role": "user",
                "content": f"Seed examples:\n{seed_text}",
            },
        ],
        temperature=0.9,
        response_format={"type": "json_object"},
    )

    result = json.loads(response.choices[0].message.content)
    return result.get("examples", result.get("data", []))

Higher temperature values (0.8-1.0) produce more diverse outputs. Lower values create examples closer to the seeds, which can cause mode collapse in your synthetic dataset.

Generating Diverse Data with Persona Variation

A common failure mode is generating data that all sounds the same. One effective technique is to vary the persona or context for each generation batch.

import random

PERSONAS = [
    "a frustrated customer who has been waiting 3 days",
    "a new user confused by the interface",
    "a power user requesting an advanced feature",
    "a business owner evaluating the product",
    "a non-native English speaker writing informally",
    "a technical user providing detailed bug reports",
]

SCENARIOS = [
    "mobile app", "web dashboard", "API integration",
    "billing portal", "onboarding flow", "settings page",
]

def generate_diverse_tickets(count: int = 50) -> list[dict]:
    all_tickets = []
    per_batch = 5

    for _ in range(count // per_batch):
        persona = random.choice(PERSONAS)
        scenario = random.choice(SCENARIOS)

        response = client.chat.completions.create(
            model="gpt-4o",
            messages=[
                {
                    "role": "system",
                    "content": (
                        f"Generate {per_batch} realistic support tickets. "
                        f"Write as {persona} experiencing issues with "
                        f"the {scenario}. Each ticket should have: "
                        f"'subject', 'body', 'category', 'priority'. "
                        f"Return a JSON array."
                    ),
                },
                {"role": "user", "content": "Generate the tickets."},
            ],
            temperature=0.95,
            response_format={"type": "json_object"},
        )

        batch = json.loads(response.choices[0].message.content)
        tickets = batch.get("tickets", batch.get("data", []))
        all_tickets.extend(tickets)

    return all_tickets

By crossing personas with scenarios, you create a matrix of variation that prevents the LLM from defaulting to generic, repetitive outputs.

Quality Control Pipeline

Generated data needs validation. Not every LLM output is usable. A quality control pipeline filters, deduplicates, and scores synthetic examples.

from difflib import SequenceMatcher

def deduplicate(examples: list[dict], field: str, threshold: float = 0.85):
    """Remove examples that are too similar to each other."""
    unique = []
    for example in examples:
        text = example[field]
        is_duplicate = False
        for kept in unique:
            similarity = SequenceMatcher(
                None, text, kept[field]
            ).ratio()
            if similarity > threshold:
                is_duplicate = True
                break
        if not is_duplicate:
            unique.append(example)
    return unique

def validate_structure(examples: list[dict], required_fields: list[str]):
    """Keep only examples with all required fields populated."""
    valid = []
    for ex in examples:
        if all(ex.get(f) and len(str(ex[f])) > 0 for f in required_fields):
            valid.append(ex)
    return valid

def quality_pipeline(
    raw_examples: list[dict],
    text_field: str,
    required_fields: list[str],
) -> list[dict]:
    step1 = validate_structure(raw_examples, required_fields)
    step2 = deduplicate(step1, text_field)
    step3 = [ex for ex in step2 if len(ex[text_field]) >= 20]
    print(f"Pipeline: {len(raw_examples)} -> {len(step1)} -> "
          f"{len(step2)} -> {len(step3)}")
    return step3

Run quality control after every generation batch. Plan to generate 30-50% more examples than you need to account for filtering losses.

Generating Labeled Pairs

For classification tasks, you often need input-label pairs. Generate them together to ensure consistency.

def generate_labeled_pairs(
    categories: dict[str, str],
    per_category: int = 20,
) -> list[dict]:
    """
    categories: mapping of category name to description.
    Example: {"spam": "Unwanted promotional emails", ...}
    """
    all_pairs = []

    for category, description in categories.items():
        response = client.chat.completions.create(
            model="gpt-4o",
            messages=[
                {
                    "role": "system",
                    "content": (
                        f"Generate {per_category} realistic examples of: "
                        f"{category} - {description}. "
                        f"Return JSON array with 'text' and 'label' fields. "
                        f"All labels should be '{category}'. "
                        f"Vary length, style, and content significantly."
                    ),
                },
                {"role": "user", "content": "Generate the examples."},
            ],
            temperature=0.9,
            response_format={"type": "json_object"},
        )

        batch = json.loads(response.choices[0].message.content)
        pairs = batch.get("examples", batch.get("data", []))
        all_pairs.extend(pairs)

    random.shuffle(all_pairs)
    return all_pairs

Common Pitfalls

Contamination with test data. If your LLM was trained on the same distribution as your test set, synthetic data may inadvertently memorize test examples. Always use a held-out real test set that you never mix with synthetic training data.

Distributional shift. LLM-generated text has subtle stylistic patterns that differ from real text. Models trained exclusively on synthetic data may underperform on real inputs. Mix synthetic data with whatever real data you have, even if the real data is small.

Label noise. When generating labeled data, the LLM sometimes produces examples that do not match the assigned label. The quality control pipeline helps, but manual spot-checking of a random sample (50-100 examples) is essential before training.

Lack of variety. Despite high temperature, LLMs tend toward common patterns. Use persona variation, explicit diversity instructions (“include examples with typos, slang, and formal language”), and multiple generation rounds with different prompts.

Summary

Synthetic data generation with LLMs is a practical tool for bootstrapping datasets, covering edge cases, and augmenting limited real data. Use persona and scenario variation to ensure diversity, run every batch through a quality control pipeline, and always validate against real examples. Treat synthetic data as a complement to real data, not a replacement.