Prompt Evaluation: Measuring and Improving Quality

The Problem with Vibes

Most prompt engineering follows this loop: write a prompt, try it on one or two examples, see if it “looks good,” and ship it. This works for prototypes. It fails catastrophically in production because:

1. You tested 3 inputs. Production sees 3,000 per day.
2. You evaluated quality with your eyes. Your users have different standards.
3. You changed the prompt last week and have no idea if it got better or worse.
4. Two team members disagree about which prompt version is better, with no data to settle it.

Prompt evaluation replaces vibes with measurement. It gives you a number that goes up when the prompt improves and goes down when it regresses.

Building an Evaluation Dataset

An eval dataset is a collection of input-output pairs where you know the correct (or acceptable) answer.

eval_dataset = [
    {
        "input": "The battery lasts forever and the screen is beautiful, but it's too expensive.",
        "expected": "mixed",
        "notes": "Positive features, negative price"
    },
    {
        "input": "Absolute garbage. Broke on day one.",
        "expected": "negative",
        "notes": "Clearly negative, strong language"
    },
    {
        "input": "It's okay for the price.",
        "expected": "neutral",
        "notes": "Mild positive with price qualifier"
    },
    {
        "input": "Best purchase I've made this year!",
        "expected": "positive",
        "notes": "Strongly positive"
    },
    {
        "input": "Meh.",
        "expected": "neutral",
        "notes": "Minimal input, ambiguous"
    },
    # ... 50-200 more examples
]

How many examples do you need?

• Minimum viable eval: 30 examples covering the main categories
• Solid eval: 100-200 examples with edge cases
• Production eval: 500+ examples, continuously expanded from real data

Where to get examples:

• Start with hand-written examples covering known edge cases
• Add real production inputs that caused failures
• Use stratified sampling to cover all categories equally
• Include adversarial examples (inputs designed to trick the model)

Exact Match Scoring

The simplest evaluation: does the output match the expected answer exactly?

import openai

client = openai.OpenAI()

def evaluate_prompt(prompt_template: str, eval_data: list[dict]) -> dict:
    """Run a prompt against eval data and score it."""
    correct = 0
    total = len(eval_data)
    failures = []

    for item in eval_data:
        response = client.chat.completions.create(
            model="gpt-4o",
            messages=[
                {"role": "user", "content": prompt_template.format(text=item["input"])}
            ],
            temperature=0,
            max_tokens=20
        )
        output = response.choices[0].message.content.strip().lower()
        expected = item["expected"].lower()

        if output == expected:
            correct += 1
        else:
            failures.append({
                "input": item["input"],
                "expected": expected,
                "got": output
            })

    accuracy = correct / total
    return {
        "accuracy": accuracy,
        "correct": correct,
        "total": total,
        "failures": failures
    }

# Compare two prompt versions
prompt_v1 = 'Classify the sentiment of this text as positive, negative, or neutral.\n\nText: "{text}"\n\nSentiment:'
prompt_v2 = 'Read the following product review. What is the overall sentiment? Respond with exactly one word: positive, negative, or neutral.\n\nReview: "{text}"\n\nSentiment:'

result_v1 = evaluate_prompt(prompt_v1, eval_dataset)
result_v2 = evaluate_prompt(prompt_v2, eval_dataset)

print(f"V1 accuracy: {result_v1['accuracy']:.1%}")
print(f"V2 accuracy: {result_v2['accuracy']:.1%}")
# V1 accuracy: 82.0%
# V2 accuracy: 91.0%

Now you have a number. Prompt V2 is measurably better. No vibes required.

Rubric-Based Scoring

Exact match does not work for open-ended outputs like summaries, explanations, or creative text. Use a rubric instead.

RUBRIC = {
    "accuracy": {
        "description": "Does the response contain factually correct information?",
        "scores": {
            5: "All facts correct",
            3: "Minor inaccuracies that don't change the meaning",
            1: "Contains significant factual errors"
        }
    },
    "completeness": {
        "description": "Does the response cover all key points?",
        "scores": {
            5: "All key points addressed",
            3: "Most key points, missing 1-2",
            1: "Missing major points"
        }
    },
    "conciseness": {
        "description": "Is the response appropriately concise?",
        "scores": {
            5: "No unnecessary content",
            3: "Some filler or redundancy",
            1: "Mostly filler, buries the answer"
        }
    }
}

You can score rubrics manually (human evaluation) or automatically (LLM-as-judge, covered below).

A/B Testing Prompts

When you have two candidate prompts, run a proper comparison instead of cherry-picking examples.

import random
from collections import defaultdict

def ab_test_prompts(
    prompt_a: str,
    prompt_b: str,
    eval_data: list[dict],
    scoring_fn,
    n_runs: int = 3
) -> dict:
    """A/B test two prompts on the same eval data."""
    scores_a = defaultdict(list)
    scores_b = defaultdict(list)

    for run in range(n_runs):
        for item in eval_data:
            score_a = scoring_fn(prompt_a, item)
            score_b = scoring_fn(prompt_b, item)
            scores_a[item["input"]].append(score_a)
            scores_b[item["input"]].append(score_b)

    # Aggregate
    avg_a = sum(sum(v)/len(v) for v in scores_a.values()) / len(scores_a)
    avg_b = sum(sum(v)/len(v) for v in scores_b.values()) / len(scores_b)

    # Count wins
    a_wins = sum(1 for k in scores_a if sum(scores_a[k]) > sum(scores_b[k]))
    b_wins = sum(1 for k in scores_a if sum(scores_b[k]) > sum(scores_a[k]))
    ties = len(scores_a) - a_wins - b_wins

    return {
        "prompt_a_avg": avg_a,
        "prompt_b_avg": avg_b,
        "a_wins": a_wins,
        "b_wins": b_wins,
        "ties": ties,
        "winner": "A" if avg_a > avg_b else "B" if avg_b > avg_a else "Tie"
    }

Run each prompt multiple times (n_runs=3) to account for non-determinism (if temperature > 0). Report wins, not just averages, because averages can be skewed by outliers.

LLM-as-Judge

The most powerful evaluation technique: use a strong LLM to grade the output of your prompt.

JUDGE_PROMPT = """You are an expert evaluator. Score the following response on a scale of 1-5 for each criterion.

CRITERIA:
- Accuracy (1-5): Is the information factually correct?
- Relevance (1-5): Does it answer the actual question asked?
- Clarity (1-5): Is it well-written and easy to understand?

QUESTION: {question}

RESPONSE TO EVALUATE: {response}

Score each criterion. Return JSON:
{{"accuracy": <int>, "relevance": <int>, "clarity": <int>, "total": <int>, "explanation": "<one sentence>"}}
"""

def llm_judge(question: str, response: str) -> dict:
    """Use GPT-4o as an automated judge."""
    import json

    result = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "user", "content": JUDGE_PROMPT.format(
                question=question, response=response
            )}
        ],
        response_format={"type": "json_object"},
        temperature=0
    )
    return json.loads(result.choices[0].message.content)

# Evaluate a prompt's output
question = "Explain how DNS works in simple terms."
response_to_grade = "DNS is like a phone book for the internet. When you type google.com, your computer asks a DNS server 'what IP address is google.com?' The DNS server looks it up and says '142.250.80.46'. Your browser then connects to that IP address. This happens in milliseconds."

score = llm_judge(question, response_to_grade)
# {"accuracy": 5, "relevance": 5, "clarity": 5, "total": 15,
#  "explanation": "Clear analogy, technically accurate, directly answers the question."}

LLM-as-judge best practices:

• Use a stronger model as judge than the model being evaluated
• Include the rubric in the judge prompt so scoring is consistent
• Randomize the order when comparing two responses (position bias is real)
• Validate the judge by checking its scores against human scores on 20-30 examples

Pairwise Comparison

Instead of absolute scores, ask the judge to compare two outputs directly.

PAIRWISE_PROMPT = """Compare these two responses to the same question.
Which is better? Consider accuracy, completeness, and clarity.

Question: {question}

Response A: {response_a}

Response B: {response_b}

Which is better, A or B? Respond with ONLY "A", "B", or "TIE".
Then explain in one sentence why."""

def pairwise_compare(question: str, response_a: str, response_b: str) -> str:
    # Randomly swap order to eliminate position bias
    if random.random() > 0.5:
        response_a, response_b = response_b, response_a
        swapped = True
    else:
        swapped = False

    result = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": PAIRWISE_PROMPT.format(
            question=question, response_a=response_a, response_b=response_b
        )}],
        temperature=0
    )
    winner = result.choices[0].message.content.strip().split("\n")[0]

    # Unswap if we randomized
    if swapped:
        if winner == "A": winner = "B"
        elif winner == "B": winner = "A"

    return winner

Pairwise comparison is more reliable than absolute scoring for subjective tasks because humans (and LLMs) are better at saying “this one is better” than assigning a number on a scale.

Regression Detection

Every time you change a prompt, run the full eval suite and compare against the previous version.

def check_regression(current_scores: dict, previous_scores: dict, threshold: float = 0.02) -> dict:
    """Detect if a prompt change caused a regression."""
    current_avg = current_scores["accuracy"]
    previous_avg = previous_scores["accuracy"]
    delta = current_avg - previous_avg

    # Check for failures on previously passing cases
    current_failures = set(f["input"] for f in current_scores["failures"])
    previous_failures = set(f["input"] for f in previous_scores["failures"])
    new_failures = current_failures - previous_failures
    fixed = previous_failures - current_failures

    return {
        "delta": delta,
        "improved": delta > threshold,
        "regressed": delta < -threshold,
        "new_failures": list(new_failures),
        "newly_fixed": list(fixed),
        "recommendation": (
            "SHIP IT" if delta > threshold and len(new_failures) == 0
            else "REVIEW" if len(new_failures) > 0
            else "REJECT" if delta < -threshold
            else "NO CHANGE"
        )
    }

The key metric is not just overall accuracy but new failures. A prompt that improves from 85% to 88% accuracy but breaks 3 previously working cases needs investigation. Those 3 cases might represent important edge cases.

Putting It All Together

1. Write prompt v1
2. Build eval dataset (30+ examples)
3. Score v1 on eval dataset -> baseline
4. Modify prompt -> v2
5. Score v2 on same eval dataset
6. Compare: accuracy, new failures, LLM judge scores
7. If improved and no regressions -> ship
8. If regressed -> debug failures, try v3
9. Add production failures to eval dataset
10. Repeat

This workflow turns prompt engineering from an art into a measurable engineering discipline. Every change is backed by data. Every regression is caught before it hits production. And your eval dataset grows over time, covering more and more edge cases from real usage.

Key Takeaways

Stop evaluating prompts by looking at one or two examples. Build an eval dataset of at least 30 examples covering normal cases, edge cases, and adversarial inputs. Use exact match for classification, rubrics for open-ended output, and LLM-as-judge for automated scoring at scale. A/B test prompt changes with multiple runs. Track regressions by checking for new failures, not just average accuracy. And continuously expand your eval dataset with real production failures.