LLM Evaluation: Metrics, Benchmarks, and Evals

Why LLM Evaluation Is Hard

Traditional ML has clean metrics. Accuracy, precision, recall, F1. You have labels, you have predictions, you compute a number. LLMs generate free-form text where the same question can have dozens of correct answers phrased differently. “The capital of France is Paris” and “Paris is France’s capital city” are both correct but have zero string overlap with each other in the right places.

You need evaluation strategies that handle this ambiguity while still giving you a reliable signal about whether your system is improving or regressing.

Level 1: Exact and Fuzzy Matching

For tasks with deterministic answers, start with simple matching.

def exact_match(predicted: str, expected: str) -> bool:
    return predicted.strip().lower() == expected.strip().lower()

def contains_match(predicted: str, expected: str) -> bool:
    return expected.strip().lower() in predicted.strip().lower()

def evaluate_batch(predictions: list[str], expected: list[str]) -> dict:
    exact = sum(exact_match(p, e) for p, e in zip(predictions, expected))
    contains = sum(contains_match(p, e) for p, e in zip(predictions, expected))
    n = len(predictions)

    return {
        "exact_match": exact / n,
        "contains_match": contains / n,
        "total": n,
    }

This works for factual QA, classification, and extraction tasks where the answer is short and unambiguous. It fails completely for open-ended generation.

Level 2: Semantic Similarity

When answers can be phrased differently, compare meaning instead of text.

from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity
import numpy as np

model = SentenceTransformer("all-MiniLM-L6-v2")

def semantic_similarity(predicted: str, expected: str) -> float:
    embeddings = model.encode([predicted, expected])
    return cosine_similarity([embeddings[0]], [embeddings[1]])[0][0]

def evaluate_semantic(predictions: list[str], expected: list[str], threshold: float = 0.8) -> dict:
    scores = [semantic_similarity(p, e) for p, e in zip(predictions, expected)]

    return {
        "mean_similarity": np.mean(scores),
        "above_threshold": sum(s >= threshold for s in scores) / len(scores),
        "min_similarity": np.min(scores),
        "max_similarity": np.max(scores),
    }

Semantic similarity captures paraphrases well but struggles with negation. “The model works” and “The model does not work” have high embedding similarity despite being opposites. Use it as one signal, not the only one.

Level 3: LLM-as-Judge

For subjective qualities like helpfulness, clarity, and tone, use a stronger LLM to evaluate a weaker one.

from openai import OpenAI
import json

client = OpenAI()

def llm_judge(question: str, answer: str, criteria: list[str]) -> dict:
    criteria_text = "\n".join(f"- {c}" for c in criteria)

    response = client.chat.completions.create(
        model="gpt-4o",
        response_format={"type": "json_object"},
        messages=[
            {
                "role": "system",
                "content": f"""You are an expert evaluator. Rate the answer on each criterion using a 1-5 scale.

Criteria:
{criteria_text}

Respond in JSON: {{"scores": {{"criterion": score}}, "reasoning": {{"criterion": "explanation"}}}}""",
            },
            {
                "role": "user",
                "content": f"Question: {question}\n\nAnswer: {answer}",
            },
        ],
        temperature=0.0,
    )

    return json.loads(response.choices[0].message.content)

# Usage
result = llm_judge(
    question="Explain how Docker containers differ from virtual machines",
    answer="Docker containers share the host OS kernel while VMs run their own OS...",
    criteria=[
        "Accuracy: Are the technical facts correct?",
        "Completeness: Does it cover the key differences?",
        "Clarity: Is the explanation easy to understand?",
    ],
)

for criterion, score in result["scores"].items():
    print(f"  {criterion}: {score}/5 - {result['reasoning'][criterion]}")

Pairwise Comparison

When you want to compare two model outputs, pairwise judging is more reliable than independent scoring.

def pairwise_judge(question: str, answer_a: str, answer_b: str) -> dict:
    response = client.chat.completions.create(
        model="gpt-4o",
        response_format={"type": "json_object"},
        messages=[
            {
                "role": "system",
                "content": """Compare two answers to the same question. Determine which is better overall.
Respond in JSON: {"winner": "A" or "B" or "tie", "reasoning": "explanation"}""",
            },
            {
                "role": "user",
                "content": f"Question: {question}\n\nAnswer A: {answer_a}\n\nAnswer B: {answer_b}",
            },
        ],
        temperature=0.0,
    )

    return json.loads(response.choices[0].message.content)

To reduce position bias (LLMs tend to prefer the first answer), run each comparison twice with swapped positions and only count cases where both runs agree.

Building an Evaluation Suite

An evaluation suite is a collection of test cases that cover your application’s requirements. Think of it as a test suite for your LLM.

from dataclasses import dataclass
from typing import Callable

@dataclass
class EvalCase:
    id: str
    input: str
    expected: str | None = None
    tags: list[str] = None
    metadata: dict = None

@dataclass
class EvalResult:
    case_id: str
    output: str
    scores: dict[str, float]
    passed: bool

class EvalSuite:
    def __init__(self, name: str):
        self.name = name
        self.cases: list[EvalCase] = []
        self.scorers: list[Callable] = []

    def add_case(self, case: EvalCase):
        self.cases.append(case)

    def add_scorer(self, scorer: Callable):
        self.scorers.append(scorer)

    def run(self, system_fn: Callable[[str], str]) -> list[EvalResult]:
        results = []

        for case in self.cases:
            output = system_fn(case.input)

            scores = {}
            for scorer in self.scorers:
                score = scorer(case.input, output, case.expected)
                scores.update(score)

            passed = all(v >= 0.7 for v in scores.values() if isinstance(v, (int, float)))
            results.append(EvalResult(
                case_id=case.id,
                output=output,
                scores=scores,
                passed=passed,
            ))

        return results

    def report(self, results: list[EvalResult]):
        passed = sum(r.passed for r in results)
        total = len(results)
        print(f"\n{self.name}: {passed}/{total} passed ({passed/total*100:.1f}%)")

        # Aggregate scores
        all_scores = {}
        for r in results:
            for key, value in r.scores.items():
                if key not in all_scores:
                    all_scores[key] = []
                all_scores[key].append(value)

        for key, values in all_scores.items():
            print(f"  {key}: {np.mean(values):.3f} (min={np.min(values):.3f})")

Using the Suite

import numpy as np

# Create scorers
def accuracy_scorer(question, output, expected):
    if expected:
        return {"accuracy": 1.0 if expected.lower() in output.lower() else 0.0}
    return {}

def length_scorer(question, output, expected):
    word_count = len(output.split())
    return {"appropriate_length": 1.0 if 50 <= word_count <= 500 else 0.5}

# Build the suite
suite = EvalSuite("QA Quality")
suite.add_scorer(accuracy_scorer)
suite.add_scorer(length_scorer)

suite.add_case(EvalCase(id="q1", input="What is Docker?", expected="containerization"))
suite.add_case(EvalCase(id="q2", input="Explain REST vs GraphQL", expected="query language"))
suite.add_case(EvalCase(id="q3", input="What is a Kubernetes pod?", expected="smallest deployable unit"))

# Run against your system
def my_system(question: str) -> str:
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": question}],
    )
    return response.choices[0].message.content

results = suite.run(my_system)
suite.report(results)

Tracking Regressions

Run evals on every change to your prompts, tools, or model version. Store results over time.

import json
from datetime import datetime

def save_eval_run(suite_name: str, results: list[EvalResult], version: str):
    run = {
        "suite": suite_name,
        "version": version,
        "timestamp": datetime.now().isoformat(),
        "summary": {
            "total": len(results),
            "passed": sum(r.passed for r in results),
            "pass_rate": sum(r.passed for r in results) / len(results),
        },
        "results": [
            {
                "case_id": r.case_id,
                "scores": r.scores,
                "passed": r.passed,
            }
            for r in results
        ],
    }

    filename = f"evals/{suite_name}_{version}_{datetime.now().strftime('%Y%m%d_%H%M%S')}.json"
    with open(filename, "w") as f:
        json.dump(run, f, indent=2)

    return run

Common Evaluation Pitfalls

Evaluating on too few examples gives noisy results. Aim for at least 50-100 test cases per capability you want to measure.

Using the same model as both the system and the judge inflates scores. A model tends to rate its own style of output favorably. Use a stronger model as the judge, or use human evaluation for critical decisions.

Optimizing for eval scores without understanding why scores change leads to overfitting your prompts to the eval set. Regularly add new test cases and rotate existing ones.

Not testing failure modes is the most common gap. Include adversarial inputs, edge cases, and inputs that should trigger refusals. A system that answers everything scores well on helpfulness but may hallucinate on topics outside its domain.

Summary

LLM evaluation requires multiple strategies layered together. Use exact matching for deterministic tasks, semantic similarity for paraphrase-tolerant comparison, and LLM-as-judge for subjective quality. Build evaluation suites that cover your application’s specific requirements. Run evals on every change and track results over time to catch regressions. Use pairwise comparison with position swapping for model-vs-model benchmarks. Always include adversarial and edge-case test cases.