LLM Evaluation: BLEU, ROUGE, and Human Metrics

You shipped an LLM feature and it seems to work. But how do you know if version 2 of your prompt is actually better than version 1? Evaluation is the most overlooked part of LLM development, yet it determines whether your system improves over time or drifts silently into failure. This guide covers both classical NLP metrics and modern evaluation techniques so you can measure what matters.

Why Evaluation Is Hard for LLMs

Traditional software has deterministic outputs. Given the same input, you get the same output, and you can write exact assertions. LLMs are different. The same prompt can produce varied but equally valid responses. “The capital of France is Paris” and “Paris is France’s capital city” are both correct, but a naive string comparison would flag one as wrong.

This fundamental challenge means you need metrics that capture semantic equivalence, not just surface-level matching.

BLEU Score: Precision-Oriented Matching

BLEU (Bilingual Evaluation Understudy) was originally designed for machine translation. It measures how many n-grams in the generated text appear in the reference text.

from collections import Counter
import math
from typing import List

def compute_bleu(reference: str, candidate: str, max_n: int = 4) -> float:
    """Compute BLEU score between reference and candidate texts."""
    ref_tokens = reference.lower().split()
    cand_tokens = candidate.lower().split()
    
    if len(cand_tokens) == 0:
        return 0.0
    
    # Brevity penalty
    bp = min(1.0, math.exp(1 - len(ref_tokens) / len(cand_tokens)))
    
    # N-gram precision for each n
    precisions = []
    for n in range(1, max_n + 1):
        ref_ngrams = Counter(
            tuple(ref_tokens[i:i+n]) for i in range(len(ref_tokens) - n + 1)
        )
        cand_ngrams = Counter(
            tuple(cand_tokens[i:i+n]) for i in range(len(cand_tokens) - n + 1)
        )
        
        # Clipped counts
        clipped = sum(
            min(count, ref_ngrams.get(ngram, 0))
            for ngram, count in cand_ngrams.items()
        )
        total = sum(cand_ngrams.values())
        
        if total == 0:
            precisions.append(0)
        else:
            precisions.append(clipped / total)
    
    # Geometric mean of precisions
    if any(p == 0 for p in precisions):
        return 0.0
    
    log_avg = sum(math.log(p) for p in precisions) / len(precisions)
    return bp * math.exp(log_avg)

# Example
reference = "The cat sat on the mat near the window"
candidate = "The cat was sitting on the mat by the window"
score = compute_bleu(reference, candidate)
print(f"BLEU score: {score:.4f}")

In practice, use the sacrebleu library for standardized BLEU computation:

import sacrebleu

references = ["The cat sat on the mat near the window"]
candidates = ["The cat was sitting on the mat by the window"]

bleu = sacrebleu.corpus_bleu(candidates, [references])
print(f"BLEU: {bleu.score:.2f}")

BLEU works best when there is a single correct answer and surface-level word overlap is meaningful. It struggles with paraphrases and creative text where valid outputs can differ substantially from references.

ROUGE Score: Recall-Oriented Matching

ROUGE (Recall-Oriented Understudy for Gisting Evaluation) was designed for summarization. While BLEU focuses on precision (how much of the candidate appears in the reference), ROUGE focuses on recall (how much of the reference is captured by the candidate).

from rouge_score import rouge_scorer

def evaluate_with_rouge(reference: str, candidate: str) -> dict:
    """Compute ROUGE scores for a candidate against a reference."""
    scorer = rouge_scorer.RougeScorer(
        ['rouge1', 'rouge2', 'rougeL'],
        use_stemmer=True
    )
    scores = scorer.score(reference, candidate)
    
    return {
        "rouge1": {
            "precision": scores['rouge1'].precision,
            "recall": scores['rouge1'].recall,
            "f1": scores['rouge1'].fmeasure
        },
        "rouge2": {
            "precision": scores['rouge2'].precision,
            "recall": scores['rouge2'].recall,
            "f1": scores['rouge2'].fmeasure
        },
        "rougeL": {
            "precision": scores['rougeL'].precision,
            "recall": scores['rougeL'].recall,
            "f1": scores['rougeL'].fmeasure
        }
    }

reference = "The quarterly revenue increased by 15% driven by strong cloud adoption."
candidate = "Revenue grew 15% in the quarter due to cloud services growth."

scores = evaluate_with_rouge(reference, candidate)
for metric, values in scores.items():
    print(f"{metric}: F1={values['f1']:.3f}")

ROUGE-1 measures unigram overlap, ROUGE-2 measures bigram overlap, and ROUGE-L measures the longest common subsequence. For summarization tasks, ROUGE-L is often the most informative because it rewards outputs that preserve the ordering of key information.

Semantic Similarity with Embeddings

Both BLEU and ROUGE fail when the candidate uses completely different words to express the same meaning. Embedding-based similarity solves this by comparing meaning rather than surface tokens.

from openai import OpenAI
import numpy as np

client = OpenAI()

def semantic_similarity(text_a: str, text_b: str) -> float:
    """Compute semantic similarity using embeddings."""
    response = client.embeddings.create(
        model="text-embedding-3-small",
        input=[text_a, text_b]
    )
    
    emb_a = np.array(response.data[0].embedding)
    emb_b = np.array(response.data[1].embedding)
    
    similarity = np.dot(emb_a, emb_b) / (
        np.linalg.norm(emb_a) * np.linalg.norm(emb_b)
    )
    return float(similarity)

# These say the same thing differently
text_a = "The server experienced downtime due to high traffic."
text_b = "Heavy load caused the server to go offline."

score = semantic_similarity(text_a, text_b)
print(f"Semantic similarity: {score:.4f}")  # High score despite different words

Embedding similarity is excellent for detecting paraphrases but poor at catching factual errors. “Revenue increased 15%” and “Revenue decreased 15%” would score very similarly despite being opposite in meaning.

LLM-as-Judge Evaluation

The most flexible modern approach uses a second LLM to evaluate the output of the first. This captures nuances that automated metrics miss entirely.

from openai import OpenAI
import json

client = OpenAI()

def llm_judge(
    question: str,
    response: str,
    criteria: list[str],
    reference: str = None
) -> dict:
    """Use an LLM to evaluate a response on multiple criteria."""
    
    criteria_text = "\n".join(f"- {c}" for c in criteria)
    
    reference_section = ""
    if reference:
        reference_section = f"\nReference answer: {reference}\n"
    
    judge_prompt = f"""Evaluate the following response on these criteria:
{criteria_text}

For each criterion, provide a score from 1 to 5 and a brief justification.

Question: {question}
{reference_section}
Response to evaluate: {response}

Return your evaluation as JSON with this structure:
{{
    "scores": {{
        "criterion_name": {{"score": 1-5, "justification": "..."}},
        ...
    }},
    "overall_score": 1-5,
    "summary": "Brief overall assessment"
}}"""

    result = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": "You are an expert evaluator. Be critical and precise."},
            {"role": "user", "content": judge_prompt}
        ],
        response_format={"type": "json_object"},
        temperature=0.1
    )
    
    return json.loads(result.choices[0].message.content)

# Usage
evaluation = llm_judge(
    question="Explain how photosynthesis works",
    response="Plants use sunlight to convert CO2 and water into glucose and oxygen through chlorophyll.",
    criteria=[
        "Factual accuracy",
        "Completeness",
        "Clarity of explanation",
        "Appropriate level of detail"
    ]
)
print(json.dumps(evaluation, indent=2))

LLM-as-judge is powerful but introduces its own biases. Studies show that LLM judges prefer longer responses, tend to favor their own outputs, and can be inconsistent across runs. Mitigate these issues by using low temperature, evaluating criteria independently, and running multiple evaluations to check agreement.

Building an Evaluation Pipeline

A production evaluation pipeline combines multiple metrics and runs automatically against a test dataset.

import json
from dataclasses import dataclass
from typing import Optional

@dataclass
class EvalCase:
    question: str
    reference_answer: str
    generated_answer: str
    metadata: dict = None

@dataclass
class EvalResult:
    case: EvalCase
    rouge_scores: dict
    semantic_sim: float
    llm_judge_score: dict
    
    @property
    def composite_score(self) -> float:
        """Weighted composite of all metrics."""
        rouge_f1 = self.rouge_scores.get("rougeL", {}).get("f1", 0)
        judge_score = self.llm_judge_score.get("overall_score", 3) / 5.0
        
        return (
            0.2 * rouge_f1 +
            0.3 * self.semantic_sim +
            0.5 * judge_score
        )

def run_evaluation(cases: list[EvalCase]) -> list[EvalResult]:
    """Run full evaluation pipeline on a set of test cases."""
    results = []
    
    for i, case in enumerate(cases):
        print(f"Evaluating case {i+1}/{len(cases)}...")
        
        # Automated metrics
        rouge = evaluate_with_rouge(case.reference_answer, case.generated_answer)
        sem_sim = semantic_similarity(case.reference_answer, case.generated_answer)
        
        # LLM judge
        judge = llm_judge(
            question=case.question,
            response=case.generated_answer,
            criteria=["Factual accuracy", "Completeness", "Clarity"],
            reference=case.reference_answer
        )
        
        result = EvalResult(
            case=case,
            rouge_scores=rouge,
            semantic_sim=sem_sim,
            llm_judge_score=judge
        )
        results.append(result)
    
    return results

def print_eval_summary(results: list[EvalResult]):
    """Print summary statistics for an evaluation run."""
    composite_scores = [r.composite_score for r in results]
    rouge_scores = [r.rouge_scores["rougeL"]["f1"] for r in results]
    sem_scores = [r.semantic_sim for r in results]
    
    print(f"\n{'='*50}")
    print(f"Evaluation Summary ({len(results)} cases)")
    print(f"{'='*50}")
    print(f"Composite Score: {sum(composite_scores)/len(composite_scores):.3f}")
    print(f"ROUGE-L F1:      {sum(rouge_scores)/len(rouge_scores):.3f}")
    print(f"Semantic Sim:    {sum(sem_scores)/len(sem_scores):.3f}")
    print(f"Min Composite:   {min(composite_scores):.3f}")
    print(f"Max Composite:   {max(composite_scores):.3f}")

Evaluation Anti-Patterns to Avoid

Several common mistakes undermine evaluation efforts:

Overfitting to metrics. If you optimize prompts to maximize BLEU or ROUGE, you will converge on outputs that parrot reference text rather than providing genuinely useful answers. Use metrics to detect regressions, not as optimization targets.

Single-metric evaluation. No single metric captures all quality dimensions. Factual accuracy, coherence, helpfulness, and safety are independent axes. A response can score perfectly on ROUGE while containing dangerous misinformation.

Small test sets. Evaluating on 5-10 examples gives you noise, not signal. Aim for at least 50-100 diverse test cases that cover edge cases and failure modes.

Ignoring distribution shift. Your test set should reflect real usage patterns. If 80% of your users ask about topic A but your test set is evenly distributed, your evaluation will not predict production quality.

Tracking Evaluation Over Time

Store evaluation results so you can track quality across prompt versions and model changes:

import json
from datetime import datetime

def log_evaluation_run(
    results: list[EvalResult],
    run_metadata: dict,
    output_file: str = "eval_log.jsonl"
):
    """Log evaluation results for tracking over time."""
    run_record = {
        "timestamp": datetime.utcnow().isoformat(),
        "metadata": run_metadata,
        "num_cases": len(results),
        "avg_composite": sum(r.composite_score for r in results) / len(results),
        "avg_rouge_l": sum(r.rouge_scores["rougeL"]["f1"] for r in results) / len(results),
        "avg_semantic_sim": sum(r.semantic_sim for r in results) / len(results),
        "failures": [
            {
                "question": r.case.question,
                "score": r.composite_score
            }
            for r in results if r.composite_score < 0.5
        ]
    }
    
    with open(output_file, "a") as f:
        f.write(json.dumps(run_record) + "\n")
    
    print(f"Logged evaluation run: {run_record['avg_composite']:.3f} avg composite")

# Run after each prompt change
log_evaluation_run(
    results=results,
    run_metadata={
        "model": "gpt-4o",
        "prompt_version": "v2.3",
        "change": "Added few-shot examples for financial questions"
    }
)

Wrapping Up

LLM evaluation requires a multi-layered approach. BLEU and ROUGE give you cheap, fast baselines for detecting regressions. Embedding similarity captures semantic equivalence that surface metrics miss. LLM-as-judge provides the most nuanced assessment but adds cost and its own biases. The best evaluation pipelines combine all three, run automatically on every change, and track results over time. Start by building a diverse test set of at least 50 cases, pick 2-3 metrics that align with your quality goals, and set up automated evaluation before you start iterating on prompts. What you cannot measure, you cannot improve.