LLM Hallucination Detection and Mitigation

LLMs generate text that sounds confident and fluent regardless of whether the content is accurate. This behavior, called hallucination, is not a bug that will be patched away. It is a fundamental property of how language models work — they predict probable next tokens, not truthful ones. Any production system using LLMs needs explicit strategies for detecting and reducing hallucinations.

Types of Hallucination

Not all hallucinations are the same, and different types require different mitigations.

Intrinsic hallucination contradicts the provided source material. If you give the model a document stating revenue was $5M and it says $8M, that is an intrinsic hallucination. These are detectable by comparing the output against the input.

Extrinsic hallucination introduces claims that cannot be verified from the provided context. The model might add plausible-sounding details that are not in the source document. These are harder to catch because the information might be true — it is just not grounded in the given context.

Fabrication is the most dangerous type: the model invents entities, citations, statistics, or events that do not exist. Fabricated legal citations, fake research papers, and invented API endpoints fall into this category.

Detection Strategy 1: Self-Consistency Checking

Generate multiple responses to the same prompt and check for consistency. Hallucinated details tend to vary between runs, while factual content remains stable.

from openai import OpenAI
from collections import Counter

client = OpenAI()

def self_consistency_check(
    messages: list[dict],
    num_samples: int = 5,
    model: str = "gpt-4o",
) -> dict:
    """Generate multiple responses and check for consistency."""
    responses = []
    for _ in range(num_samples):
        response = client.chat.completions.create(
            model=model,
            messages=messages,
            temperature=0.7,
        )
        responses.append(response.choices[0].message.content)

    # Ask the model to identify inconsistencies
    comparison_prompt = (
        "Compare these responses to the same question. "
        "Identify any claims that appear in some responses but not others. "
        "For each inconsistent claim, note how many responses include it. "
        "Return JSON with 'consistent_claims' and 'inconsistent_claims' arrays."
    )

    formatted = "\n---\n".join(
        f"Response {i+1}:\n{r}" for i, r in enumerate(responses)
    )

    analysis = client.chat.completions.create(
        model=model,
        messages=[
            {"role": "system", "content": comparison_prompt},
            {"role": "user", "content": formatted},
        ],
        response_format={"type": "json_object"},
    )

    import json
    return json.loads(analysis.choices[0].message.content)

Claims that appear in all five responses are likely factual (or at least consistently represented in the model’s training data). Claims appearing in only one or two responses are suspect.

Detection Strategy 2: Source Verification

When working with grounded generation (RAG or document Q&A), verify that the model’s claims actually appear in the source material.

def verify_against_source(
    response: str,
    source_documents: list[str],
    model: str = "gpt-4o",
) -> dict:
    """Check if claims in the response are supported by sources."""
    sources_text = "\n---\n".join(
        f"Source {i+1}:\n{doc}" for i, doc in enumerate(source_documents)
    )

    verification = client.chat.completions.create(
        model=model,
        messages=[
            {
                "role": "system",
                "content": (
                    "You are a fact-checker. For each factual claim in the "
                    "response, determine if it is SUPPORTED, CONTRADICTED, "
                    "or NOT_FOUND in the source documents. Return JSON with "
                    "an array of objects, each having 'claim', 'verdict', "
                    "and 'source_reference' fields."
                ),
            },
            {
                "role": "user",
                "content": (
                    f"Response to verify:\n{response}\n\n"
                    f"Source documents:\n{sources_text}"
                ),
            },
        ],
        response_format={"type": "json_object"},
    )

    import json
    result = json.loads(verification.choices[0].message.content)
    claims = result.get("claims", [])

    supported = [c for c in claims if c["verdict"] == "SUPPORTED"]
    contradicted = [c for c in claims if c["verdict"] == "CONTRADICTED"]
    unverified = [c for c in claims if c["verdict"] == "NOT_FOUND"]

    return {
        "supported": supported,
        "contradicted": contradicted,
        "unverified": unverified,
        "groundedness_score": len(supported) / max(len(claims), 1),
    }

A groundedness score below 0.7 should trigger a warning or regeneration. Contradicted claims should always be flagged or removed.

Mitigation 1: Grounded Generation

Force the model to only use information from provided sources. This is the single most effective mitigation for factual applications.

GROUNDED_SYSTEM_PROMPT = """Answer the user's question using ONLY the
information in the provided documents. Follow these rules strictly:

1. If the answer is not in the documents, say "I don't have information
   about that in the provided documents."
2. Never add information from your general knowledge.
3. For every claim, include a citation like [Source 1] or [Source 2].
4. If the documents contain conflicting information, note the conflict
   and cite both sources.
5. If you are uncertain about an interpretation, say so explicitly."""

def grounded_answer(
    question: str,
    documents: list[str],
    model: str = "gpt-4o",
) -> str:
    doc_text = "\n---\n".join(
        f"[Source {i+1}]:\n{doc}" for i, doc in enumerate(documents)
    )

    response = client.chat.completions.create(
        model=model,
        messages=[
            {"role": "system", "content": GROUNDED_SYSTEM_PROMPT},
            {
                "role": "user",
                "content": f"Documents:\n{doc_text}\n\nQuestion: {question}",
            },
        ],
        temperature=0,  # Low temperature reduces creative fabrication
    )
    return response.choices[0].message.content

Key elements: explicit instructions to avoid general knowledge, mandatory citations, and low temperature. The citation requirement makes it easy to verify claims and gives users transparency into the model’s reasoning.

Mitigation 2: Structured Extraction over Free Generation

Free-form text generation gives the model maximum opportunity to hallucinate. Constrained, structured outputs reduce this by limiting what the model can produce.

from pydantic import BaseModel, Field

class VerifiedAnswer(BaseModel):
    answer: str = Field(description="Direct answer to the question")
    confidence: str = Field(description="HIGH, MEDIUM, or LOW")
    sources_used: list[int] = Field(description="Source indices referenced")
    caveats: list[str] = Field(
        description="Any uncertainties or limitations",
        default_factory=list,
    )

def structured_answer(question: str, documents: list[str]) -> VerifiedAnswer:
    doc_text = "\n---\n".join(
        f"[{i+1}]: {doc}" for i, doc in enumerate(documents)
    )

    response = client.beta.chat.completions.parse(
        model="gpt-4o",
        messages=[
            {
                "role": "system",
                "content": (
                    "Answer based only on the provided documents. "
                    "Set confidence to LOW if the documents do not "
                    "clearly support the answer."
                ),
            },
            {
                "role": "user",
                "content": f"Documents:\n{doc_text}\n\nQuestion: {question}",
            },
        ],
        response_format=VerifiedAnswer,
        temperature=0,
    )
    return response.choices[0].message.parsed

The confidence and caveats fields give the model an explicit channel to express uncertainty instead of masking it with confident-sounding text.

Mitigation 3: Chain-of-Verification

Have the model generate an answer, then systematically verify its own claims in a second pass.

def chain_of_verification(question: str, context: str) -> dict:
    # Step 1: Generate initial answer
    initial = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": "Answer based on the context provided."},
            {"role": "user", "content": f"Context: {context}\n\nQuestion: {question}"},
        ],
    ).choices[0].message.content

    # Step 2: Extract verification questions
    verif_questions = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {
                "role": "system",
                "content": (
                    "List specific factual claims from this answer that "
                    "can be verified. For each claim, write a yes/no "
                    "verification question. Return as a JSON array of strings."
                ),
            },
            {"role": "user", "content": initial},
        ],
        response_format={"type": "json_object"},
    ).choices[0].message.content

    import json
    questions = json.loads(verif_questions).get("questions", [])

    # Step 3: Verify each claim against context
    verified = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {
                "role": "system",
                "content": (
                    "Answer each verification question using ONLY the context. "
                    "For each, respond YES, NO, or CANNOT_VERIFY."
                ),
            },
            {
                "role": "user",
                "content": (
                    f"Context: {context}\n\n"
                    f"Questions: {json.dumps(questions)}"
                ),
            },
        ],
    ).choices[0].message.content

    # Step 4: Revise answer removing unverified claims
    revised = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {
                "role": "system",
                "content": "Revise the answer, removing or correcting any "
                           "claims that could not be verified.",
            },
            {
                "role": "user",
                "content": (
                    f"Original answer: {initial}\n\n"
                    f"Verification results: {verified}\n\n"
                    f"Context: {context}"
                ),
            },
        ],
    ).choices[0].message.content

    return {"initial": initial, "revised": revised, "verification": verified}

This pattern trades latency (four LLM calls instead of one) for accuracy. Use it for high-stakes outputs where correctness matters more than speed.

Measuring Hallucination Rates

Track hallucination metrics across your application to identify trends and measure the impact of mitigations.

@dataclass
class HallucinationMetrics:
    total_responses: int = 0
    verified_responses: int = 0
    avg_groundedness: float = 0.0
    contradictions_found: int = 0

    def record(self, groundedness_score: float, contradictions: int):
        self.total_responses += 1
        self.verified_responses += 1
        # Running average
        self.avg_groundedness = (
            (self.avg_groundedness * (self.verified_responses - 1)
             + groundedness_score) / self.verified_responses
        )
        self.contradictions_found += contradictions

A groundedness score that drops over time may indicate that your retrieval system is returning less relevant documents, forcing the model to rely more on general knowledge.

Summary

Hallucination is inherent to how LLMs work, so mitigation must be built into your system architecture. Use grounded generation with mandatory citations for factual applications. Add source verification to catch contradictions. Use structured outputs to constrain the model’s response space. Apply chain-of-verification for high-stakes decisions. Measure groundedness continuously and treat declining scores as a system health signal.