LLM Parameters: Temperature, Top-P, and Beyond

How LLMs Choose Words

Before tuning parameters, you need to understand what happens when an LLM generates a response. At each step, the model produces a probability distribution over its entire vocabulary. The next token could be any word, but some are far more likely than others.

For the prompt “The capital of France is,” the model might assign:

Paris     -> 92.3%
Lyon      ->  1.8%
Marseille ->  0.9%
the       ->  0.7%
a         ->  0.4%
... thousands more tokens with tiny probabilities

Parameters like temperature and top-p control how the model picks from this distribution. They do not change the probabilities themselves, but they change the selection strategy.

Temperature

Temperature scales the probability distribution before sampling. It controls randomness.

Temperature 0.0:  Always pick the highest probability token
                 Paris (92.3%) -> ALWAYS selected

Temperature 0.7:  Slightly spread out, mostly top choices
                 Paris (85%), Lyon (5%), Marseille (3%), ...

Temperature 1.0:  Use probabilities as-is (model default)
                 Paris (92.3%), Lyon (1.8%), Marseille (0.9%), ...

Temperature 1.5:  Flatten the distribution, more randomness
                 Paris (60%), Lyon (10%), Marseille (8%), ...

Temperature 2.0:  Nearly uniform, highly random
                 Paris (30%), Lyon (15%), Marseille (12%), ...

Temperature 0 is deterministic. The model always picks the most likely token. Use this when you want the same answer every time: data extraction, classification, code generation.

Temperature 0.3-0.7 adds slight variation while keeping output sensible. Good for writing tasks where you want some creativity but not chaos.

Temperature 1.0+ increases randomness significantly. The model explores less likely tokens, which can produce creative or surprising output, but also nonsense.

import openai

client = openai.OpenAI()

# Deterministic: always the same output
response_deterministic = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "What is 2 + 2?"}],
    temperature=0
)
# Always: "4" or "2 + 2 = 4"

# Creative: different each time
response_creative = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Write a one-sentence story about a robot."}],
    temperature=1.0
)
# Run 1: "The last robot on Earth spent its days painting sunsets it had never seen."
# Run 2: "Unit 7 discovered that the humans had left behind not just their cities, but their loneliness."
# Run 3: "She was made of steel and starlight, and she dreamed in binary."

Top-P (Nucleus Sampling)

Top-p is an alternative to temperature for controlling randomness. Instead of scaling probabilities, it limits which tokens are even considered.

Top-p = 0.9 means: consider only the smallest set of tokens whose cumulative probability is at least 90%. Discard everything else.

Full distribution:
  Paris (92.3%), Lyon (1.8%), Marseille (0.9%), the (0.7%), ...

Top-p = 0.95:
  Paris (92.3%), Lyon (1.8%), Marseille (0.9%) = 95.0%
  -> Only these three tokens are candidates

Top-p = 0.5:
  Paris (92.3%) > 50% already
  -> Only Paris is a candidate (nearly deterministic)

Top-p = 1.0:
  All tokens are candidates (no filtering)

Tokens sorted by probability:
[Paris 92.3%] [Lyon 1.8%] [Marseille 0.9%] [the 0.7%] [a 0.4%] ...
|___________________________|
top-p = 0.95 cutoff

Only tokens within the cutoff are sampled from.
Everything after the line is discarded.

Do not combine temperature and top-p aggressively. Most API documentation recommends adjusting one and leaving the other at default. If you set temperature=0.8 and top_p=0.5, the effects interact unpredictably.

# Good: adjust one parameter
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": prompt}],
    temperature=0.7,  # Adjusted
    top_p=1.0          # Default
)

# Also good
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": prompt}],
    temperature=1.0,  # Default
    top_p=0.9          # Adjusted
)

# Avoid: both adjusted
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": prompt}],
    temperature=1.5,
    top_p=0.5  # Fighting each other
)

Max Tokens

Max tokens caps the length of the response. This is not a suggestion; it is a hard cutoff. If the model hits the limit mid-sentence, it stops.

# Short response: classification
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Is this email spam? 'You won a prize!' Answer yes or no."}],
    max_tokens=5  # Enough for "yes" or "no"
)

# Medium response: summary
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": f"Summarize this article in 2 paragraphs:\n{article}"}],
    max_tokens=300
)

# Long response: code generation
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Write a complete REST API for user management."}],
    max_tokens=4000
)

Setting max_tokens too low causes truncated output. Setting it too high wastes nothing (you only pay for tokens actually generated), but it removes a safety net against runaway responses.

Practical tip: Check response.choices[0].finish_reason. If it is "length", the model was cut off and you need a higher max_tokens. If it is "stop", the model finished naturally.

result = response.choices[0]
if result.finish_reason == "length":
    print("WARNING: Response was truncated. Increase max_tokens.")
elif result.finish_reason == "stop":
    print("Response completed normally.")

Frequency Penalty and Presence Penalty

These two parameters reduce repetition, but in different ways.

Frequency penalty (range: -2.0 to 2.0) reduces the probability of tokens proportional to how many times they have already appeared. Higher values make the model avoid repeating the same words.

Presence penalty (range: -2.0 to 2.0) reduces the probability of tokens that have appeared at all, regardless of how many times. It encourages the model to talk about new topics.

# Without penalty: model may repeat phrases
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "List 20 creative uses for a paperclip."}],
    frequency_penalty=0,
    presence_penalty=0
)
# May repeat: "You can use a paperclip to..." over and over

# With penalties: more diverse output
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "List 20 creative uses for a paperclip."}],
    frequency_penalty=0.5,   # Discourage repeated words
    presence_penalty=0.3     # Encourage new topics
)
# More varied vocabulary and ideas

When to use them:

• Brainstorming or idea generation: frequency_penalty=0.5, presence_penalty=0.5
• Normal writing: leave both at 0
• Code generation: leave both at 0 (code naturally repeats patterns like variable names)

Parameter Presets for Common Tasks

Here are tested parameter combinations for common use cases.

PRESETS = {
    "data_extraction": {
        "temperature": 0,
        "top_p": 1,
        "frequency_penalty": 0,
        "presence_penalty": 0,
        "max_tokens": 500,
    },
    "classification": {
        "temperature": 0,
        "top_p": 1,
        "frequency_penalty": 0,
        "presence_penalty": 0,
        "max_tokens": 10,
    },
    "code_generation": {
        "temperature": 0,
        "top_p": 1,
        "frequency_penalty": 0,
        "presence_penalty": 0,
        "max_tokens": 4000,
    },
    "creative_writing": {
        "temperature": 0.9,
        "top_p": 1,
        "frequency_penalty": 0.5,
        "presence_penalty": 0.5,
        "max_tokens": 2000,
    },
    "brainstorming": {
        "temperature": 1.0,
        "top_p": 0.95,
        "frequency_penalty": 0.7,
        "presence_penalty": 0.7,
        "max_tokens": 1500,
    },
    "summarization": {
        "temperature": 0.3,
        "top_p": 1,
        "frequency_penalty": 0,
        "presence_penalty": 0,
        "max_tokens": 500,
    },
    "conversation": {
        "temperature": 0.7,
        "top_p": 1,
        "frequency_penalty": 0.3,
        "presence_penalty": 0.3,
        "max_tokens": 1000,
    },
}

def call_llm(prompt: str, preset: str) -> str:
    params = PRESETS[preset]
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": prompt}],
        **params
    )
    return response.choices[0].message.content

Finding Optimal Parameters

Do not guess. Test systematically.

import itertools

def parameter_sweep(prompt: str, expected_output: str):
    """Test different parameter combinations and score results."""
    temperatures = [0, 0.3, 0.7, 1.0]
    top_ps = [0.9, 0.95, 1.0]

    results = []
    for temp, top_p in itertools.product(temperatures, top_ps):
        outputs = []
        for _ in range(3):  # 3 runs per combination
            response = client.chat.completions.create(
                model="gpt-4o",
                messages=[{"role": "user", "content": prompt}],
                temperature=temp,
                top_p=top_p
            )
            outputs.append(response.choices[0].message.content)

        # Measure consistency (how similar are the 3 runs?)
        unique_outputs = len(set(outputs))
        # Measure quality (does it match expected?)
        matches = sum(1 for o in outputs if expected_output.lower() in o.lower())

        results.append({
            "temperature": temp,
            "top_p": top_p,
            "consistency": 1 / unique_outputs,  # 1.0 = all same
            "accuracy": matches / 3,
            "sample": outputs[0][:100]
        })

    # Sort by accuracy first, then consistency
    results.sort(key=lambda x: (x["accuracy"], x["consistency"]), reverse=True)
    return results

Run this on 10-20 representative prompts from your actual use case. The best parameters for a customer support bot are different from the best parameters for a poetry generator.

Key Takeaways

Temperature and top-p control randomness: use 0 for deterministic tasks, 0.7-1.0 for creative tasks. Do not adjust both at once. Max tokens is a hard cutoff, not a suggestion: check finish_reason to detect truncation. Frequency and presence penalties reduce repetition for brainstorming and creative writing, but should be left at 0 for code and extraction tasks. Start with the presets above and test systematically on your actual data rather than guessing.