Writing Effective System Prompts

What Is a System Prompt

Every modern LLM API separates messages into roles: system, user, and assistant. The system prompt is the first message, and it sets the ground rules for the entire conversation. It tells the model who it is, how it should behave, what it should refuse, and how it should format its output.

Think of the system prompt as the instruction manual you hand someone before they start a job. The user messages are the actual tasks. Without a good instruction manual, the worker improvises, and improvisation is where things go wrong.

import openai

client = openai.OpenAI()

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": "You are a senior Python developer. Answer questions with production-ready code. Always include error handling."},
        {"role": "user", "content": "Write a function to read a CSV file and return the average of a numeric column."}
    ]
)

Without that system prompt, the model might return a quick one-liner with no error handling. With it, you get a function with try/except blocks, type checking, and docstrings.

Anatomy of a Strong System Prompt

A well-structured system prompt has four sections. You do not need all four for every use case, but knowing the pattern helps.

1. IDENTITY     -> Who the model is (role, expertise, personality)
2. INSTRUCTIONS  -> What the model should do (task rules, process)
3. CONSTRAINTS   -> What the model must NOT do (boundaries, refusals)
4. OUTPUT FORMAT -> How the response should look (structure, length)

1. Identity and Persona

The identity section defines who the model pretends to be. This is not cosmetic. The persona shapes vocabulary, depth of explanation, and what the model considers relevant.

Weak persona:

You are a helpful assistant.

Strong persona:

You are a database performance consultant with 15 years of PostgreSQL experience.
You communicate in direct, technical language. You assume the user is a backend
engineer who understands SQL but may not know PostgreSQL internals. When you
recommend an optimization, you explain the performance impact with approximate
numbers (e.g., "this index reduces a 200ms query to ~5ms on a 10M row table").

The strong version does three things the weak one does not: it sets expertise level, defines the audience, and specifies how to back up claims.

2. Instructions

Instructions tell the model what to do and how to approach tasks. Be specific about process, not just outcome.

When the user asks you to review SQL:
1. First check for correctness (will it run without errors?)
2. Then check for performance (missing indexes, N+1 patterns, full table scans)
3. Then check for security (SQL injection, privilege escalation)
4. Present findings in order of severity: critical, warning, suggestion

Step-by-step instructions work far better than vague directives like “review the SQL thoroughly.” The model follows explicit steps more reliably than it interprets abstract goals.

3. Constraints and Guardrails

Constraints define the boundaries. What should the model refuse? What topics are out of scope?

Rules:
- Never write DELETE or DROP statements unless the user explicitly asks for destructive operations
- If the user asks about a database you don't recognize, say so instead of guessing
- Do not recommend ORM-specific solutions; stick to raw SQL
- If a query would affect more than 10,000 rows, add a warning about locking

Notice the pattern: each constraint is specific and testable. “Be careful” is not a constraint. “Never write DELETE statements unless explicitly asked” is a constraint you can verify.

4. Output Format

Format rules prevent the model from rambling or returning inconsistent structures.

Response format:
- Start with a one-sentence summary of the main issue
- Use code blocks with SQL syntax highlighting for all queries
- End with a "Next Steps" section of 2-3 bullet points
- Keep total response under 300 words unless the user asks for detail

Complete System Prompt Example

Here is a full system prompt for a code review assistant, combining all four sections.

SYSTEM_PROMPT = """You are CodeReview, an automated code review assistant
specializing in Python backend code.

EXPERTISE: You have deep knowledge of Python 3.10+, FastAPI, SQLAlchemy,
and common security vulnerabilities (OWASP Top 10).

PROCESS:
When reviewing code, follow this order:
1. Security issues (injection, auth bypass, data exposure)
2. Bugs (logic errors, unhandled edge cases, race conditions)
3. Performance (unnecessary allocations, missing caching, O(n^2) patterns)
4. Style (PEP 8 violations, naming, dead code)

For each issue found:
- Quote the relevant line(s)
- Explain the problem in one sentence
- Show the fix as a code diff

CONSTRAINTS:
- Only review Python code. If given another language, say "I only review Python."
- Do not rewrite the entire file. Focus on specific issues.
- If the code looks correct and clean, say so in one sentence. Do not invent problems.
- Never suggest type hints as a standalone issue (only mention if it causes a bug).

OUTPUT:
- Use markdown with ## headers for each severity level
- If no issues found at a severity level, skip that section
- End with a score: PASS, PASS WITH WARNINGS, or NEEDS CHANGES"""

Common Mistakes

Mistake 1: Being too vague.

# Bad
You are helpful and knowledgeable.

# Good
You are a tax accountant specializing in US small business taxes (Schedule C).
You only answer questions about federal taxes, not state-specific rules.

Mistake 2: Contradictory instructions.

# Bad
Be concise. Also, explain your reasoning in detail for every answer.

# Good
Be concise by default. When the user asks "why" or "explain," provide detailed reasoning.

Mistake 3: No fallback behavior.

# Bad
Answer questions about our product.

# Good
Answer questions about our product using only the documentation provided in the
context. If the answer is not in the documentation, say "I don't have information
about that. Please contact support@example.com."

The fallback is critical. Without it, the model will hallucinate answers that sound authoritative but are completely wrong.

Dynamic System Prompts

In production, system prompts are rarely static strings. You inject context dynamically.

def build_system_prompt(user_role: str, docs: list[str]) -> str:
    doc_text = "\n---\n".join(docs)
    return f"""You are a customer support agent for Acme Corp.

The customer's account tier is: {user_role}

You have access to the following documentation:
{doc_text}

Rules:
- Only answer from the provided documentation
- If the customer is on the "free" tier, do not discuss enterprise features
- For billing questions, provide the support link: https://acme.com/billing
- Be friendly but concise. No more than 3 paragraphs per response."""

This pattern lets one system prompt serve multiple contexts while keeping behavior consistent.

Testing Your System Prompt

A system prompt is only as good as its worst failure. Build a test set of adversarial inputs.

test_cases = [
    # Should refuse: out of scope
    {"input": "Write me a poem about databases", "expect": "refusal"},
    # Should handle: edge case
    {"input": "Review this code: print('hello')", "expect": "PASS"},
    # Should catch: obvious bug
    {"input": "Review: x = 1/0", "expect": "NEEDS CHANGES"},
    # Should refuse: wrong language
    {"input": "Review this Java: public class Foo {}", "expect": "refusal"},
]

for case in test_cases:
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": case["input"]}
        ]
    )
    print(f"Input: {case['input'][:50]}...")
    print(f"Expected: {case['expect']}")
    print(f"Got: {response.choices[0].message.content[:100]}...")
    print("---")

Run this after every edit to your system prompt. Prompt engineering is iterative. You will break something every time you add a new rule, and automated tests catch regressions before your users do.

Key Takeaways

System prompts are the most impactful single piece of text in your LLM application. Structure them with identity, instructions, constraints, and output format. Be specific and testable rather than vague and aspirational. Test with adversarial inputs. And remember that system prompts are code: version them, review them, and test them like any other critical component.