Multi-Turn Conversations: Context and Memory Patterns

How Multi-Turn Context Works

LLMs do not have memory. Every API call is stateless. When you send a “conversation,” you are actually sending the entire message history each time, and the model generates the next response based on everything it sees.

import openai

client = openai.OpenAI()

# Turn 1
messages = [
    {"role": "system", "content": "You are a helpful cooking assistant."},
    {"role": "user", "content": "I want to make pasta for 4 people."}
]
response = client.chat.completions.create(model="gpt-4o", messages=messages)
assistant_reply = response.choices[0].message.content
messages.append({"role": "assistant", "content": assistant_reply})

# Turn 2 - the model sees the FULL history
messages.append({"role": "user", "content": "What if two of them are gluten-free?"})
response = client.chat.completions.create(model="gpt-4o", messages=messages)
# The model knows we're talking about pasta for 4 people
# because it can see the entire conversation

This means every turn costs more tokens than the last because you resend everything. A 20-turn conversation where each turn averages 200 tokens costs roughly 20 x 200 = 4,000 input tokens on the final call, plus all the turns before it.

Turn 1:  system + user_1                          = ~100 tokens
Turn 2:  system + user_1 + asst_1 + user_2         = ~300 tokens  
Turn 3:  system + user_1 + asst_1 + user_2 + asst_2 + user_3 = ~500 tokens
...
Turn 20: system + all 19 previous turns + user_20   = ~4000 tokens

Each turn sends everything again. Cost grows quadratically.

The Context Window Problem

Every model has a context window limit. GPT-4o supports 128K tokens. Claude supports 200K. But practical limits are tighter than theoretical ones:

• Cost: You pay per token. Long contexts are expensive.
• Latency: More input tokens = slower response time.
• Attention degradation: Models pay less attention to information in the middle of very long contexts (the “lost in the middle” phenomenon).

You need strategies to keep conversations within practical limits while preserving important context.

Strategy 1: Sliding Window

Keep only the most recent N turns. Simple, cheap, and works for casual conversations.

class SlidingWindowChat:
    def __init__(self, system_prompt: str, max_turns: int = 10):
        self.system_prompt = system_prompt
        self.history = []
        self.max_turns = max_turns

    def send(self, user_message: str) -> str:
        self.history.append({"role": "user", "content": user_message})

        # Keep only the last N turns (each turn = user + assistant)
        if len(self.history) > self.max_turns * 2:
            self.history = self.history[-(self.max_turns * 2):]

        messages = [
            {"role": "system", "content": self.system_prompt},
            *self.history
        ]

        response = client.chat.completions.create(
            model="gpt-4o",
            messages=messages,
            temperature=0.7
        )
        assistant_reply = response.choices[0].message.content
        self.history.append({"role": "assistant", "content": assistant_reply})
        return assistant_reply

chat = SlidingWindowChat(
    system_prompt="You are a customer support agent for a SaaS product.",
    max_turns=10
)

Limitation: The model forgets everything beyond the window. If the user said their name in turn 1 and you are now on turn 15, the name is gone.

Strategy 2: Summarization

Periodically summarize old turns and replace them with the summary.

class SummarizedChat:
    def __init__(self, system_prompt: str, summarize_after: int = 8):
        self.system_prompt = system_prompt
        self.history = []
        self.summary = ""
        self.summarize_after = summarize_after
        self.turn_count = 0

    def _summarize_history(self):
        """Compress old conversation into a summary."""
        old_messages = self.history[:-4]  # Keep last 2 turns intact
        recent_messages = self.history[-4:]

        summary_prompt = [
            {"role": "system", "content": "Summarize this conversation in 2-3 sentences. Include all key facts, decisions, and user preferences mentioned."},
            {"role": "user", "content": "\n".join(
                f"{m['role']}: {m['content']}" for m in old_messages
            )}
        ]

        response = client.chat.completions.create(
            model="gpt-4o-mini",  # Use a cheap model for summarization
            messages=summary_prompt,
            temperature=0,
            max_tokens=200
        )

        self.summary = response.choices[0].message.content
        self.history = recent_messages  # Replace history with recent only

    def send(self, user_message: str) -> str:
        self.history.append({"role": "user", "content": user_message})
        self.turn_count += 1

        # Summarize when history gets long
        if self.turn_count % self.summarize_after == 0 and len(self.history) > 6:
            self._summarize_history()

        # Build messages with summary context
        system_content = self.system_prompt
        if self.summary:
            system_content += f"\n\nPrevious conversation summary:\n{self.summary}"

        messages = [
            {"role": "system", "content": system_content},
            *self.history
        ]

        response = client.chat.completions.create(
            model="gpt-4o",
            messages=messages,
            temperature=0.7
        )
        assistant_reply = response.choices[0].message.content
        self.history.append({"role": "assistant", "content": assistant_reply})
        return assistant_reply

This approach keeps costs constant regardless of conversation length while preserving key facts from earlier turns.

Strategy 3: Entity Memory

Track specific entities (names, preferences, decisions) in a structured store.

import json

class EntityMemoryChat:
    def __init__(self, system_prompt: str):
        self.system_prompt = system_prompt
        self.history = []
        self.entities = {}  # Key facts extracted from conversation

    def _extract_entities(self, conversation_turn: str):
        """Extract key entities from the latest exchange."""
        response = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[
                {"role": "system", "content": """Extract key facts from this conversation exchange.
Return JSON with entity types as keys and values.
Only include facts that would be important to remember later.
Example: {"user_name": "Sarah", "budget": "$5000", "preference": "window seat"}
If no important facts, return {}."""},
                {"role": "user", "content": conversation_turn}
            ],
            response_format={"type": "json_object"},
            temperature=0,
            max_tokens=200
        )
        new_entities = json.loads(response.choices[0].message.content)
        self.entities.update(new_entities)

    def send(self, user_message: str) -> str:
        self.history.append({"role": "user", "content": user_message})

        # Build context with entity memory
        system_content = self.system_prompt
        if self.entities:
            entity_text = "\n".join(f"- {k}: {v}" for k, v in self.entities.items())
            system_content += f"\n\nKnown facts about this user:\n{entity_text}"

        # Keep only recent history for conversation flow
        recent_history = self.history[-6:]

        messages = [
            {"role": "system", "content": system_content},
            *recent_history
        ]

        response = client.chat.completions.create(
            model="gpt-4o",
            messages=messages,
            temperature=0.7
        )
        assistant_reply = response.choices[0].message.content
        self.history.append({"role": "assistant", "content": assistant_reply})

        # Extract entities from this exchange
        exchange = f"User: {user_message}\nAssistant: {assistant_reply}"
        self._extract_entities(exchange)

        return assistant_reply

# Usage
chat = EntityMemoryChat("You are a travel booking assistant.")
chat.send("Hi, I'm Marco. I'm planning a trip to Japan in October.")
chat.send("I prefer business class and boutique hotels under $300/night.")
# entities: {"user_name": "Marco", "destination": "Japan",
#            "travel_month": "October", "flight_class": "business",
#            "hotel_preference": "boutique", "hotel_budget": "$300/night"}

# 20 turns later, the model still knows Marco's name and preferences
# because they're injected via the entity store, not conversation history

Conversation Design Patterns

Slot Filling

Guide the user through collecting required information.

SLOT_FILLING_SYSTEM = """You are a restaurant reservation assistant.

You need to collect these details before making a reservation:
- Party size (required)
- Date (required)
- Time (required)
- Name for the reservation (required)
- Dietary restrictions (optional, ask once)
- Seating preference: indoor/outdoor/no preference (optional)

RULES:
- Ask for missing information naturally, 1-2 fields at a time
- Do not ask for information already provided
- Once all required fields are collected, confirm the details
- If the user changes a field, update it and re-confirm

COLLECTED SO FAR:
{collected_slots}

MISSING REQUIRED:
{missing_slots}"""

class SlotFillingChat:
    def __init__(self):
        self.slots = {
            "party_size": None,
            "date": None,
            "time": None,
            "name": None,
            "dietary": None,
            "seating": None
        }
        self.required = {"party_size", "date", "time", "name"}
        self.history = []

    def _update_slots(self, user_message: str, assistant_reply: str):
        """Use LLM to extract slot values from the conversation."""
        response = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[
                {"role": "system", "content": f"""Extract reservation details from this exchange.
Current slots: {json.dumps(self.slots)}
Return JSON with only the slots that have new values. Use null for unchanged slots."""},
                {"role": "user", "content": f"User: {user_message}\nAssistant: {assistant_reply}"}
            ],
            response_format={"type": "json_object"},
            temperature=0
        )
        updates = json.loads(response.choices[0].message.content)
        for k, v in updates.items():
            if v is not None and k in self.slots:
                self.slots[k] = v

    def send(self, user_message: str) -> str:
        self.history.append({"role": "user", "content": user_message})

        collected = {k: v for k, v in self.slots.items() if v is not None}
        missing = self.required - set(collected.keys())

        system = SLOT_FILLING_SYSTEM.format(
            collected_slots=json.dumps(collected, indent=2) if collected else "None yet",
            missing_slots=", ".join(missing) if missing else "All required fields collected!"
        )

        messages = [{"role": "system", "content": system}, *self.history[-6:]]
        response = client.chat.completions.create(
            model="gpt-4o", messages=messages, temperature=0.7
        )
        reply = response.choices[0].message.content
        self.history.append({"role": "assistant", "content": reply})

        self._update_slots(user_message, reply)
        return reply

Guided Conversation with Phases

Structure the conversation into phases, each with different behavior.

PHASE_PROMPTS = {
    "greeting": """You are starting a customer support conversation.
Greet the user warmly and ask how you can help today.
Keep it to 1-2 sentences.""",

    "diagnosis": """You are diagnosing a customer's technical issue.
Ask focused questions to narrow down the problem.
Ask one question at a time. Do not suggest solutions yet.""",

    "solution": """You have diagnosed the issue. Now provide the solution.
Give step-by-step instructions. Ask if each step worked before
moving to the next. If the solution doesn't work, escalate.""",

    "closing": """The issue is resolved. Summarize what was done,
ask if there's anything else, and close the conversation."""
}

Phase management keeps the conversation focused and prevents the model from jumping to solutions before understanding the problem.

Token-Efficient Patterns

Reduce token usage without losing context quality.

def compress_message(message: str, max_length: int = 200) -> str:
    """Compress a long message while preserving key information."""
    if len(message.split()) <= max_length:
        return message

    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Compress this message to its key information. Keep facts, remove filler. Be concise."},
            {"role": "user", "content": message}
        ],
        temperature=0,
        max_tokens=max_length
    )
    return response.choices[0].message.content

def prepare_efficient_history(history: list[dict], max_tokens: int = 2000) -> list[dict]:
    """Prepare conversation history within a token budget."""
    # Always keep the most recent 2 exchanges intact
    recent = history[-4:] if len(history) >= 4 else history
    older = history[:-4] if len(history) > 4 else []

    if not older:
        return recent

    # Compress older messages
    compressed = []
    for msg in older:
        compressed.append({
            "role": msg["role"],
            "content": compress_message(msg["content"], max_length=50)
        })

    return compressed + recent

Key Takeaways

Multi-turn conversations require explicit context management because LLMs are stateless. Choose your strategy based on conversation length: sliding window for short chats, summarization for medium ones, and entity memory for long conversations where specific facts matter. Design conversations with phases and slot filling to guide users efficiently. Monitor token usage because costs grow with every turn. And always test your memory strategy by asking the model about information from early turns to verify it is retained.