LLM Context Window Management Strategies

Every LLM has a context window — the maximum number of tokens it can process in a single request. When your conversation or document exceeds this limit, the API returns an error or silently drops content. For production applications that handle multi-turn conversations, long documents, or complex tool-use sequences, context window management is not optional.

This guide covers practical strategies for keeping your prompts within limits while preserving the information the model needs.

Understanding Context Windows

A context window includes everything sent to the model: the system prompt, all conversation messages, tool definitions, and the expected output. Common limits as of 2025:

Model	Context Window
GPT-4o	128K tokens
Claude 3.5 Sonnet	200K tokens
Llama 3 (8B)	8K tokens
Gemini 1.5 Pro	1M tokens

Larger windows do not eliminate the problem. They increase cost (you pay per token), add latency (processing 100K tokens is slower than 10K), and can reduce output quality (models sometimes lose focus on relevant details within very long contexts).

Token Counting

Before managing context, you need to measure it. Count tokens proactively so you never hit the limit unexpectedly.

import tiktoken

def count_tokens(messages: list[dict], model: str = "gpt-4o") -> int:
    """Count tokens for a list of chat messages."""
    encoding = tiktoken.encoding_for_model(model)
    token_count = 0

    for message in messages:
        # Every message has role/content overhead
        token_count += 4  # message framing tokens
        for key, value in message.items():
            if isinstance(value, str):
                token_count += len(encoding.encode(value))

    token_count += 2  # reply priming tokens
    return token_count

def tokens_remaining(
    messages: list[dict],
    model: str = "gpt-4o",
    max_tokens: int = 128000,
    reserve_output: int = 4096,
) -> int:
    """Check how many tokens are available before hitting the limit."""
    used = count_tokens(messages, model)
    return max_tokens - reserve_output - used

Always reserve tokens for the model’s output. If your context window is 128K tokens and you fill 127K, the model has almost no room to respond.

Strategy 1: Sliding Window Truncation

The simplest approach drops the oldest messages when the conversation exceeds the limit. Keep the system prompt and the most recent messages.

def sliding_window(
    system_prompt: str,
    messages: list[dict],
    max_tokens: int = 120000,
    model: str = "gpt-4o",
) -> list[dict]:
    """Keep system prompt + most recent messages within token limit."""
    system_msg = {"role": "system", "content": system_prompt}
    system_tokens = count_tokens([system_msg], model)
    available = max_tokens - system_tokens

    # Add messages from newest to oldest
    kept = []
    running_total = 0

    for msg in reversed(messages):
        msg_tokens = count_tokens([msg], model)
        if running_total + msg_tokens > available:
            break
        kept.insert(0, msg)
        running_total += msg_tokens

    return [system_msg] + kept

This works well for casual conversations but has a critical weakness: the model loses all context from dropped messages. A user referencing something they said 50 messages ago gets a confused response.

Strategy 2: Conversation Summarization

Instead of dropping old messages entirely, summarize them. The model retains the key context from earlier in the conversation without the full token cost.

from openai import OpenAI

client = OpenAI()

def summarize_messages(messages: list[dict]) -> str:
    """Compress a block of messages into a summary."""
    formatted = "\n".join(
        f"{m['role'].upper()}: {m['content']}" for m in messages
    )

    response = client.chat.completions.create(
        model="gpt-4o-mini",  # Use a smaller model for summarization
        messages=[
            {
                "role": "system",
                "content": (
                    "Summarize this conversation concisely. Preserve: "
                    "key decisions, user preferences, important facts, "
                    "and any unresolved questions. Use bullet points."
                ),
            },
            {"role": "user", "content": formatted},
        ],
        max_tokens=500,
        temperature=0,
    )
    return response.choices[0].message.content

def managed_conversation(
    system_prompt: str,
    messages: list[dict],
    max_tokens: int = 100000,
    summary_threshold: int = 80000,
    model: str = "gpt-4o",
) -> list[dict]:
    """Summarize older messages when approaching the token limit."""
    system_msg = {"role": "system", "content": system_prompt}
    total = count_tokens([system_msg] + messages, model)

    if total <= summary_threshold:
        return [system_msg] + messages

    # Split: summarize the older half, keep the newer half
    midpoint = len(messages) // 2
    older = messages[:midpoint]
    newer = messages[midpoint:]

    summary = summarize_messages(older)
    summary_msg = {
        "role": "system",
        "content": f"Summary of earlier conversation:\n{summary}",
    }

    return [system_msg, summary_msg] + newer

Use a cheaper, faster model for summarization since it is a simple task that does not need top-tier reasoning. Summarize in the background when token usage crosses a threshold, not when you hit the limit.

Strategy 3: Hierarchical Memory

For long-running agents, maintain multiple memory levels with different granularity and storage mechanisms.

from dataclasses import dataclass, field

@dataclass
class HierarchicalMemory:
    """Three-tier memory: working, short-term, long-term."""

    # Working memory: last N messages, full detail
    working: list[dict] = field(default_factory=list)
    working_limit: int = 20

    # Short-term: summarized conversation segments
    short_term: list[str] = field(default_factory=list)
    short_term_limit: int = 5

    # Long-term: key facts extracted across sessions
    long_term: list[str] = field(default_factory=list)

    def add_message(self, message: dict):
        self.working.append(message)

        if len(self.working) > self.working_limit:
            # Summarize oldest messages into short-term
            overflow = self.working[:10]
            self.working = self.working[10:]
            summary = summarize_messages(overflow)
            self.short_term.append(summary)

        if len(self.short_term) > self.short_term_limit:
            # Compress oldest short-term summaries into long-term facts
            old_summary = self.short_term.pop(0)
            facts = extract_key_facts(old_summary)
            self.long_term.extend(facts)

    def build_context(self, system_prompt: str) -> list[dict]:
        messages = [{"role": "system", "content": system_prompt}]

        if self.long_term:
            facts = "\n".join(f"- {f}" for f in self.long_term)
            messages.append({
                "role": "system",
                "content": f"Key facts from past conversations:\n{facts}",
            })

        if self.short_term:
            summaries = "\n---\n".join(self.short_term)
            messages.append({
                "role": "system",
                "content": f"Recent conversation summaries:\n{summaries}",
            })

        messages.extend(self.working)
        return messages

def extract_key_facts(summary: str) -> list[str]:
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {
                "role": "system",
                "content": "Extract key facts as a JSON array of short strings. "
                           "Only include facts that would be useful in future "
                           "conversations. Discard small talk and resolved issues.",
            },
            {"role": "user", "content": summary},
        ],
        response_format={"type": "json_object"},
    )
    import json
    result = json.loads(response.choices[0].message.content)
    return result.get("facts", [])

Strategy 4: Retrieval-Augmented Context

For document-heavy applications, store content externally and retrieve only the relevant portions for each query.

def retrieval_context(
    query: str,
    system_prompt: str,
    messages: list[dict],
    documents: list[str],
    max_doc_tokens: int = 4000,
) -> list[dict]:
    """Retrieve relevant document chunks instead of including everything."""
    # Embed and rank documents by relevance to the query
    relevant_chunks = retrieve_relevant(query, documents, top_k=3)

    # Build context with only relevant chunks
    doc_context = "\n---\n".join(relevant_chunks)
    system_with_docs = (
        f"{system_prompt}\n\n"
        f"Relevant documentation:\n{doc_context}"
    )

    return [{"role": "system", "content": system_with_docs}] + messages

This approach is covered in detail in RAG-focused articles, but the key insight for context management is that you should never dump entire documents into the context when only specific sections are relevant.

Choosing the Right Strategy

Scenario	Recommended Strategy
Simple chatbot, short conversations	Sliding window
Customer support with session history	Summarization
Long-running autonomous agents	Hierarchical memory
Document Q&A applications	Retrieval-augmented context
Multi-day user sessions	Hierarchical + retrieval

Most production systems combine two or more strategies. A support bot might use summarization for conversation history and retrieval for knowledge base access.

Summary

Context window management is a core infrastructure concern for any LLM application. Count tokens proactively, not reactively. Use sliding windows for simple cases, summarization when historical context matters, hierarchical memory for long-running agents, and retrieval for document-heavy workflows. Always reserve space for the model’s output and use cheaper models for summarization tasks.