LLM Context Window: Strategies for Long Documents

Large language models are powerful, but they have a hard constraint: the context window. Whether you are working with a 4K, 128K, or even a 1M token model, eventually your documents will exceed the limit. Even when they fit, stuffing the entire context window degrades quality and increases cost. This guide walks you through battle-tested strategies for handling long documents effectively.

Understanding Context Windows

A context window is the maximum number of tokens an LLM can process in a single request. This includes both your input (system prompt, user message, documents) and the generated output.

import tiktoken

def count_tokens(text: str, model: str = "gpt-4o") -> int:
    """Count tokens for a given text and model."""
    encoding = tiktoken.encoding_for_model(model)
    return len(encoding.encode(text))

document = open("long_report.txt").read()
token_count = count_tokens(document)
print(f"Document tokens: {token_count:,}")

# Check if it fits in context (leaving room for output)
max_context = 128_000
max_output = 4_096
available_for_input = max_context - max_output

if token_count > available_for_input:
    print(f"Document exceeds available input space by {token_count - available_for_input:,} tokens")
else:
    print(f"Document fits with {available_for_input - token_count:,} tokens to spare")

Even when a document fits, there are reasons to split it. LLMs suffer from the “lost in the middle” problem where information in the center of long contexts gets less attention than content at the beginning or end.

Strategy 1: Fixed-Size Chunking

The simplest approach splits text into chunks of a fixed token count with optional overlap. Overlap ensures that sentences or ideas spanning chunk boundaries are not lost.

from typing import List
import tiktoken

def chunk_by_tokens(
    text: str,
    chunk_size: int = 2000,
    overlap: int = 200,
    model: str = "gpt-4o"
) -> List[str]:
    """Split text into fixed-size token chunks with overlap."""
    encoding = tiktoken.encoding_for_model(model)
    tokens = encoding.encode(text)
    
    chunks = []
    start = 0
    
    while start < len(tokens):
        end = start + chunk_size
        chunk_tokens = tokens[start:end]
        chunk_text = encoding.decode(chunk_tokens)
        chunks.append(chunk_text)
        start = end - overlap  # Step back by overlap amount
    
    return chunks

document = open("long_report.txt").read()
chunks = chunk_by_tokens(document, chunk_size=2000, overlap=200)
print(f"Split into {len(chunks)} chunks")

Fixed-size chunking is fast and predictable, but it has a significant drawback: it can split sentences, paragraphs, or even words mid-stream. This leads to chunks that lack coherent meaning at their boundaries.

Strategy 2: Semantic Chunking

A smarter approach respects the natural structure of text by splitting on paragraph or section boundaries while staying within a token budget.

import re
import tiktoken

def chunk_by_paragraphs(
    text: str,
    max_chunk_tokens: int = 2000,
    model: str = "gpt-4o"
) -> List[str]:
    """Split text into chunks respecting paragraph boundaries."""
    encoding = tiktoken.encoding_for_model(model)
    
    # Split on double newlines (paragraphs) or markdown headers
    segments = re.split(r'\n\n+|(?=^## )', text, flags=re.MULTILINE)
    segments = [s.strip() for s in segments if s.strip()]
    
    chunks = []
    current_chunk = []
    current_tokens = 0
    
    for segment in segments:
        segment_tokens = len(encoding.encode(segment))
        
        if current_tokens + segment_tokens > max_chunk_tokens and current_chunk:
            chunks.append("\n\n".join(current_chunk))
            current_chunk = []
            current_tokens = 0
        
        current_chunk.append(segment)
        current_tokens += segment_tokens
    
    if current_chunk:
        chunks.append("\n\n".join(current_chunk))
    
    return chunks

For markdown or HTML documents, you can split on headers to create topically coherent chunks:

def chunk_by_headers(markdown_text: str, max_tokens: int = 3000) -> List[dict]:
    """Split markdown by headers, keeping sections intact."""
    encoding = tiktoken.encoding_for_model("gpt-4o")
    
    # Split on h2 headers
    sections = re.split(r'(^## .+$)', markdown_text, flags=re.MULTILINE)
    
    chunks = []
    current_header = "Introduction"
    current_body = []
    
    for section in sections:
        if section.startswith("## "):
            if current_body:
                body_text = "\n".join(current_body)
                token_count = len(encoding.encode(body_text))
                chunks.append({
                    "header": current_header,
                    "content": body_text,
                    "tokens": token_count
                })
            current_header = section.strip("# ").strip()
            current_body = []
        else:
            current_body.append(section)
    
    # Don't forget the last section
    if current_body:
        body_text = "\n".join(current_body)
        chunks.append({
            "header": current_header,
            "content": body_text,
            "tokens": len(encoding.encode(body_text))
        })
    
    return chunks

Strategy 3: Map-Reduce Processing

When you need to analyze or summarize an entire long document, the map-reduce pattern is one of the most reliable approaches. You process each chunk independently (map), then combine the results (reduce).

from openai import OpenAI

client = OpenAI()

def map_reduce_summarize(
    chunks: List[str],
    final_instruction: str = "Provide a comprehensive summary."
) -> str:
    """Summarize a long document using map-reduce."""
    
    # Map phase: summarize each chunk
    chunk_summaries = []
    for i, chunk in enumerate(chunks):
        response = client.chat.completions.create(
            model="gpt-4o",
            messages=[
                {"role": "system", "content": "Summarize the following text section concisely. Preserve key facts, numbers, and conclusions."},
                {"role": "user", "content": chunk}
            ],
            temperature=0.3
        )
        summary = response.choices[0].message.content
        chunk_summaries.append(f"[Section {i+1}]: {summary}")
        print(f"Mapped chunk {i+1}/{len(chunks)}")
    
    # Reduce phase: combine summaries into final output
    combined = "\n\n".join(chunk_summaries)
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": "You are given summaries of consecutive sections of a long document. Combine them into a single coherent summary."},
            {"role": "user", "content": f"{final_instruction}\n\nSection summaries:\n{combined}"}
        ],
        temperature=0.3
    )
    
    return response.choices[0].message.content

# Usage
document = open("annual_report.txt").read()
chunks = chunk_by_paragraphs(document, max_chunk_tokens=3000)
final_summary = map_reduce_summarize(chunks)
print(final_summary)

For more complex tasks, you can add a hierarchical reduce step where intermediate summaries are themselves summarized if they exceed the context window.

Strategy 4: Sliding Window with Accumulation

This pattern processes chunks sequentially, carrying forward a running summary. Each step sees the accumulated context plus the next chunk, giving the model awareness of everything it has seen so far.

def sliding_window_analysis(
    chunks: List[str],
    question: str
) -> str:
    """Process chunks sequentially with accumulated context."""
    
    accumulated_notes = ""
    
    for i, chunk in enumerate(chunks):
        response = client.chat.completions.create(
            model="gpt-4o",
            messages=[
                {
                    "role": "system",
                    "content": (
                        "You are analyzing a long document chunk by chunk. "
                        "Update the running notes with any relevant information from the current chunk. "
                        "Keep notes concise but complete."
                    )
                },
                {
                    "role": "user",
                    "content": (
                        f"Question to answer: {question}\n\n"
                        f"Running notes so far:\n{accumulated_notes or '(none yet)'}\n\n"
                        f"Current chunk ({i+1}/{len(chunks)}):\n{chunk}"
                    )
                }
            ],
            temperature=0.2
        )
        accumulated_notes = response.choices[0].message.content
        print(f"Processed chunk {i+1}/{len(chunks)}")
    
    # Final synthesis
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": "Provide a final, well-structured answer based on your complete notes from analyzing the entire document."},
            {"role": "user", "content": f"Question: {question}\n\nComplete notes:\n{accumulated_notes}"}
        ],
        temperature=0.3
    )
    
    return response.choices[0].message.content

This pattern works well for question-answering over long documents because it preserves temporal ordering and allows the model to track evolving themes.

Strategy 5: Relevant Chunk Selection with Embeddings

When you do not need to process the entire document, you can select only the most relevant chunks using semantic similarity. This is the core idea behind RAG, but it applies equally to any long-document workflow.

import numpy as np
from openai import OpenAI

client = OpenAI()

def get_embedding(text: str) -> List[float]:
    """Get embedding vector for text."""
    response = client.embeddings.create(
        model="text-embedding-3-small",
        input=text
    )
    return response.data[0].embedding

def cosine_similarity(a: List[float], b: List[float]) -> float:
    """Compute cosine similarity between two vectors."""
    a, b = np.array(a), np.array(b)
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

def select_relevant_chunks(
    chunks: List[str],
    query: str,
    top_k: int = 5
) -> List[str]:
    """Select the most relevant chunks for a query."""
    query_embedding = get_embedding(query)
    
    scored_chunks = []
    for chunk in chunks:
        chunk_embedding = get_embedding(chunk)
        score = cosine_similarity(query_embedding, chunk_embedding)
        scored_chunks.append((score, chunk))
    
    scored_chunks.sort(key=lambda x: x[0], reverse=True)
    
    selected = [chunk for _, chunk in scored_chunks[:top_k]]
    return selected

# Usage
chunks = chunk_by_paragraphs(document, max_chunk_tokens=1000)
relevant = select_relevant_chunks(chunks, "What were the Q3 revenue figures?", top_k=3)

context = "\n\n---\n\n".join(relevant)
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": "Answer the question using only the provided context."},
        {"role": "user", "content": f"Context:\n{context}\n\nQuestion: What were the Q3 revenue figures?"}
    ]
)

Choosing the Right Strategy

The right strategy depends on your task:

Strategy	Best For	Trade-offs
Fixed chunking	Simple pipelines, ingestion	May break mid-sentence
Semantic chunking	Structured documents	Requires document format knowledge
Map-reduce	Full-document summarization	Higher API cost, parallel-friendly
Sliding window	Sequential analysis, QA	Slow (sequential), risk of drift
Embedding selection	Targeted QA, search	Misses context not in top chunks

In practice, you often combine strategies. For example, use semantic chunking to split the document, embedding selection to find relevant chunks, and then a sliding window to synthesize an answer across those selected chunks.

Cost and Latency Optimization

Processing long documents can be expensive. Here are practical ways to reduce cost:

def estimate_cost(
    text: str,
    model: str = "gpt-4o",
    strategy: str = "map_reduce"
) -> dict:
    """Estimate API cost for processing a long document."""
    encoding = tiktoken.encoding_for_model(model)
    total_tokens = len(encoding.encode(text))
    
    # Pricing per 1M tokens (approximate, check current rates)
    pricing = {
        "gpt-4o": {"input": 2.50, "output": 10.00},
        "gpt-4o-mini": {"input": 0.15, "output": 0.60},
    }
    
    rates = pricing.get(model, pricing["gpt-4o"])
    
    if strategy == "map_reduce":
        # Each chunk is processed once, plus one reduce step
        estimated_input = total_tokens * 1.1  # overlap adds ~10%
        estimated_output = total_tokens * 0.3  # summaries are shorter
    elif strategy == "sliding_window":
        # Accumulated context grows, roughly 50% extra input
        estimated_input = total_tokens * 1.5
        estimated_output = total_tokens * 0.5
    else:
        estimated_input = total_tokens
        estimated_output = total_tokens * 0.2
    
    input_cost = (estimated_input / 1_000_000) * rates["input"]
    output_cost = (estimated_output / 1_000_000) * rates["output"]
    
    return {
        "total_tokens": total_tokens,
        "estimated_input_tokens": int(estimated_input),
        "estimated_output_tokens": int(estimated_output),
        "estimated_cost_usd": round(input_cost + output_cost, 4)
    }

A common optimization is to use a cheaper model for the map phase and a more capable model for the reduce phase. The map phase only needs to extract and summarize, while the reduce phase does the harder synthesis work.

Wrapping Up

Context window limits are a fundamental constraint when working with LLMs, but they are manageable with the right strategies. Fixed-size and semantic chunking give you the building blocks. Map-reduce and sliding window patterns let you process documents of any length. Embedding-based selection keeps costs down when you only need specific information. Start with the simplest strategy that meets your needs, measure quality and cost, and add complexity only when the results justify it. As context windows continue to grow, these patterns remain valuable because even a million-token window benefits from focused, well-structured input.