Python Profiling: Find and Fix Performance Bottlenecks

Why Profiling Matters

Guessing where your code is slow almost never works. Developers routinely optimize the wrong function, shave milliseconds off code that runs once, and ignore the loop that runs a million times. Profiling removes the guesswork by measuring exactly where time and memory are spent.

Python provides several profiling tools, from built-in modules like cProfile and timeit to third-party libraries like line_profiler and memory_profiler. This guide walks through each one with practical examples.

Quick Timing with timeit

For micro-benchmarks, timeit is the simplest option. It runs a snippet many times and reports the average duration.

import timeit

# Compare list comprehension vs map
list_comp = timeit.timeit(
    '[x ** 2 for x in range(1000)]',
    number=10_000
)

map_version = timeit.timeit(
    'list(map(lambda x: x ** 2, range(1000)))',
    number=10_000
)

print(f"List comprehension: {list_comp:.4f}s")
print(f"map + lambda:       {map_version:.4f}s")

You can also use timeit from the command line:

# From your terminal
# python -m timeit "sum(range(10_000))"

The key rule with timeit is to only use it for comparing small, isolated snippets. For profiling real applications, you need something more powerful.

Function-Level Profiling with cProfile

The cProfile module is built into Python and measures how many times each function is called and how long each call takes.

Basic Usage

import cProfile

def fibonacci(n):
    if n < 2:
        return n
    return fibonacci(n - 1) + fibonacci(n - 2)

def process_data():
    results = []
    for i in range(30):
        results.append(fibonacci(i))
    return results

# Profile the function
cProfile.run('process_data()')

This prints a table showing each function, the number of calls, total time, and time per call. The output looks like:

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
  2692537    0.893    0.000    0.893    0.000 script.py:3(fibonacci)
       30    0.000    0.000    0.893    0.030 script.py:3(fibonacci)
        1    0.000    0.000    0.893    0.893 script.py:8(process_data)

Saving and Sorting Results

For larger programs, save the profile data to a file and analyze it separately:

import cProfile
import pstats

def run_application():
    # Your application logic here
    data = [i ** 2 for i in range(100_000)]
    sorted_data = sorted(data, reverse=True)
    filtered = [x for x in sorted_data if x % 7 == 0]
    return filtered

# Save profile to file
cProfile.run('run_application()', 'profile_output.prof')

# Load and analyze
stats = pstats.Stats('profile_output.prof')
stats.sort_stats('cumulative')
stats.print_stats(20)  # Top 20 functions

You can sort by different columns:

# Sort by total time spent inside the function (excluding sub-calls)
stats.sort_stats('tottime')
stats.print_stats(10)

# Sort by number of calls
stats.sort_stats('calls')
stats.print_stats(10)

# Show callers of a specific function
stats.print_callers('fibonacci')

# Show what a specific function calls
stats.print_callees('process_data')

Profiling as a Context Manager

For profiling specific sections of code rather than entire scripts:

import cProfile
import pstats
from io import StringIO

def profile_section():
    profiler = cProfile.Profile()
    profiler.enable()

    # Code to profile
    result = sum(x ** 2 for x in range(100_000))

    profiler.disable()

    stream = StringIO()
    stats = pstats.Stats(profiler, stream=stream)
    stats.sort_stats('cumulative')
    stats.print_stats(10)
    print(stream.getvalue())
    return result

profile_section()

Visualizing Profiles with snakeviz

Raw cProfile output can be hard to read. snakeviz provides an interactive browser-based visualization.

# Install: pip install snakeviz

# First, generate a profile file
# python -m cProfile -o output.prof your_script.py

# Then visualize it
# snakeviz output.prof

This opens a browser with an interactive sunburst chart where each arc represents a function. Wider arcs mean more time spent. You can click into any function to zoom in on its callees.

Programmatic Profile Generation

import cProfile

def expensive_computation():
    """Simulate a realistic workload."""
    data = {}
    for i in range(50_000):
        key = f"item_{i}"
        data[key] = i ** 2

    # Simulate searching
    results = []
    for target in range(0, 50_000, 100):
        key = f"item_{target}"
        if key in data:
            results.append(data[key])

    return results

# Generate the profile file
profiler = cProfile.Profile()
profiler.enable()
expensive_computation()
profiler.disable()
profiler.dump_stats('computation.prof')
# Then run: snakeviz computation.prof

Line-by-Line Profiling with line_profiler

cProfile tells you which function is slow, but line_profiler tells you which line inside that function is the bottleneck.

# Install: pip install line_profiler

Using the @profile Decorator

Create a script and decorate the functions you want to profile:

# slow_script.py
import hashlib

@profile  # This decorator is added by line_profiler at runtime
def process_records(records):
    results = []
    for record in records:
        # Which of these lines is slowest?
        cleaned = record.strip().lower()
        hashed = hashlib.sha256(cleaned.encode()).hexdigest()
        validated = len(cleaned) > 3 and cleaned.isalnum()
        if validated:
            results.append({'original': record, 'hash': hashed})
    return results

records = [f"  Record_{i}  " for i in range(100_000)]
process_records(records)

Run it with:

# kernprof -l -v slow_script.py

The output shows time spent on each line:

Line #   Hits     Time  Per Hit   % Time  Line Contents
     6  100000   50000.0      0.5     5.2  cleaned = record.strip().lower()
     7  100000  750000.0      7.5    78.1  hashed = hashlib.sha256(...)
     8  100000  100000.0      1.0    10.4  validated = len(cleaned) > ...
     9   90000   60000.0      0.7     6.3  results.append({...})

Now you know that the sha256 hashing takes 78% of the time, and you can decide whether caching or a faster hash would help.

Memory Profiling with memory_profiler

When your program uses too much RAM, memory_profiler shows exactly where allocations happen.

# Install: pip install memory_profiler

Line-by-Line Memory Usage

# memory_script.py
from memory_profiler import profile

@profile
def create_large_structures():
    # Each step's memory impact is measured
    small_list = list(range(10_000))          # ~0.08 MB
    medium_list = list(range(1_000_000))      # ~8 MB
    large_dict = {i: str(i) for i in range(500_000)}  # ~40 MB

    # Deleting frees memory
    del medium_list  # -8 MB

    return large_dict

create_large_structures()

Run with:

# python memory_script.py

Tracking Memory Over Time

from memory_profiler import memory_usage
import time

def leaky_function():
    """Simulate a function that accumulates data."""
    cache = []
    for i in range(100):
        cache.append(' ' * 1_000_000)  # 1 MB string each iteration
        time.sleep(0.01)
    return len(cache)

# Sample memory every 0.1 seconds
mem_usage = memory_usage(leaky_function, interval=0.1)
print(f"Peak memory: {max(mem_usage):.1f} MB")
print(f"Memory growth: {mem_usage[-1] - mem_usage[0]:.1f} MB")

Profiling with the Built-in trace Module

For understanding code flow without installing anything extra, trace can count how many times each line executes:

import trace
import sys

def find_primes(limit):
    primes = []
    for num in range(2, limit):
        is_prime = True
        for divisor in range(2, int(num ** 0.5) + 1):
            if num % divisor == 0:
                is_prime = False
                break
        if is_prime:
            primes.append(num)
    return primes

tracer = trace.Trace(count=True, trace=False)
tracer.runfunc(find_primes, 1000)

# Print which lines were hit most
results = tracer.results()
results.write_results(show_missing=True, coverdir='./trace_output')

A Practical Profiling Workflow

Here is a realistic example that ties everything together. Suppose you have a data pipeline that is too slow:

import cProfile
import pstats
import csv
from io import StringIO
from collections import Counter

def read_data(filename):
    """Read CSV data into a list of dicts."""
    with open(filename) as f:
        return list(csv.DictReader(f))

def clean_record(record):
    """Normalize a single record."""
    return {
        key: value.strip().lower()
        for key, value in record.items()
    }

def find_duplicates(records, key_field):
    """Find duplicate values in a specific field."""
    seen = Counter()
    duplicates = []
    for record in records:
        value = record.get(key_field, '')
        seen[value] += 1
        if seen[value] > 1:
            duplicates.append(record)
    return duplicates

def aggregate_by_field(records, field):
    """Group records by a field and count."""
    groups = {}
    for record in records:
        key = record.get(field, 'unknown')
        if key not in groups:
            groups[key] = []
        groups[key].append(record)
    return {k: len(v) for k, v in groups.items()}

def pipeline(filename):
    raw = read_data(filename)
    cleaned = [clean_record(r) for r in raw]
    dupes = find_duplicates(cleaned, 'email')
    summary = aggregate_by_field(cleaned, 'department')
    return summary, dupes

# Profile the pipeline
# cProfile.run("pipeline('large_dataset.csv')", 'pipeline.prof')
# stats = pstats.Stats('pipeline.prof')
# stats.sort_stats('cumulative').print_stats(15)

Step-by-Step Optimization

After profiling, suppose clean_record takes 60% of the time because it is called per-row. You can optimize it:

def clean_records_batch(records):
    """Clean all records at once -- avoids per-record function call overhead."""
    return [
        {key: value.strip().lower() for key, value in record.items()}
        for record in records
    ]

If find_duplicates is slow because of the Counter, you might switch to a set-based approach:

def find_duplicates_fast(records, key_field):
    """Faster duplicate detection using sets."""
    seen = set()
    duplicates = []
    for record in records:
        value = record.get(key_field, '')
        if value in seen:
            duplicates.append(record)
        else:
            seen.add(value)
    return duplicates

Always re-profile after each change to confirm the improvement.

Common Performance Pitfalls

Here are patterns that profiling frequently reveals:

# 1. String concatenation in loops (O(n^2))
# Bad
result = ""
for item in large_list:
    result += str(item)  # Creates a new string each time

# Good
result = "".join(str(item) for item in large_list)

# 2. Repeated attribute lookups
# Bad
for item in data:
    item.lower().strip().replace(" ", "_")

# Good -- bind method references
lower = str.lower
strip = str.strip
for item in data:
    strip(lower(item)).replace(" ", "_")

# 3. Using list where set is appropriate
# Bad -- O(n) lookup each time
if item in large_list:
    pass

# Good -- O(1) lookup
large_set = set(large_list)
if item in large_set:
    pass

# 4. Global variable lookups (slower than local)
# Bad
MULTIPLIER = 2.5
def compute(values):
    return [v * MULTIPLIER for v in values]

# Good
def compute(values, multiplier=2.5):
    return [v * multiplier for v in values]

Profiling in Production

For production code, you typically cannot attach a full profiler. Instead, use targeted timing:

import time
import functools
import logging

logger = logging.getLogger(__name__)

def timed(func):
    """Decorator that logs execution time."""
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        start = time.perf_counter()
        result = func(*args, **kwargs)
        elapsed = time.perf_counter() - start
        logger.info(f"{func.__name__} took {elapsed:.4f}s")
        return result
    return wrapper

@timed
def fetch_user_data(user_id):
    # Simulate work
    time.sleep(0.1)
    return {"id": user_id, "name": "Alice"}

# For async code
def timed_async(func):
    @functools.wraps(func)
    async def wrapper(*args, **kwargs):
        start = time.perf_counter()
        result = await func(*args, **kwargs)
        elapsed = time.perf_counter() - start
        logger.info(f"{func.__name__} took {elapsed:.4f}s")
        return result
    return wrapper

Wrapping Up

Effective profiling follows a simple loop: measure, identify the bottleneck, optimize that specific spot, and measure again. Start with timeit for quick comparisons, use cProfile to find which functions are slow, drill into specific lines with line_profiler, and track memory with memory_profiler. The tools are straightforward once you build the habit of profiling before optimizing. Never guess where the bottleneck is — let the profiler tell you.