Python Generators Deep Dive

A generator is a function that produces a sequence of values lazily, one at a time, instead of building the entire sequence in memory. Every time it yields a value, it suspends. When you ask for the next value, it resumes from exactly where it left off. This makes generators ideal for processing large datasets, building pipelines, and any situation where you do not need all the data at once.

Generator basics

A generator function contains at least one yield expression. Calling it does not execute the body; it returns a generator object.

def count_up_to(n):
    print("Starting")
    i = 1
    while i <= n:
        print(f"About to yield {i}")
        yield i
        print(f"Resumed after yielding {i}")
        i += 1
    print("Done")

gen = count_up_to(3)
print(type(gen))  # <class 'generator'>

# Nothing printed yet. The function body hasn't executed.

print(next(gen))
# Starting
# About to yield 1
# 1

print(next(gen))
# Resumed after yielding 1
# About to yield 2
# 2

print(next(gen))
# Resumed after yielding 2
# About to yield 3
# 3

print(next(gen))
# Resumed after yielding 3
# Done
# Raises StopIteration

Each next() call runs the function body until the next yield, returns the yielded value, and suspends. When the function returns (falls off the end or hits return), Python raises StopIteration.

Generator expressions

Generator expressions look like list comprehensions but use parentheses.

# List comprehension: builds the entire list in memory
squares_list = [x**2 for x in range(1_000_000)]

# Generator expression: produces values one at a time
squares_gen = (x**2 for x in range(1_000_000))

# Memory difference is dramatic
import sys
print(sys.getsizeof(squares_list))  # ~8 MB
print(sys.getsizeof(squares_gen))   # 200 bytes (just the generator object)

Generator expressions work anywhere an iterable is expected.

# Sum of squares without building the list
total = sum(x**2 for x in range(1_000_000))

# Find the first match
first_even = next(x for x in data if x % 2 == 0)

# Join strings lazily
result = ",".join(str(x) for x in range(100))

yield from

yield from delegates to another iterable or generator. It passes through all values, sends, and throws.

def flatten(nested):
    for item in nested:
        if isinstance(item, (list, tuple)):
            yield from flatten(item)
        else:
            yield item

data = [1, [2, 3], [4, [5, 6]], 7]
print(list(flatten(data)))
# [1, 2, 3, 4, 5, 6, 7]

Without yield from, you would need an explicit loop:

def flatten_verbose(nested):
    for item in nested:
        if isinstance(item, (list, tuple)):
            for sub_item in flatten_verbose(item):
                yield sub_item
        else:
            yield item

yield from also handles the send() and throw() protocols correctly, which the manual loop does not.

send(): two-way communication

Generators are not one-way. The send() method sends a value into the generator, which becomes the result of the yield expression.

def running_average():
    total = 0
    count = 0
    average = None
    
    while True:
        value = yield average
        if value is None:
            break
        total += value
        count += 1
        average = total / count

avg = running_average()
next(avg)  # Prime the generator (advance to first yield)

print(avg.send(10))   # 10.0
print(avg.send(20))   # 15.0
print(avg.send(30))   # 20.0
print(avg.send(15))   # 18.75

The first next() call (or send(None)) is required to advance the generator to the first yield. After that, each send(value) resumes the generator with value as the result of the yield expression.

Practical example: stateful parser

def csv_parser():
    """Parse CSV data sent line by line."""
    headers = None
    
    while True:
        line = yield
        if line is None:
            break
        
        parts = [p.strip() for p in line.split(",")]
        
        if headers is None:
            headers = parts
            continue
        
        row = dict(zip(headers, parts))
        print(f"Parsed row: {row}")

parser = csv_parser()
next(parser)  # Prime

parser.send("name, age, city")
parser.send("Alice, 30, NYC")
# Parsed row: {'name': 'Alice', 'age': '30', 'city': 'NYC'}
parser.send("Bob, 25, London")
# Parsed row: {'name': 'Bob', 'age': '25', 'city': 'London'}

throw() and close()

throw() raises an exception inside the generator at the point where it is suspended. close() raises GeneratorExit.

def resilient_reader(file_paths):
    for path in file_paths:
        try:
            with open(path) as f:
                for line in f:
                    yield line.strip()
        except GeneratorExit:
            print(f"Generator closed while reading {path}")
            return
        except Exception as e:
            print(f"Error reading {path}: {e}")
            continue

reader = resilient_reader(["a.txt", "b.txt", "c.txt"])
for i, line in enumerate(reader):
    if i >= 100:
        reader.close()  # Raises GeneratorExit inside the generator
        break
    process(line)

Using throw() to signal errors from outside:

def data_processor():
    while True:
        try:
            data = yield
            result = process(data)
            print(f"Processed: {result}")
        except ValueError as e:
            print(f"Skipping bad data: {e}")

proc = data_processor()
next(proc)

proc.send("good data")     # Processed: ...
proc.throw(ValueError, "corrupt record")  # Skipping bad data: corrupt record
proc.send("more good data")  # Processed: ...

Generator pipelines

Generators compose beautifully into pipelines where each stage processes one item at a time. The entire pipeline uses constant memory regardless of data size.

import os

def read_lines(path):
    """Stage 1: Read lines from a file."""
    with open(path) as f:
        for line in f:
            yield line.strip()

def filter_nonempty(lines):
    """Stage 2: Skip blank lines."""
    for line in lines:
        if line:
            yield line

def parse_log_entry(lines):
    """Stage 3: Parse each line into a structured dict."""
    for line in lines:
        parts = line.split(" | ")
        if len(parts) >= 3:
            yield {
                "timestamp": parts[0],
                "level": parts[1],
                "message": parts[2],
            }

def filter_errors(entries):
    """Stage 4: Keep only error entries."""
    for entry in entries:
        if entry["level"] == "ERROR":
            yield entry

# Pipeline: compose the stages
def get_errors(log_path):
    lines = read_lines(log_path)
    lines = filter_nonempty(lines)
    entries = parse_log_entry(lines)
    errors = filter_errors(entries)
    return errors

# Process a 10 GB log file using only a few KB of memory
for error in get_errors("/var/log/app.log"):
    alert(error)

Multi-file pipeline

def read_all_logs(directory):
    """Read from all log files in a directory."""
    for filename in sorted(os.listdir(directory)):
        if filename.endswith(".log"):
            path = os.path.join(directory, filename)
            yield from read_lines(path)

def batch(iterable, size):
    """Group items into fixed-size batches."""
    batch = []
    for item in iterable:
        batch.append(item)
        if len(batch) == size:
            yield batch
            batch = []
    if batch:
        yield batch

# Process all logs in batches of 100
errors = get_errors_from_dir("/var/log/")
for error_batch in batch(errors, 100):
    bulk_insert(error_batch)

Memory efficiency

Generators versus lists — the numbers speak for themselves.

import sys
import tracemalloc

def process_with_list(n):
    data = [x ** 2 for x in range(n)]
    result = sum(x for x in data if x % 3 == 0)
    return result

def process_with_generator(n):
    data = (x ** 2 for x in range(n))
    result = sum(x for x in data if x % 3 == 0)
    return result

# Measure memory usage
tracemalloc.start()

process_with_list(1_000_000)
list_peak = tracemalloc.get_traced_memory()[1]

tracemalloc.reset()

process_with_generator(1_000_000)
gen_peak = tracemalloc.get_traced_memory()[1]

tracemalloc.stop()

print(f"List peak memory: {list_peak / 1024 / 1024:.1f} MB")
# List peak memory: ~35 MB
print(f"Generator peak memory: {gen_peak / 1024 / 1024:.3f} MB")
# Generator peak memory: ~0.001 MB

itertools: the generator toolkit

The itertools module provides composable building blocks that work with generators.

import itertools

# Chain multiple iterables
all_data = itertools.chain(file1_lines, file2_lines, file3_lines)

# Take the first N items
first_100 = itertools.islice(large_generator, 100)

# Group consecutive items
for key, group in itertools.groupby(sorted_data, key=lambda x: x["category"]):
    items = list(group)
    print(f"{key}: {len(items)} items")

# Infinite counter
counter = itertools.count(start=1)
for i, item in zip(counter, data):
    print(f"Item {i}: {item}")

# Sliding window (Python 3.12+)
from itertools import pairwise
for prev, curr in pairwise(sensor_readings):
    delta = curr - prev
    if abs(delta) > threshold:
        alert(f"Spike detected: {delta}")

Common pitfalls

Generators are single-use. Once exhausted, they produce no more values.

gen = (x for x in range(5))
print(list(gen))  # [0, 1, 2, 3, 4]
print(list(gen))  # [] -- exhausted

Generators are lazy. Side effects do not happen until you iterate.

# This does NOT execute any file writes
results = (write_to_file(item) for item in data)

# You must consume the generator
list(results)  # Now the writes happen
# Or better: use a regular loop if you need side effects
for item in data:
    write_to_file(item)

Do not hold references to generator intermediates if you want memory savings.

# Bad: the list reference keeps all data in memory
lines = list(read_lines("huge.txt"))
errors = (line for line in lines if "ERROR" in line)

# Good: chain generators without materializing
lines = read_lines("huge.txt")
errors = (line for line in lines if "ERROR" in line)

Generators are one of Python’s most elegant features. They let you write code that reads like it processes everything at once but actually handles data one piece at a time. Master them and you can process datasets larger than your RAM without breaking a sweat.