Python collections Module: The Data Structures You Are Missing

Python’s built-in dict, list, set, and tuple cover most needs, but the collections module provides specialized containers that solve common patterns more cleanly and efficiently. This article walks through the five you should know and shows where each one fits.

defaultdict — No More Key Checks

A defaultdict automatically creates missing keys using a factory function. This eliminates the pattern of checking whether a key exists before appending or incrementing.

The Problem

# Grouping without defaultdict
groups = {}
for item in items:
    key = item["category"]
    if key not in groups:
        groups[key] = []
    groups[key].append(item)

The Solution

from collections import defaultdict

groups = defaultdict(list)
for item in items:
    groups[item["category"]].append(item)

The factory function (list, int, set, or any callable) is called whenever a missing key is accessed.

Common Factories

from collections import defaultdict

# Counting (int defaults to 0)
word_count = defaultdict(int)
for word in "the cat sat on the mat".split():
    word_count[word] += 1
# {'the': 2, 'cat': 1, 'sat': 1, 'on': 1, 'mat': 1}

# Collecting unique values
tag_index = defaultdict(set)
articles = [
    {"title": "Python Tips", "tags": ["python", "tips"]},
    {"title": "Async Guide", "tags": ["python", "async"]},
]
for article in articles:
    for tag in article["tags"]:
        tag_index[tag].add(article["title"])
# {'python': {'Python Tips', 'Async Guide'}, 'tips': {'Python Tips'}, ...}

# Nested defaultdicts
tree = lambda: defaultdict(tree)
config = tree()
config["database"]["primary"]["host"] = "localhost"
config["database"]["primary"]["port"] = 5432

Counter — Count, Rank, and Compare

Counter is a specialized dictionary for counting hashable objects. It provides ranking, arithmetic, and set operations out of the box.

from collections import Counter

# Count elements
inventory = Counter(["apple", "banana", "apple", "cherry", "banana", "apple"])
print(inventory)
# Counter({'apple': 3, 'banana': 2, 'cherry': 1})

# Most common elements
print(inventory.most_common(2))
# [('apple', 3), ('banana', 2)]

# Count from a string
letters = Counter("mississippi")
print(letters)
# Counter({'s': 4, 'i': 4, 'p': 2, 'm': 1})

Arithmetic with Counters

from collections import Counter

sales_monday = Counter({"widget": 10, "gadget": 5, "gizmo": 3})
sales_tuesday = Counter({"widget": 7, "gadget": 8, "doohickey": 2})

# Addition combines counts
total = sales_monday + sales_tuesday
print(total)
# Counter({'widget': 17, 'gadget': 13, 'gizmo': 3, 'doohickey': 2})

# Subtraction (drops zero and negative)
diff = sales_monday - sales_tuesday
print(diff)
# Counter({'widget': 3, 'gizmo': 3})

# Intersection (minimum of each)
common = sales_monday & sales_tuesday
print(common)
# Counter({'gadget': 5, 'widget': 7})

Practical Use: Log Analysis

from collections import Counter
from pathlib import Path

def top_error_codes(log_path: str, n: int = 5) -> list:
    codes = Counter()
    for line in Path(log_path).read_text().splitlines():
        if "HTTP" in line:
            status = line.split("HTTP")[1].strip()[:3]
            codes[status] += 1
    return codes.most_common(n)

deque — Fast Double-Ended Queue

deque (pronounced “deck”) supports efficient append and pop from both ends. A regular list takes O(n) time for insert(0, x) and pop(0), while deque does both in O(1).

from collections import deque

# Basic usage
q = deque()
q.append("first")       # Add to right
q.append("second")
q.appendleft("zeroth")  # Add to left
print(q)  # deque(['zeroth', 'first', 'second'])

q.pop()       # Remove from right -> 'second'
q.popleft()   # Remove from left -> 'zeroth'

Bounded deque as a Sliding Window

Set maxlen to create a fixed-size buffer that automatically drops old items:

from collections import deque

# Keep last 5 measurements
recent = deque(maxlen=5)
for value in [10, 20, 30, 40, 50, 60, 70]:
    recent.append(value)
print(recent)  # deque([30, 40, 50, 60, 70], maxlen=5)

# Moving average
def moving_average(data: list[float], window: int) -> list[float]:
    buf = deque(maxlen=window)
    averages = []
    for value in data:
        buf.append(value)
        if len(buf) == window:
            averages.append(sum(buf) / window)
    return averages

print(moving_average([1, 2, 3, 4, 5, 6], window=3))
# [2.0, 3.0, 4.0, 5.0]

Rotation

from collections import deque

d = deque([1, 2, 3, 4, 5])
d.rotate(2)   # Rotate right
print(d)  # deque([4, 5, 1, 2, 3])

d.rotate(-2)  # Rotate left
print(d)  # deque([1, 2, 3, 4, 5])

namedtuple — Lightweight Immutable Objects

namedtuple creates tuple subclasses with named fields. They use no more memory than regular tuples but are self-documenting.

from collections import namedtuple

Point = namedtuple("Point", ["x", "y"])
p = Point(3, 4)
print(p.x, p.y)  # 3 4
print(p[0])       # 3 (still indexable like a tuple)

# Unpacking works
x, y = p
distance = (x ** 2 + y ** 2) ** 0.5
print(distance)  # 5.0

Adding Defaults and Methods

from collections import namedtuple

Color = namedtuple("Color", ["r", "g", "b", "alpha"], defaults=[255])
red = Color(255, 0, 0)
print(red)  # Color(r=255, g=0, b=0, alpha=255)

# Creating from a dictionary
data = {"r": 100, "g": 150, "b": 200, "alpha": 128}
c = Color(**data)

# _replace creates a modified copy (immutable, remember)
faded = red._replace(alpha=50)
print(faded)  # Color(r=255, g=0, b=0, alpha=50)

namedtuple vs dataclass

Use namedtuple when you want immutability, tuple compatibility, and minimal memory. Use dataclass when you need mutability, methods, or more complex behavior.

ChainMap — Layered Lookups

ChainMap groups multiple dictionaries into a single view. Lookups search each dictionary in order. This is perfect for layered configuration.

from collections import ChainMap

defaults = {"theme": "light", "language": "en", "page_size": 25}
user_prefs = {"theme": "dark"}
cli_args = {"page_size": 50}

config = ChainMap(cli_args, user_prefs, defaults)
print(config["theme"])      # "dark" (from user_prefs)
print(config["page_size"])  # 50 (from cli_args)
print(config["language"])   # "en" (from defaults)

Writes only affect the first mapping, so defaults remain untouched:

config["language"] = "fr"
print(cli_args)  # {"page_size": 50, "language": "fr"}
print(defaults)  # {"theme": "light", "language": "en", "page_size": 25}

Key Takeaways

The collections module gives you purpose-built containers for patterns that are awkward with basic types. Use defaultdict to avoid key-existence checks, Counter for counting and ranking, deque for queues and sliding windows, namedtuple for lightweight immutable records, and ChainMap for layered dictionary lookups. Reaching for these tools instead of writing custom logic makes your code shorter, faster, and easier to read.