cgroups for Resource Isolation and Limits on Linux

What Are cgroups?

Control groups (cgroups) are a Linux kernel feature that organizes processes into hierarchical groups and applies resource limits to those groups. They answer the question: “How do I prevent a single process or set of processes from consuming all the CPU, memory, or disk I/O on my system?”

cgroups are one of the two foundational technologies behind Linux containers (the other being namespaces). Every Docker container, Kubernetes pod, and systemd service uses cgroups for resource management.

cgroups v1 vs v2

Linux has two versions of cgroups:

• cgroups v1 — Each resource controller (cpu, memory, blkio, etc.) has its own separate hierarchy. This is the legacy system.
• cgroups v2 — A unified hierarchy where all controllers share a single tree. This is the modern standard.

Most current distributions (Ubuntu 22.04+, Fedora 31+, Debian 11+) default to cgroups v2. This guide focuses on v2.

Check which version your system uses:

# If this directory exists and has content, you're on v2
ls /sys/fs/cgroup/cgroup.controllers

# Or check the mount type
mount | grep cgroup
# cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)

The cgroup Filesystem

cgroups are managed through a pseudo-filesystem mounted at /sys/fs/cgroup/. Each subdirectory is a cgroup, and resource limits are configured by writing to files in those directories.

# View the root cgroup's available controllers
cat /sys/fs/cgroup/cgroup.controllers
# cpuset cpu io memory hugetlb pids rdma misc

# View which controllers are enabled for child groups
cat /sys/fs/cgroup/cgroup.subtree_control

# See current processes in the root cgroup
cat /sys/fs/cgroup/cgroup.procs

Creating a cgroup

To create a cgroup, simply create a directory under /sys/fs/cgroup/:

# Create a cgroup called "myapp"
sudo mkdir /sys/fs/cgroup/myapp

# Enable controllers for this cgroup
sudo sh -c 'echo "+cpu +memory +io +pids" > /sys/fs/cgroup/cgroup.subtree_control'

# Add a process to the cgroup (use its PID)
sudo sh -c 'echo 12345 > /sys/fs/cgroup/myapp/cgroup.procs'

# Or run a new process inside the cgroup
sudo systemd-run --scope --slice=myapp.slice stress --cpu 2

Memory Limits

Memory is often the most critical resource to limit. Without limits, a runaway process can trigger the OOM killer and take down unrelated services.

# Set a hard memory limit of 512MB
sudo sh -c 'echo 536870912 > /sys/fs/cgroup/myapp/memory.max'

# Set a soft limit (memory.high) -- the kernel will throttle, not kill
sudo sh -c 'echo 268435456 > /sys/fs/cgroup/myapp/memory.high'

# Disable swap for this cgroup
sudo sh -c 'echo 0 > /sys/fs/cgroup/myapp/memory.swap.max'

# View current memory usage
cat /sys/fs/cgroup/myapp/memory.current

# View detailed memory statistics
cat /sys/fs/cgroup/myapp/memory.stat

Key memory files:

• memory.max — Hard limit. The OOM killer activates if exceeded.
• memory.high — Soft limit. The kernel aggressively reclaims memory.
• memory.low — Best-effort memory protection (minimum guaranteed).
• memory.current — Current usage in bytes.

CPU Limits

CPU limits control how much processing time a cgroup gets:

# Limit to 50% of one CPU core (50ms out of every 100ms)
sudo sh -c 'echo "50000 100000" > /sys/fs/cgroup/myapp/cpu.max'

# Limit to 200% (2 full cores)
sudo sh -c 'echo "200000 100000" > /sys/fs/cgroup/myapp/cpu.max'

# Set CPU weight (relative priority, default is 100)
sudo sh -c 'echo 50 > /sys/fs/cgroup/myapp/cpu.weight'

# View CPU usage statistics
cat /sys/fs/cgroup/myapp/cpu.stat

The cpu.max file takes two values: quota and period in microseconds. A quota of 50000 with a period of 100000 means the cgroup gets 50ms of CPU time for every 100ms wall-clock time, effectively 50% of one core.

I/O Limits

Block I/O limits prevent processes from saturating your disks:

# Find the major:minor device number for your disk
lsblk -d -o NAME,MAJ:MIN
# sda   8:0

# Limit read bandwidth to 10MB/s on device 8:0
sudo sh -c 'echo "8:0 rbps=10485760" > /sys/fs/cgroup/myapp/io.max'

# Limit write bandwidth to 5MB/s
sudo sh -c 'echo "8:0 wbps=5242880" > /sys/fs/cgroup/myapp/io.max'

# Limit both read and write IOPS
sudo sh -c 'echo "8:0 riops=1000 wiops=500" > /sys/fs/cgroup/myapp/io.max'

# Set I/O weight (relative priority)
sudo sh -c 'echo "8:0 200" > /sys/fs/cgroup/myapp/io.weight'

# View I/O statistics
cat /sys/fs/cgroup/myapp/io.stat

Process (PID) Limits

Prevent fork bombs and runaway process creation:

# Limit to 100 processes
sudo sh -c 'echo 100 > /sys/fs/cgroup/myapp/pids.max'

# View current count
cat /sys/fs/cgroup/myapp/pids.current

Using systemd for cgroup Management

systemd is the most practical way to manage cgroups on modern Linux. Every systemd service automatically gets its own cgroup, and you can set limits directly in the service file:

# /etc/systemd/system/myapp.service
[Unit]
Description=My Application

[Service]
ExecStart=/opt/myapp/run.sh
MemoryMax=512M
MemoryHigh=256M
CPUQuota=150%
IOReadBandwidthMax=/dev/sda 10M
IOWriteBandwidthMax=/dev/sda 5M
TasksMax=100

[Install]
WantedBy=multi-user.target

You can also set limits at runtime without editing unit files:

# Set memory limit on a running service
sudo systemctl set-property myapp.service MemoryMax=512M

# Set CPU quota
sudo systemctl set-property myapp.service CPUQuota=200%

# View the cgroup tree for a service
systemd-cgls /system.slice/myapp.service

# View resource usage for all slices
systemd-cgtop

Monitoring cgroup Resource Usage

# Real-time cgroup resource monitor
systemd-cgtop

# View the full cgroup hierarchy
systemd-cgls

# Check a specific cgroup's resource usage
cat /sys/fs/cgroup/myapp/memory.current
cat /sys/fs/cgroup/myapp/cpu.stat

# Use systemctl to see limits and usage
systemctl status myapp.service
systemctl show myapp.service | grep -E 'Memory|CPU|Tasks'

How Containers Use cgroups

When you run a Docker container with resource limits, Docker creates cgroups behind the scenes:

# This Docker command...
docker run --memory=256m --cpus=1.5 --pids-limit=50 nginx

# ...creates a cgroup with these approximate settings:
# memory.max = 268435456
# cpu.max = 150000 100000
# pids.max = 50

You can inspect the cgroup of a running container:

# Find the container's cgroup path
docker inspect --format '{{.HostConfig.CgroupParent}}' <container_id>

# View its memory limit
cat /sys/fs/cgroup/docker/<container_id>/memory.max

Kubernetes uses the same mechanism. When you set resource limits in a pod spec, the kubelet translates them into cgroup settings on the node.

Practical Example: Isolating a Build Server

Suppose you run CI builds on the same machine as a production web server. You want builds to never consume more than 2 CPU cores, 4GB RAM, and limited disk I/O:

# Create a slice for builds
sudo mkdir -p /sys/fs/cgroup/builds

# Enable controllers
sudo sh -c 'echo "+cpu +memory +io +pids" > /sys/fs/cgroup/cgroup.subtree_control'

# Set limits
sudo sh -c 'echo "200000 100000" > /sys/fs/cgroup/builds/cpu.max'
sudo sh -c 'echo 4294967296 > /sys/fs/cgroup/builds/memory.max'
sudo sh -c 'echo 200 > /sys/fs/cgroup/builds/pids.max'

# Run your build inside the cgroup
sudo systemd-run --scope --slice=builds.slice make -j4

Or better yet, create a systemd slice file:

# /etc/systemd/system/builds.slice
[Slice]
MemoryMax=4G
CPUQuota=200%
TasksMax=200

Summary

cgroups give you precise control over how system resources are distributed among processes. The key controllers are memory (hard/soft limits, swap control), cpu (quotas and weights), io (bandwidth and IOPS limits), and pids (process count limits). In practice, you will most often manage cgroups through systemd unit files rather than writing to the filesystem directly. Understanding cgroups is also essential for understanding how containers achieve resource isolation.