sed for Stream Editing and Text Transformation

What Is sed?

sed (stream editor) is a non-interactive text editor that processes input line by line, applies transformations, and writes the result to standard output. Unlike a text editor where you open a file and make manual changes, sed operates as a filter in a pipeline or transforms files in place from the command line.

The basic syntax is:

sed 'command' filename

sed reads each line into a buffer called the “pattern space,” applies the command, prints the result, and moves to the next line.

Substitution: The Most Common Operation

The s (substitute) command is what most people use sed for. Its syntax is s/pattern/replacement/flags:

# Replace first occurrence of 'foo' with 'bar' on each line
sed 's/foo/bar/' file.txt

# Replace ALL occurrences on each line (global flag)
sed 's/foo/bar/g' file.txt

# Case-insensitive replacement (GNU sed)
sed 's/error/WARNING/gi' log.txt

# Replace only the 2nd occurrence on each line
sed 's/the/THE/2' file.txt

You can use any delimiter, not just /. This is helpful when working with file paths:

# These are all equivalent
sed 's/\/usr\/local\/bin/\/opt\/bin/g' config.txt
sed 's|/usr/local/bin|/opt/bin|g' config.txt
sed 's#/usr/local/bin#/opt/bin#g' config.txt

In-Place File Editing

By default, sed writes to stdout and does not modify the original file. Use the -i flag for in-place editing:

# Edit file in place (GNU sed)
sed -i 's/old/new/g' file.txt

# Edit in place with a backup (creates file.txt.bak)
sed -i.bak 's/old/new/g' file.txt

# macOS/BSD sed requires an argument to -i (use empty string for no backup)
sed -i '' 's/old/new/g' file.txt

Always create a backup when using -i on important files until you are confident in your command.

Addressing: Targeting Specific Lines

You can restrict sed commands to specific lines using line numbers, ranges, or patterns:

# Only on line 5
sed '5s/foo/bar/' file.txt

# Lines 10 through 20
sed '10,20s/foo/bar/g' file.txt

# From line 5 to the end of the file
sed '5,$s/foo/bar/g' file.txt

# Only on lines matching a pattern
sed '/ERROR/s/foo/bar/g' log.txt

# Between two patterns (inclusive)
sed '/START/,/END/s/foo/bar/g' file.txt

# Every 3rd line starting from line 1
sed '1~3s/^/>>> /' file.txt

Deleting Lines

The d command deletes lines from the output:

# Delete line 3
sed '3d' file.txt

# Delete lines 5 through 10
sed '5,10d' file.txt

# Delete blank lines
sed '/^$/d' file.txt

# Delete lines matching a pattern
sed '/DEBUG/d' app.log

# Delete lines NOT matching a pattern (! negates)
sed '/IMPORTANT/!d' notes.txt

# Delete comment lines (lines starting with #)
sed '/^#/d' config.txt

# Delete both comment lines and blank lines
sed '/^#/d; /^$/d' config.txt

Inserting, Appending, and Changing Lines

# Insert a line BEFORE line 3
sed '3i\This line is inserted before line 3' file.txt

# Append a line AFTER line 3
sed '3a\This line is appended after line 3' file.txt

# Replace (change) line 3 entirely
sed '3c\This replaces line 3 completely' file.txt

# Insert before every line matching a pattern
sed '/\[Section\]/i\# --- Section Boundary ---' config.ini

# Append after the last line
sed '$a\# End of file' file.txt

Printing and Suppressing Output

By default, sed prints every line. The -n flag suppresses automatic printing, and the p command explicitly prints:

# Print only lines matching a pattern (like grep)
sed -n '/error/p' log.txt

# Print lines 10 to 20
sed -n '10,20p' file.txt

# Print the first line (like head -1)
sed -n '1p' file.txt

# Print the last line (like tail -1)
sed -n '$p' file.txt

# Print lines matching a pattern with line numbers
sed -n '/TODO/=' file.txt

Capture Groups and Back-References

sed supports capture groups with \( and \) and back-references with \1, \2, etc.:

# Swap first and last name
echo "John Smith" | sed 's/\(.*\) \(.*\)/\2, \1/'
# Output: Smith, John

# Add quotes around each word
echo "hello world" | sed 's/\([a-z]*\)/"\1"/g'
# Output: "hello" "world"

# Extract a version number
echo "version: 3.14.2" | sed 's/.*: \([0-9.]*\)/\1/'
# Output: 3.14.2

# Duplicate a matched group
echo "abc" | sed 's/\(b\)/[\1\1]/'
# Output: a[bb]c

With extended regex (-E or -r), you can use ( and ) without escaping:

# Reformat a date from MM/DD/YYYY to YYYY-MM-DD
echo "07/02/2026" | sed -E 's|([0-9]{2})/([0-9]{2})/([0-9]{4})|\3-\1-\2|'
# Output: 2026-07-02

Multiple Commands

You can chain multiple sed commands with -e or semicolons:

# Multiple -e flags
sed -e 's/foo/bar/g' -e 's/baz/qux/g' file.txt

# Semicolons
sed 's/foo/bar/g; s/baz/qux/g' file.txt

# Using a sed script file
cat > transform.sed << 'EOF'
s/error/ERROR/gi
s/warn/WARNING/gi
/DEBUG/d
EOF
sed -f transform.sed app.log

Real-World Examples

Configuration File Management

# Update a config value
sed -i 's/^max_connections = .*/max_connections = 200/' postgresql.conf

# Comment out a line
sed -i 's/^listen_addresses/# listen_addresses/' postgresql.conf

# Uncomment a line
sed -i 's/^# *listen_addresses/listen_addresses/' postgresql.conf

# Add a config line after a specific line
sed -i '/^\[mysqld\]/a\max_connections = 500' my.cnf

Log Processing

# Remove ANSI color codes from log output
sed 's/\x1b\[[0-9;]*m//g' colored.log

# Extract timestamps from log lines
sed -n 's/^\[\([0-9-]* [0-9:]*\)\].*/\1/p' app.log

# Anonymize IP addresses
sed -E 's/[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}/xxx.xxx.xxx.xxx/g' access.log

Code Refactoring

# Rename a function across files
find . -name "*.py" -exec sed -i 's/old_function_name/new_function_name/g' {} +

# Add a copyright header to all source files
find . -name "*.js" -exec sed -i '1i\// Copyright 2026 MyCompany. All rights reserved.' {} +

# Remove trailing whitespace
sed -i 's/[[:space:]]*$//' file.txt

# Convert Windows line endings (CRLF) to Unix (LF)
sed -i 's/\r$//' file.txt

Data Transformation

# Convert a comma-separated list to newline-separated
echo "a,b,c,d" | sed 's/,/\n/g'

# Wrap each line in XML tags
sed 's/.*/<item>&<\/item>/' items.txt

# Number non-blank lines
sed '/./=' file.txt | sed 'N; s/\n/\t/'

# Join every two lines
sed 'N;s/\n/ /' file.txt

The Hold Space

sed has a secondary buffer called the “hold space” that enables multi-line operations. While advanced, it is worth knowing:

# Reverse the order of lines (like tac)
sed -n '1!G;h;$p' file.txt

# Print duplicate lines only
sed -n '$!N; /^\(.*\)\n\1$/p' sorted.txt

# Double-space a file (add blank line after each line)
sed 'G' file.txt

The hold space commands are h (copy pattern to hold), H (append to hold), g (copy hold to pattern), G (append hold to pattern), and x (exchange).

sed vs Other Tools

Task	Best Tool
Simple find-and-replace	sed
Column-based processing	awk
Complex pattern matching	grep / perl
In-place multi-file edits	sed with find
JSON/structured data	jq / python

Summary

sed is the go-to tool for automated text transformations on Linux. The most important concepts are the s/pattern/replacement/flags substitution command, address ranges for targeting specific lines, in-place editing with -i, capture groups for reordering and extracting text, and combining multiple commands. Once you master these fundamentals, sed becomes an incredibly efficient tool for everything from config management to log processing to code refactoring.