Linux awk and sed: Text Processing Power Tools

Why awk and sed Still Matter

Despite the availability of Python, Perl, and other scripting languages, awk and sed remain essential tools for every Linux user. They are installed on virtually every Unix-like system, they work directly in shell pipelines, and for many text-processing tasks they are faster to write and execute than a full script.

sed is a stream editor. It reads input line by line, applies transformations, and writes the result. It excels at find-and-replace operations.

awk is a pattern-scanning and processing language. It splits each line into fields and lets you apply logic based on patterns. It excels at extracting and reformatting columnar data.

sed Fundamentals

Basic Substitution

The most common sed operation is substitution:

# Replace first occurrence of 'foo' with 'bar' on each line
echo "foo foo foo" | sed 's/foo/bar/'
# Output: bar foo foo

# Replace ALL occurrences (global flag)
echo "foo foo foo" | sed 's/foo/bar/g'
# Output: bar bar bar

The general syntax is s/pattern/replacement/flags. Common flags:

• g - Replace all occurrences on each line, not just the first.
• i or I - Case-insensitive matching.
• p - Print the modified line (used with -n).
• A number like 2 - Replace only the Nth occurrence.

Delimiters

You do not have to use / as the delimiter. When your pattern contains slashes, use a different character:

# Replace a file path
sed 's|/usr/local/bin|/opt/bin|g' config.txt

# Any character works
sed 's#old#new#g' file.txt

In-Place Editing

Edit files directly with the -i flag:

# Edit the file in place (GNU sed)
sed -i 's/DEBUG=true/DEBUG=false/g' config.env

# Create a backup before editing
sed -i.bak 's/DEBUG=true/DEBUG=false/g' config.env
# Original saved as config.env.bak

On macOS (BSD sed), -i requires an argument:

# macOS: empty string means no backup
sed -i '' 's/old/new/g' file.txt

Line Selection

Apply operations to specific lines:

# Only line 5
sed '5s/old/new/' file.txt

# Lines 10 through 20
sed '10,20s/old/new/g' file.txt

# Lines matching a pattern
sed '/ERROR/s/old/new/g' logfile.txt

# From a pattern to end of file
sed '/START/,$ s/old/new/g' file.txt

Deleting Lines

# Delete line 3
sed '3d' file.txt

# Delete lines 5 through 10
sed '5,10d' file.txt

# Delete blank lines
sed '/^$/d' file.txt

# Delete lines containing 'DEBUG'
sed '/DEBUG/d' logfile.txt

# Delete lines NOT containing 'ERROR'
sed '/ERROR/!d' logfile.txt

Inserting and Appending Text

# Insert a line before line 3
sed '3i\New line inserted above' file.txt

# Append a line after line 3
sed '3a\New line inserted below' file.txt

# Add a header line at the beginning
sed '1i\# Configuration File' config.txt

Multi-Command sed

Chain multiple operations with -e or semicolons:

# Multiple substitutions
sed -e 's/foo/bar/g' -e 's/baz/qux/g' file.txt

# Using semicolons
sed 's/foo/bar/g; s/baz/qux/g' file.txt

Capture Groups

Use parentheses and backreferences to reuse matched text:

# Swap first and last name
echo "John Smith" | sed 's/\(\w*\) \(\w*\)/\2 \1/'
# Output: Smith John

# Add quotes around words
echo "hello world" | sed 's/\(\w\+\)/"\1"/g'
# Output: "hello" "world"

# Extract version number and reformat
echo "version-2.3.1" | sed 's/version-\([0-9]*\)\.\([0-9]*\)\.\([0-9]*\)/v\1.\2.\3/'
# Output: v2.3.1

awk Fundamentals

How awk Processes Input

awk reads input line by line (each line is a “record”). Each record is split into fields by a delimiter (whitespace by default). Fields are accessed as $1, $2, etc. $0 is the entire line.

# Print the second field of each line
echo -e "Alice 30 Engineer\nBob 25 Designer" | awk '{print $2}'
# Output:
# 30
# 25

# Print multiple fields
echo -e "Alice 30 Engineer\nBob 25 Designer" | awk '{print $1, $3}'
# Output:
# Alice Engineer
# Bob Designer

Custom Field Separator

Use -F to set the delimiter:

# Parse /etc/passwd (colon-separated)
awk -F: '{print $1, $3}' /etc/passwd

# Parse CSV data
awk -F, '{print $1, $3}' data.csv

# Multiple delimiters
awk -F'[,;:]' '{print $1, $2}' mixed.txt

Pattern Matching

awk applies actions only to lines matching a pattern:

# Lines containing 'ERROR'
awk '/ERROR/ {print $0}' logfile.txt

# Lines where field 3 is greater than 100
awk '$3 > 100 {print $1, $3}' data.txt

# Lines where field 1 equals a specific string
awk '$1 == "Alice" {print $0}' people.txt

# Negate a pattern
awk '!/DEBUG/' logfile.txt

Built-In Variables

Variable	Meaning
NR	Current record (line) number
NF	Number of fields in the current record
FS	Field separator (same as -F)
OFS	Output field separator
RS	Record separator (default: newline)
ORS	Output record separator
FILENAME	Name of the current input file

# Print line numbers
awk '{print NR, $0}' file.txt

# Print the last field on each line
awk '{print $NF}' file.txt

# Print lines with more than 5 fields
awk 'NF > 5' file.txt

# Set output separator
awk -F: 'BEGIN {OFS=","} {print $1, $3, $7}' /etc/passwd

BEGIN and END Blocks

BEGIN runs before processing any input. END runs after all input is processed.

# Calculate average of a column
awk '{sum += $2; count++} END {print "Average:", sum/count}' grades.txt

# Add a CSV header
awk 'BEGIN {print "Name,Age,Role"} {print $1","$2","$3}' people.txt

# Count lines matching a pattern
awk '/ERROR/ {count++} END {print "Errors:", count}' logfile.txt

Formatted Output with printf

# Right-aligned table
awk '{printf "%-15s %5d %s\n", $1, $2, $3}' people.txt

# Output:
# Alice              30 Engineer
# Bob                25 Designer

Conditional Logic

# If-else in awk
awk '{
    if ($3 > 90) grade = "A"
    else if ($3 > 80) grade = "B"
    else if ($3 > 70) grade = "C"
    else grade = "F"
    print $1, grade
}' scores.txt

Associative Arrays

awk has built-in hash maps (associative arrays):

# Count occurrences of each value in column 1
awk '{count[$1]++} END {for (key in count) print key, count[key]}' access.log

# Sum values grouped by category
awk -F, '{total[$1] += $2} END {for (cat in total) print cat, total[cat]}' sales.csv

Real-World One-Liners

Log Analysis

# Top 10 IP addresses in an Apache access log
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -10

# Requests per hour
awk '{print substr($4, 2, 14)}' access.log | sort | uniq -c

# Find all 5xx errors
awk '$9 ~ /^5/ {print $7, $9}' access.log

# Average response time (assuming it is the last field)
awk '{sum += $NF; count++} END {printf "Avg: %.2fms\n", sum/count}' access.log

Data Transformation

# Convert TSV to CSV
sed 's/\t/,/g' data.tsv > data.csv

# Remove trailing whitespace
sed 's/[[:space:]]*$//' file.txt

# Extract email addresses from text
grep -oE '[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}' file.txt

# Convert key=value pairs to JSON-ish format
awk -F= '{printf "  \"%s\": \"%s\",\n", $1, $2}' config.env

System Administration

# Find processes using more than 1GB memory
ps aux | awk '$6 > 1048576 {print $11, $6/1024 "MB"}'

# Disk usage summary, sorted
df -h | awk 'NR>1 {print $5, $6}' | sort -rn

# List users with bash shell
awk -F: '$7 ~ /bash/ {print $1}' /etc/passwd

# Kill all processes matching a name
ps aux | awk '/zombie_process/ && !/awk/ {print $2}' | xargs kill -9

File Processing

# Remove duplicate lines (preserving order)
awk '!seen[$0]++' file.txt

# Print lines between two markers
sed -n '/BEGIN_SECTION/,/END_SECTION/p' file.txt

# Number non-empty lines
awk 'NF {printf "%4d %s\n", ++n, $0; next} {print}' file.txt

# Merge every two lines
awk 'NR%2==1 {prev=$0; next} {print prev, $0}' file.txt

Combining awk and sed in Pipelines

The real power comes from combining tools:

# Parse a JSON-like log, extract fields, and summarize
cat app.log \
  | sed 's/.*"status":\([0-9]*\).*"duration":\([0-9.]*\).*/\1 \2/' \
  | awk '$1 >= 500 {sum += $2; count++} END {
      printf "5xx errors: %d, Avg duration: %.2fms\n", count, sum/count
    }'

# Reformat git log into a changelog
git log --oneline --since="2026-01-01" \
  | sed 's/^[a-f0-9]* //' \
  | awk '{print "- " $0}'

# Extract and reformat CSV columns
awk -F, '{print $3, $1}' data.csv \
  | sed 's/^ *//; s/ *$//' \
  | sort -u

Performance Tips

For very large files, awk and sed are significantly faster than Python or Ruby because they are compiled C programs optimized for line-by-line processing.

# Process a 10GB log file - awk handles it line by line
# without loading the entire file into memory
awk '/CRITICAL/ {print NR": "$0}' huge.log

# Use LC_ALL=C for faster processing when you do not need locale
LC_ALL=C awk '/pattern/' huge.log

# Quit early if you only need the first match
sed -n '/pattern/{p;q}' huge.log

When to Reach for Something Else

awk and sed are best for line-oriented text processing. Consider other tools when:

• You need to parse structured formats like JSON or XML. Use jq for JSON or xmlstarlet for XML.
• You need complex data structures beyond associative arrays. Use Python.
• You need to modify binary files. Use xxd or a hex editor.
• Your “one-liner” has grown beyond 3 or 4 lines. Write a proper script.

Wrapping Up

sed is your tool for search-and-replace, line deletion, and in-place file editing. awk is your tool for field extraction, pattern matching, and columnar data processing. Together they handle the vast majority of text-processing tasks you encounter on Linux. Start with simple one-liners, build up to multi-step pipelines, and keep a cheat sheet handy until the syntax becomes second nature.