awk for Text Processing: A Practical Guide with Real Examples

What Is awk?

awk is a powerful text processing language built into virtually every Unix and Linux system. It reads input line by line, splits each line into fields, and lets you apply patterns and actions to transform data. Despite being created in 1977, awk remains one of the most useful tools in any Linux user’s toolkit.

The basic syntax is:

awk 'pattern { action }' filename

If you omit the pattern, the action runs on every line. If you omit the action, matching lines are printed.

Field Extraction Basics

By default, awk splits each line on whitespace and assigns fields to variables $1, $2, $3, and so on. $0 represents the entire line.

# Print the first and third columns of a file
awk '{ print $1, $3 }' data.txt

# Print usernames from /etc/passwd (fields separated by colons)
awk -F: '{ print $1 }' /etc/passwd

# Print the last field of each line
awk '{ print $NF }' data.txt

# Print the second-to-last field
awk '{ print $(NF-1) }' data.txt

The -F flag sets the field separator. You can use any character or regex:

# CSV files
awk -F, '{ print $2 }' report.csv

# Tab-separated files
awk -F'\t' '{ print $1, $3 }' data.tsv

# Multiple separators (colon or comma)
awk -F'[,:]' '{ print $1, $2 }' mixed.txt

Built-in Variables

awk provides several built-in variables that are essential to know:

# NR = current line number (Number of Records)
awk '{ print NR, $0 }' file.txt        # add line numbers

# NF = number of fields in current line
awk '{ print NF, $0 }' file.txt        # show field count per line

# FS = field separator (same as -F)
awk 'BEGIN { FS=":" } { print $1 }' /etc/passwd

# OFS = output field separator
awk -F: 'BEGIN { OFS="\t" } { print $1, $3 }' /etc/passwd

# RS = record separator (default is newline)
# ORS = output record separator

Pattern Matching

Patterns control which lines awk processes. You can use regular expressions, comparisons, and special patterns:

# Lines matching a regex
awk '/error/ { print }' /var/log/syslog

# Lines NOT matching a regex
awk '!/debug/ { print }' app.log

# Field-level regex matching
awk '$3 ~ /fail/ { print $1, $3 }' status.log

# Numeric comparisons
awk '$3 > 100 { print $1, $3 }' sales.txt

# String comparisons
awk '$1 == "admin" { print }' users.txt

# Compound conditions
awk '$3 > 50 && $4 == "active" { print $1 }' accounts.txt

# Range patterns (from first match to second match)
awk '/START/,/END/ { print }' config.txt

BEGIN and END Blocks

BEGIN runs before any input is processed. END runs after all input is consumed. These are essential for initialization and summary output:

# Print a header and footer
awk 'BEGIN { print "Name\tScore" } { print $1, $2 } END { print "---done---" }' scores.txt

# Sum a column
awk '{ sum += $3 } END { print "Total:", sum }' expenses.txt

# Count lines matching a pattern
awk '/error/ { count++ } END { print "Errors:", count }' app.log

# Calculate an average
awk '{ sum += $2; count++ } END { print "Average:", sum/count }' grades.txt

Real-World Examples

Analyzing Apache/Nginx Access Logs

# Count requests per IP address
awk '{ ips[$1]++ } END { for (ip in ips) print ips[ip], ip }' access.log | sort -rn | head -20

# Find all 500 errors
awk '$9 == 500 { print $1, $7 }' access.log

# Calculate total bytes transferred
awk '{ total += $10 } END { printf "Total: %.2f GB\n", total/1024/1024/1024 }' access.log

# Requests per hour
awk -F'[/ :]' '{ hours[$7]++ } END { for (h in hours) print h, hours[h] }' access.log | sort

Processing CSV Data

# Skip the header row and print specific columns
awk -F, 'NR > 1 { print $1, $4 }' employees.csv

# Filter rows where salary (column 4) exceeds 80000
awk -F, 'NR > 1 && $4 > 80000 { printf "%s: $%d\n", $1, $4 }' employees.csv

# Sum a column with header skip
awk -F, 'NR > 1 { sum += $4 } END { printf "Total payroll: $%d\n", sum }' employees.csv

System Administration

# Find users with UID >= 1000 (regular users)
awk -F: '$3 >= 1000 && $3 < 65534 { print $1, $3 }' /etc/passwd

# Show disk usage percentages over 80%
df -h | awk 'NR > 1 { gsub(/%/, "", $5); if ($5 > 80) print $6, $5"%" }'

# List all listening ports with process names
ss -tlnp | awk 'NR > 1 { print $4, $6 }'

# Memory usage summary from /proc/meminfo
awk '/MemTotal|MemFree|MemAvailable/ { printf "%s\t%.2f GB\n", $1, $2/1024/1024 }' /proc/meminfo

Text Transformation

# Convert a single-column list to comma-separated
awk '{ printf "%s%s", sep, $0; sep="," } END { print "" }' names.txt

# Swap first and second columns
awk '{ temp=$1; $1=$2; $2=temp; print }' data.txt

# Remove duplicate lines (preserving order)
awk '!seen[$0]++' file.txt

# Print lines between two markers (exclusive)
awk '/BEGIN_SECTION/{flag=1; next} /END_SECTION/{flag=0} flag' config.txt

String Functions

awk has built-in string functions for more complex transformations:

# Convert to uppercase/lowercase (gawk)
awk '{ print toupper($1), $2 }' names.txt
awk '{ print tolower($0) }' file.txt

# Substring extraction
awk '{ print substr($1, 1, 3) }' file.txt    # first 3 chars

# Find and replace
awk '{ gsub(/old/, "new"); print }' file.txt

# Split a field into an array
awk -F, '{ split($3, parts, "-"); print parts[1] }' data.csv

# String length
awk 'length($0) > 80 { print NR": "$0 }' code.txt   # lines over 80 chars

Associative Arrays

One of awk’s most powerful features is associative arrays (hash maps):

# Word frequency counter
awk '{ for (i=1; i<=NF; i++) words[$i]++ } END { for (w in words) print words[w], w }' book.txt | sort -rn | head -20

# Group and sum by category
awk -F, '{ totals[$2] += $3 } END { for (cat in totals) print cat, totals[cat] }' transactions.csv

# Cross-tabulation
awk -F, '{ counts[$1][$2]++ }' data.csv   # (gawk 4.0+ only)

Multi-line awk Programs

For complex processing, store your awk program in a file:

# report.awk
BEGIN {
    FS = ","
    printf "%-20s %10s %10s\n", "Department", "Employees", "Avg Salary"
    printf "%-20s %10s %10s\n", "----------", "---------", "----------"
}
NR > 1 {
    dept = $2
    salary = $4
    count[dept]++
    total[dept] += salary
}
END {
    for (d in count) {
        avg = total[d] / count[d]
        printf "%-20s %10d %10.2f\n", d, count[d], avg
    }
}

Run it with:

awk -f report.awk employees.csv

awk vs Other Tools

Use awk when you need column-based extraction with logic. For simple field extraction, cut is faster. For simple pattern matching, grep is more readable. For full-blown data transformations, consider Python or Perl. awk sits in the sweet spot for one-liners and short scripts that process structured text.

# These are equivalent, but awk is more flexible
cut -d: -f1 /etc/passwd
awk -F: '{ print $1 }' /etc/passwd

Summary

awk is an indispensable tool for text processing on Linux. The key concepts are field-based processing with $1, $2, etc., pattern-action pairs for conditional logic, BEGIN/END blocks for setup and summaries, and associative arrays for aggregation. Once you internalize these fundamentals, you will find yourself reaching for awk whenever you need to slice, dice, or summarize text data from the command line.