The two stream processors. grep finds lines; sed and awk transform them.

Written 2026-08-11 — sed appeared in linux-reference
as lookup material but was never taught in the numbered sequence, and heavy sed use
during a documentation migration made the gap obvious.

Assumes text processing
(grep, find, regex) and bash scripting.


Which tool for which job

NeedTool
Find lines matching a patterngrep
Substitute text, delete lines, extract line rangessed
Work with columns, arithmetic, accumulate totalsawk
Anything needing real data structuresPython

The honest rule: sed for line-oriented edits, awk for column-oriented ones, and when
you find yourself writing an awk program longer than a few lines, switch to Python.


sed — the stream editor

sed reads input line by line, applies commands, and prints the result. It does not
modify files unless you tell it to.

The -n flag, and the classic beginner bug

By default sed prints every line it reads. The p command prints too. So:

sed '2,4p' file.txt      # ❌ lines 2-4 appear TWICE
sed -n '2,4p' file.txt   # ✅ only lines 2-4

-n suppresses the automatic printing, leaving p as the only source of output. Almost
every use of p wants -n.

Line ranges — extracting a slice of a file

sed -n '42p'        file.txt   # just line 42
sed -n '10,20p'     file.txt   # lines 10 through 20
sed -n '100,$p'     file.txt   # line 100 to end of file  ($ = last line)
sed -n '1,/START/p' file.txt   # from line 1 until the first line matching START

This is the practical way to carve up a file too large to open. Combined with grep -n to
find the boundaries:

grep -n "^## " big-document.md      # get heading line numbers
sed -n '245,388p' big-document.md   # extract one section

That two-step is exactly how a 2,500-line document gets split into chapters. Watch the
boundaries
— an off-by-one between where one range ends and the next begins silently
drops content. Verify afterwards rather than assuming (see
socioboom/01-shell for a worked verification).

Substitution — the command everyone knows

sed 's/old/new/'      file.txt   # first occurrence PER LINE
sed 's/old/new/g'     file.txt   # every occurrence (g = global)
sed 's/old/new/2'     file.txt   # only the 2nd occurrence on each line
sed 's/old/new/gi'    file.txt   # global, case-insensitive

The delimiter doesn’t have to be /. When the text contains slashes, switch it — this
avoids a wall of escaping:

sed 's/\/usr\/local\/bin/\/opt\/bin/g'   # painful
sed 's|/usr/local/bin|/opt/bin|g'        # same thing, readable

Any character works after s. |, #, and , are common choices.

Editing files in place

sed -i        's/old/new/g' file.txt   # GNU: edit in place, no backup
sed -i.bak    's/old/new/g' file.txt   # keep file.txt.bak
sed -i '' -e  's/old/new/g' file.txt   # BSD/macOS: -i REQUIRES an argument

The -i portability trap: GNU sed (Linux) takes an optional suffix attached to the
flag. BSD sed (macOS) requires a separate argument. A script using bare -i works on
Linux and, on macOS, silently consumes the next argument as the backup suffix. If a script
must run on both, use perl -pi -e instead, or test the platform.

Always dry-run first — omit -i, check the output, then add it.

Deleting and selecting lines

sed '3d'            file.txt   # delete line 3
sed '2,5d'          file.txt   # delete lines 2-5
sed '/^#/d'         file.txt   # delete comment lines
sed '/^$/d'         file.txt   # delete blank lines
sed '/^#/!d'        file.txt   # ! inverts — delete everything EXCEPT comments
sed -n '/BEGIN/,/END/p' f.txt  # print the block between two patterns

Addressing — apply a command only to matching lines

Any command can be prefixed with an address:

sed '/^server/ s/8080/3000/'  config.txt   # substitute only on lines starting "server"
sed '10,20 s/foo/bar/g'       file.txt     # substitute only within lines 10-20
sed '$ a\Appended last line'  file.txt     # a = append, $ = last line
sed '1 i\Inserted first line' file.txt     # i = insert before

Multiple commands

sed -e 's/a/b/' -e 's/c/d/' file.txt
sed 's/a/b/; s/c/d/'        file.txt
sed -f script.sed           file.txt   # commands from a file

Capture groups

echo "John Smith" | sed -E 's/(\w+) (\w+)/\2, \1/'    # → Smith, John

-E enables extended regex (no backslashes before (, +, ?). Without it you need
\( and \). Prefer -E — it matches the regex syntax you already know from
grep -E and most programming languages.

& in the replacement means “the whole match”:

echo "hello" | sed 's/.*/[&]/'    # → [hello]

awk — the column processor

awk splits each line into fields and runs a program against them. Where sed thinks in
lines, awk thinks in records and fields.

awk '{print $1}'        file.txt   # first whitespace-separated field
awk '{print $NF}'       file.txt   # LAST field (NF = number of fields)
awk '{print $1, $3}'    file.txt   # comma inserts a space
awk '{print NR, $0}'    file.txt   # NR = record number, $0 = whole line

Built-in variables

VariableMeaning
$0The entire line
$1, $2…Individual fields
NFNumber of fields on this line
NRCurrent record (line) number
FSInput field separator (default: whitespace)
OFSOutput field separator

Changing the separator

awk -F: '{print $1}' /etc/passwd          # colon-separated → usernames
awk -F, '{print $2}' data.csv             # CSV (naive — breaks on quoted commas)
awk -F'\t' '{print $1}' data.tsv          # tab

Patterns — the real power

The structure is pattern { action }. Omit the pattern and it runs on every line; omit
the action and it prints matching lines:

awk '/error/'                  log.txt   # like grep
awk '$3 > 100'                 data.txt  # rows where field 3 exceeds 100
awk 'NF > 5'                   file.txt  # lines with more than 5 fields
awk 'NR % 2 == 0'              file.txt  # every even line
awk '$1 == "POST" {print $7}'  access.log

BEGIN, END, and accumulation

This is what sed fundamentally cannot do — carry state across lines:

# Sum a column
awk '{sum += $3} END {print sum}' data.txt
 
# Average
awk '{sum += $1; n++} END {print sum/n}' numbers.txt
 
# Count occurrences into an associative array
awk '{count[$1]++} END {for (k in count) print k, count[k]}' access.log
 
# With a header
awk 'BEGIN {print "USER\tSHELL"} -F: {print $1"\t"$7}' /etc/passwd

BEGIN runs before any input, END after all of it.

Practical one-liners

# Top 10 IPs hitting a web server
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -10
 
# Total bytes served (field 10 in combined log format)
awk '{sum += $10} END {print sum/1024/1024 " MB"}' access.log
 
# Processes using more than 100MB RSS
ps aux | awk '$6 > 100000 {print $11, $6/1024 "MB"}'
 
# Non-system users (UID >= 1000)
awk -F: '$3 >= 1000 {print $1}' /etc/passwd

Gotcha: a pattern range re-opens every time the start matches

sed -n '/START/,/END/p' and awk '/START/,/END/' don’t stop after the first
block. Whenever START matches again after a block has closed, a new range
opens and runs to the next END. Extracting one SQL function with
/function foo/,/^\$\$;/ also picked up a later revoke all on function foo(...)
line, which opened a second range that ran to the end of a different function.

awk '/function foo/,/^\$\$;/' f.sql                          # ✗ also fires on "revoke … function foo"
awk '/^create (or replace )?function foo/,/^\$\$;/' f.sql     # ✓ anchored to the definition

The fix is to anchor START (^) to the one line you mean. The same applies to
grep -l when hunting for definitions. Checking the result with
diff <(old) <(new) is what exposed it.

Combining them

The pipeline idiom: grep narrows, sed reshapes, awk computes.

grep "ERROR" app.log \
  | sed 's/^\[\([^]]*\)\].*/\1/' \
  | awk '{count[$1]++} END {for (d in count) print d, count[d]}' \
  | sort

Read as: find error lines → extract the bracketed timestamp → count per day → sort.

Each stage does one thing. When a stage starts needing conditionals and nested logic, that’s
the signal to stop and write a script instead.


Seen in the wild

  • mypaddie: shell — awk range + sed + diff <(…) <(…) to re-create a SQL function from its latest migration, and the re-opening range gotcha above
  • arete: sed & awk — a deeper treatment written against real files
  • arete: regex from zero to advanced — the regex layer underneath
  • grep and find in depth — currently the vault’s deepest shell material, and it lives in a project folder
↑ Back to Linux