Command-line technique that came up while replatforming usghcc.org.
See also: 02-content-modelling.md ·
03-frontend.md · index.md
1. Heredocs: << 'EOF' vs << EOF
A heredoc feeds a block of text to a command’s standard input:
cat > file.txt << 'EOF'
hello
EOFcat > file.txt means “write what you read on stdin into file.txt”. The
heredoc supplies that stdin. Everything between the opening line and the
closing delimiter becomes the file.
The quoting is the part that matters. Compare:
cat << EOF # UNQUOTED delimiter
Cost: $100 and `date`
EOF
# → Cost: and Wed 24 Sep ... ($100 expanded to nothing, `date` ran)
cat << 'EOF' # QUOTED delimiter
Cost: $100 and `date`
EOF
# → Cost: $100 and `date` (literal, exactly as typed)With an unquoted delimiter the shell expands $variables, `backticks`
and history ! inside the body before the command ever sees it. Quoting
the delimiter turns all of that off.
Practical rule: when writing code or markdown, always quote the delimiter.
TypeScript is full of ${...} template literals and Tailwind classes contain
$; an unquoted heredoc silently eats them and you get a file that looks right
in your editor buffer but is wrong on disk.
The trap I actually hit writing this file
Writing this very document with cat > 01-shell.md << 'EOF' failed, because
the document itself contains a line that is exactly EOF at column 0 — the
example above. The shell saw it and closed the heredoc early, leaving the rest
of the file as broken shell commands:
(eval):250: parse error
The fix: pick a delimiter that cannot appear in the body. This file was
written with << 'MDDOC'. Any document about heredocs, or containing shell
examples, needs an unusual delimiter for exactly this reason.
The closing delimiter must also be at column 0 — no leading spaces — unless you
use <<-, which strips leading tabs only (not spaces, which is why <<-
is usually more trouble than it is worth).
2. python3 - — when sed stops being the right tool
python3 - << 'PY'
import pathlib
p = pathlib.Path("src/lib/schemas.ts")
s = p.read_text()
old = "export const homePageSchema"
assert old in s, "anchor not found"
s = s.replace(old, "// ...", 1)
p.write_text(s)
PYThe - means “read the program from stdin” rather than from a file. So the
heredoc becomes the script. Nothing is written to disk, nothing needs cleaning
up.
Why reach for this over sed: sed is line-oriented and its regex dialect
is awkward. The moment an edit needs to span multiple lines, or be conditional,
or happen exactly once, sed becomes a puzzle. Python is just code.
The habit that makes it safe: assert before writing
assert old in s, "anchor not found"
s = s.replace(old, new, 1)Two separate ideas, both important:
assert old in s— if the text you meant to replace is not there, the
script fails loudly. Without it,.replace()on a non-match returns the
string unchanged andwrite_texthappily writes the original back. You get a
silent no-op and believe the edit worked. This is the most common way
scripted edits go wrong.- the
1in.replace(old, new, 1)— replace the first occurrence
only. Without it, every occurrence changes.
Verifying after, not just asserting before
assert s.count("export const homePageSchema") == 1Assert the end state, not only the precondition. “There is now exactly one of
these” catches a duplicate that “the anchor existed” would not. On this project
a duplicated homePageSchema plus byte-identical duplicated import lines
appeared in two files — exactly the shape an insertion applied twice leaves
behind, and exactly what an end-state assertion would have caught immediately.
3. Reading a big XML export without a parser
The WordPress export (WXR) was a single 300-item XML file. You rarely need a
real XML parser to answer “what is in here”.
Count occurrences of a marker:
grep -c '<wp:post_type><!\[CDATA\[attachment\]\]></wp:post_type>' migration/*.xml
# → 217grep -c counts matching lines, not matches. If two markers sit on one line
you get 1. Fine for one-per-line XML, wrong for minified files — know which you
have.
Note \[ — in a bracket-heavy CDATA marker, [ is a regex metacharacter and
must be escaped.
Extract every URL of a given shape, deduplicated:
grep -oE 'https://[^<"]*\.(jpg|jpeg|png|webp)' migration/*.xml | sort -u | wc -l
# → 131-oprints only the matched part, not the whole line. This is what turns
grep from a search tool into an extraction tool.-Eselects extended regex, so(a|b)and+work without backslashes.[^<"]*— “any run of characters that is not<or"”. Defining a match
by what ends it is usually more robust than describing what it contains.sort -u— sort and drop duplicates.uniqalone will not do this:uniq
only collapses adjacent duplicates, so it is nearly always wrong without a
precedingsort.
Collapsing WordPress’s resized variants. WordPress stores
image-1024x683.jpg, image-300x200.jpg and image.jpg as separate URLs. To
get back to originals:
re.sub(r'-\d+x\d+(?=\.[a-z]+$)', '', url)(?=...) is a lookahead — it asserts the extension follows without
consuming it, so the extension survives the substitution.
4. Diagnosing a download that “works” but returns nothing
Symptom: curl exited 0, files were created, everything looked fine — but each
file was ~220 bytes.
file test.jpeg
# → HTML document, ASCII textfile reads magic bytes, not the extension. A .jpeg that is actually
HTML is instantly visible. Make this the first check whenever a downloaded file
is the wrong size.
Then read the headers:
curl -sSI --max-time 20 "https://usghcc.org/wp-content/uploads/2024/09/image00059.jpeg"-I— HEAD request: headers only, no body.-s— silent (no progress meter);-S— but still show errors.-sS
together is the combination you almost always want in a script: quiet when it
works, loud when it does not.
The response:
HTTP/2 202
content-type: text/html
sg-captcha: challenge
Three things, each decisive:
- 202 Accepted, not 200 OK. 202 means “received, not acted on yet”. No
static file server returns 202 for an image. content-type: text/htmlfor a.jpeg— the server is not sending the
image.sg-captcha: challenge— SiteGround’s anti-bot. The body was an HTML
<meta http-equiv="refresh">pointing at a captcha URL.
Why -w is worth knowing:
curl ... -w 'http=%{http_code} type=%{content_type} size=%{size_download}\n'-w prints a format string after the transfer, surfacing the status code
without a second request — otherwise curl succeeds silently on a 404 or a 202
and you never see it.
The lesson: curl exiting 0 means “the HTTP conversation completed”, not
“you got what you asked for”. A challenge page is a successful HTTP
transaction. Faking a browser User-Agent and Referer did not help either,
because the challenge requires executing JavaScript — no amount of header
spoofing substitutes for a JS engine. At that point stop scraping and use the
credentialed path (SFTP, or the host’s file manager).
The same trap wearing a different hat: identical sizes
Later, probing a map-tile CDN, the headers were clean — 200 OK,
content-type: image/png, and the file really was a PNG. The tell was the
size:
for u in "$A" "$B" "$C"; do
printf '%-70s ' "${u#https://}"
curl -s -o out.png -w '%{http_code} %{size_download}b\n' -m 20 -A "$UA" "$u"
done
# → 200 2049b
# → 200 2049b
# → 200 2049bThree different styles, three different map locations — exactly 2049 bytes
every time. Real tiles vary with how much is drawn on them; a dense city
block was 25 KB from a working server. Byte-identical responses to requests for
different content means you are being served a placeholder.
Generalise it: after file says the type is right, ask whether the size
varies the way the content should. A constant size across varying requests is
a stub, a cached error page, or a quota wall — regardless of the status code.
5. Small things worth keeping
Loop over a file of URLs:
while read -r u; do
curl -sSL --max-time 30 -o "$(basename "$u")" "$u"
done < urls.txtread -r disables backslash escaping — without -r, a backslash in the input
is eaten. Use -r every time.
-L follows redirects. --max-time bounds the whole operation, so one
unresponsive host cannot hang the loop forever.
Quote every expansion. "$u", "$(basename "$u")" — an unquoted variable
undergoes word-splitting and globbing, so a filename with a space becomes two
arguments. WordPress uploads are full of spaces and parentheses.
Outstanding — promote to the general vault
General technique, not usghcc-specific; should also land in devops/01-linux/:
- Heredoc quoting, delimiter collisions, and
python3 -as an editing tool -
grep -oEas an extraction tool;sort -uvsuniq - Diagnosing HTTP with
curl -I/-w, andfilefor magic-byte checks
Software engineering
Infrastructure & security
Computer science
Hardware & engineering