[Beginner → Intermediate] · Reading and writing without corrupting data — encodings, modes, atomic writes, and the bytes/text boundary.
Named as a gap by this course’s own README and confirmed by the roadmap.sh Python track (file-handling, glob).
Always use with
with open("data.txt") as f:
content = f.read()
# closed here, exception or notWithout it you leak file handles — and a process has a limit (ulimit -n, often 1024). A long-running service that leaks handles dies with OSError: Too many open files hours after the actual bug → context managers.
Modes, and the one that destroys data
| Mode | Does | Missing file |
|---|---|---|
"r" | read (default) | FileNotFoundError |
"w" | write — truncates to zero first | creates |
"a" | append | creates |
"x" | exclusive create | creates; errors if it exists |
"r+" | read and write | error |
+"b" | binary — gives bytes |
open(path, "w") empties the file the moment it succeeds, before you write anything. If your program then crashes, the original is gone. That’s the most common way to lose data with Python, and the fix is the atomic write below.
"x" is underused — it’s how you avoid a race between “does this exist?” and “create it”, the same TOCTOU issue as EAFP.
Encoding — always state it
open("data.txt", encoding="utf-8") # ✓ always
open("data.txt") # ✗ platform-dependent defaultWithout encoding=, Python uses the platform default — UTF-8 on modern Linux/macOS, but historically cp1252 on Windows. The result is code that works on your laptop and raises UnicodeDecodeError in production, or worse, silently mangles non-ASCII characters. Nigerian names, accented characters, emoji, currency symbols — all break.
(Python 3.15 moves toward UTF-8 as the default; encoding= is still the honest thing to write.)
Handling imperfect input:
open(p, encoding="utf-8", errors="replace") # bad bytes → U+FFFD
open(p, encoding="utf-8", errors="ignore") # bad bytes dropped — data loss, be sure
open(p, newline="") # required for the csv moduleText vs binary
Text mode decodes bytes to str and translates line endings. Binary mode gives raw bytes.
with open("img.png", "rb") as f:
header = f.read(8) # bytes
with open("out.bin", "wb") as f:
f.write(b"\x89PNG")Anything not text opens in binary: images, PDFs, archives, protocol buffers, hashes. Opening a PNG in text mode corrupts it — the encoding layer mutates bytes that were never characters.
Reading large files
with open("50gb.log") as f:
for line in f: # ✓ lazy — one line in memory
process(line)A file object is already an iterator over lines. f.read() and f.readlines() both load everything → generators.
Fixed-size chunks, for binary:
while chunk := f.read(64 * 1024): # walrus operator
hasher.update(chunk)pathlib, for everything else
from pathlib import Path
p = Path("data") / "raw" / "input.csv"
p.read_text(encoding="utf-8")
p.write_text(content, encoding="utf-8")
p.read_bytes(); p.write_bytes(data)
p.exists(); p.is_file(); p.is_dir()
p.stat().st_size; p.stat().st_mtime
p.suffix; p.stem; p.name; p.parent
p.mkdir(parents=True, exist_ok=True)
p.unlink(missing_ok=True)
p.rename(other); p.resolve()Globbing:
list(Path("data").glob("*.csv")) # one level
list(Path("data").rglob("*.py")) # recursive
list(Path(".").glob("**/*.json")) # same as rglobglob returns a generator — wrap in list() if you need it twice.
Writing safely — the atomic pattern
The problem: a crash, a power cut, or a full disk mid-write leaves a truncated file, and the original is already gone. Readers see corruption.
import os, tempfile
from pathlib import Path
def atomic_write(path: Path, data: str) -> None:
fd, tmp = tempfile.mkstemp(dir=path.parent) # same filesystem — required
try:
with os.fdopen(fd, "w", encoding="utf-8") as f:
f.write(data)
f.flush()
os.fsync(f.fileno()) # force to disk, not just the OS cache
os.replace(tmp, path) # ATOMIC rename on POSIX and Windows
except BaseException:
os.unlink(tmp)
raiseos.replace is atomic: readers see either the old file or the new one, never a half-written one. The temp file must be on the same filesystem, or the rename becomes a copy and loses atomicity.
This is exactly what databases do — write elsewhere, flush, then atomically swap → durability and recovery. Any config file, cache or state file your program owns deserves this.
Structured formats
import json, csv
with open("data.json", encoding="utf-8") as f:
data = json.load(f)
with open("out.csv", "w", newline="", encoding="utf-8") as f: # newline="" matters
w = csv.DictWriter(f, fieldnames=["name", "total"])
w.writeheader()
w.writerows(rows)newline="" is not optional for csv — without it you get blank rows between records on Windows, because both Python and the csv module translate line endings.
Never parse CSV by line.split(","). Quoted fields containing commas, embedded newlines and escaped quotes are all legal, and the csv module handles them.
Temporary files and directories
import tempfile
with tempfile.TemporaryDirectory() as d:
work = Path(d) / "scratch.txt"
...
# directory and contents removed here, even on exceptionNever build a temp path by hand in /tmp — it’s a symlink-attack and collision surface → attacks.
Paths from users are dangerous
target = (base / user_input).resolve()
if not target.is_relative_to(base.resolve()): # 3.9+
raise ValueError("path traversal attempt")user_input of ../../etc/passwd escapes your directory. Path traversal is a top-tier web vulnerability, and the defence is to resolve and then verify containment → input validation.
Related
- the standard library —
pathlib,json,csvin context - asyncio — why async file I/O barely exists
- filesystems and storage — what
fsyncactually does - I/O models — blocking, non-blocking, io_uring
Source: [reference] — from the Python docs; roadmap.sh-cross-referenced.
Software engineering
Infrastructure & security
Computer science
Hardware & engineering