[Intermediate] — the memory model an exploit abuses, and why memory-unsafe languages made this a whole discipline. Assumes the authorization in rules of engagement — this is lab, CTF and authorised-testing material.
The kid version first
A program keeps its data in a long row of numbered boxes. Some boxes hold your input; some hold the address the program will jump to next when a function finishes.
The bug is that the program often doesn’t check how much you wrote. Write past the end of the input box and you spill into the box that holds “where to go next” — and now you decide where the program goes. That’s the whole idea, and everything else is detail about how the boxes are arranged and what defends them.
Why this exists at all
Memory-unsafe languages — C and C++ — let a program read and write memory the programmer didn’t intend. There is no bounds check on an array by default, no check that a pointer still points at something valid. The language trades safety for control and speed.
So this entire discipline is the flip side of C’s design. In a memory-safe language (Rust, Java, Go, Python) these bugs are structurally prevented — which is exactly why “rewrite it in Rust” is a security argument, not just a fashion. Binary exploitation is what you learn to attack the enormous body of C/C++ that runs the world: kernels, browsers, embedded devices, and every CTF pwn challenge.
The process memory map
When a program runs, the OS gives it a virtual address space laid out roughly like this:
HIGH addresses
┌────────────────────┐
│ stack │ ← local variables, return addresses. GROWS DOWNWARD ↓
│ ↓ │
│ │
│ ↑ │
│ heap │ ← malloc/new. Grows upward ↑
├────────────────────┤
│ bss / data │ ← globals (uninitialised / initialised)
├────────────────────┤
│ text │ ← the machine code itself. Usually read-only + executable
└────────────────────┘
LOW addresses
The stack is where most beginner exploitation happens, because it holds both your input (local buffers) and the return addresses that control the program’s flow — right next to each other → the stack.
The heap is the harder, later target → the heap.
The classes of memory bug
An exploit needs a bug. The families, in rough order of how you’ll meet them:
| Bug | What happens |
|---|---|
| Stack buffer overflow | Write past a stack buffer, overwrite adjacent data including the return address |
| Heap overflow | Write past a heap chunk, corrupt heap metadata or an adjacent object |
| Use-after-free | Keep using a pointer after the memory was freed and reused for something else |
| Format string | printf(user_input) — the user controls the format, so they can read and write memory |
| Integer overflow | A size calculation wraps, so a “small” allocation gets a huge write |
| Off-by-one | Write one byte too many — enough to corrupt a length or a pointer’s low byte |
| Type confusion | Treat an object as the wrong type — common in browser and JIT exploitation |
Each is a primitive you build on. A bug rarely hands you code execution directly; it gives you a write-what-where, or a read, or control of one register, and exploitation is the craft of chaining those into full control.
Registers you must know
On x86-64, the ones that matter constantly:
RIP instruction pointer — the address of the NEXT instruction. THE PRIZE
RSP stack pointer — top of the stack
RBP base pointer — frame reference
RDI, RSI, RDX, RCX, R8, R9 the first six function arguments, in order
RAX return value
Controlling RIP is the goal of most exploits — whoever controls RIP controls what executes next. And the argument-register order (RDI first) is load-bearing later: to call system("/bin/sh") you need /bin/sh in RDI → ROP.
You do not need to write assembly fluently, but you must read it → assembly.
Endianness, the thing that trips everyone up first
x86 is little-endian — a multi-byte value is stored low byte first. So the address 0x401256 is written into memory as the bytes 56 12 40 00.
Every address you inject must be byte-reversed. Get this wrong and your first ten exploits fail for a reason that looks like magic. pwntools’ p64() does it for you, which is most of why everyone uses it → the toolkit.
Key insight
Binary exploitation is the discipline of turning “the program wrote somewhere it shouldn’t” into “I control what the program does.” It exists because C and C++ don’t check memory access, and it’s a chaining game — a bug gives you a small primitive (a read, a write, one controlled register), and the skill is composing primitives into control of RIP. Learn the memory map and the registers first; everything else is built on knowing where things are.
Related
- the stack and buffer overflows — the first real exploit
- C — the language whose design makes all of this possible
- memory allocation · the memory hierarchy
- build your own allocator — write the bugs, then attack them
Source: [reference] — Aug 2026. Lab/CTF/authorised-testing material.
Software engineering
Infrastructure & security
Computer science
Hardware & engineering