[Intermediate] — the memory model an exploit abuses, and why memory-unsafe languages made this a whole discipline. Assumes the authorization in rules of engagement — this is lab, CTF and authorised-testing material.

The kid version first

A program keeps its data in a long row of numbered boxes. Some boxes hold your input; some hold the address the program will jump to next when a function finishes.

The bug is that the program often doesn’t check how much you wrote. Write past the end of the input box and you spill into the box that holds “where to go next” — and now you decide where the program goes. That’s the whole idea, and everything else is detail about how the boxes are arranged and what defends them.

Why this exists at all

Memory-unsafe languages — C and C++ — let a program read and write memory the programmer didn’t intend. There is no bounds check on an array by default, no check that a pointer still points at something valid. The language trades safety for control and speed.

So this entire discipline is the flip side of C’s design. In a memory-safe language (Rust, Java, Go, Python) these bugs are structurally prevented — which is exactly why “rewrite it in Rust” is a security argument, not just a fashion. Binary exploitation is what you learn to attack the enormous body of C/C++ that runs the world: kernels, browsers, embedded devices, and every CTF pwn challenge.

The process memory map

When a program runs, the OS gives it a virtual address space laid out roughly like this:

   HIGH addresses
   ┌────────────────────┐
   │       stack        │  ← local variables, return addresses. GROWS DOWNWARD ↓
   │         ↓          │
   │                    │
   │         ↑          │
   │        heap        │  ← malloc/new. Grows upward ↑
   ├────────────────────┤
   │   bss / data       │  ← globals (uninitialised / initialised)
   ├────────────────────┤
   │       text         │  ← the machine code itself. Usually read-only + executable
   └────────────────────┘
   LOW addresses

The stack is where most beginner exploitation happens, because it holds both your input (local buffers) and the return addresses that control the program’s flow — right next to each other → the stack.

The heap is the harder, later target → the heap.

The classes of memory bug

An exploit needs a bug. The families, in rough order of how you’ll meet them:

BugWhat happens
Stack buffer overflowWrite past a stack buffer, overwrite adjacent data including the return address
Heap overflowWrite past a heap chunk, corrupt heap metadata or an adjacent object
Use-after-freeKeep using a pointer after the memory was freed and reused for something else
Format stringprintf(user_input) — the user controls the format, so they can read and write memory
Integer overflowA size calculation wraps, so a “small” allocation gets a huge write
Off-by-oneWrite one byte too many — enough to corrupt a length or a pointer’s low byte
Type confusionTreat an object as the wrong type — common in browser and JIT exploitation

Each is a primitive you build on. A bug rarely hands you code execution directly; it gives you a write-what-where, or a read, or control of one register, and exploitation is the craft of chaining those into full control.

Registers you must know

On x86-64, the ones that matter constantly:

RIP   instruction pointer — the address of the NEXT instruction. THE PRIZE
RSP   stack pointer — top of the stack
RBP   base pointer — frame reference
RDI, RSI, RDX, RCX, R8, R9   the first six function arguments, in order
RAX   return value

Controlling RIP is the goal of most exploits — whoever controls RIP controls what executes next. And the argument-register order (RDI first) is load-bearing later: to call system("/bin/sh") you need /bin/sh in RDI → ROP.

You do not need to write assembly fluently, but you must read it → assembly.

Endianness, the thing that trips everyone up first

x86 is little-endian — a multi-byte value is stored low byte first. So the address 0x401256 is written into memory as the bytes 56 12 40 00.

Every address you inject must be byte-reversed. Get this wrong and your first ten exploits fail for a reason that looks like magic. pwntools’ p64() does it for you, which is most of why everyone uses it → the toolkit.

Key insight

Binary exploitation is the discipline of turning “the program wrote somewhere it shouldn’t” into “I control what the program does.” It exists because C and C++ don’t check memory access, and it’s a chaining game — a bug gives you a small primitive (a read, a write, one controlled register), and the skill is composing primitives into control of RIP. Learn the memory map and the registers first; everything else is built on knowing where things are.

Source: [reference] — Aug 2026. Lab/CTF/authorised-testing material.

↑ Back to Binary Exploitation