[Intermediate] — the payload you execute once you control RIP, and why it’s written the way it is.
The kid version first
Once you control where the program jumps, you need something to jump to. Shellcode is a tiny blob of machine code — usually one that launches a shell, so you get a command prompt on the target.
It’s written under strange constraints: it can’t contain zero bytes, it must work no matter where it’s loaded, and it has to be tiny. Those constraints are what make it look like arcane craft rather than ordinary programming.
What it does
The classic goal is a shell via the execve syscall:
execve("/bin/sh", NULL, NULL);In assembly that’s: put the syscall number in RAX, "/bin/sh"’s address in RDI, null in RSI and RDX, then syscall.
xor rsi, rsi ; RSI = 0 (argv)
xor rdx, rdx ; RDX = 0 (envp)
mov rdi, ... ; RDI -> "/bin/sh"
mov rax, 59 ; execve syscall number
syscallYou rarely write this from scratch — but you must understand it, because the constraints below shape every real payload.
The constraints, and why they exist
1. No null bytes. The overflow that delivers your shellcode is usually a string copy (strcpy), and a string ends at the first null byte. A \x00 in your shellcode truncates it mid-payload.
So mov rax, 59 — which assembles with leading zero bytes — becomes:
xor rax, rax ; RAX = 0, with NO null bytes in the encoding
mov al, 59 ; set only the low byte“Null-free” is the number-one shellcode constraint, and half of shellcode craft is avoiding zero bytes in the encoding.
2. Position independence. You don’t know the absolute address your shellcode lands at. So it can’t hard-code "/bin/sh"’s address — it computes it at runtime, classically by pushing the string onto the stack and pointing RDI at RSP:
mov rbx, 0x68732f6e69622f ; "/bin//sh" packed into a register
push rbx
mov rdi, rsp ; RDI -> the string, wherever the stack is3. Size. It has to fit in the buffer you’re overflowing. A /bin/sh payload is ~23–30 bytes; people compete to shrink it.
4. Sometimes a restricted character set — only printable ASCII, or no newlines — if the input path filters bytes. This is where encoders come in.
Getting shellcode without writing it
# pwntools — assembles correct, null-free shellcode for the arch
python3 -c 'from pwn import *; context.arch="amd64"; print(asm(shellcraft.sh()))'
# msfvenom — the classic generator, with encoders and bad-char avoidance
msfvenom -p linux/x64/exec CMD=/bin/sh -f python -b '\x00\x0a'-b '\x00\x0a' tells msfvenom which “bad characters” to avoid — the bytes the input path would eat. Identifying bad characters is a real step: you send all 256 byte values, see which get mangled, and exclude them.
shell-storm.org and exploit-db host hand-written shellcodes for niche cases.
Encoders and decoder stubs
If the input can’t contain certain bytes, you encode the payload into safe bytes and prepend a tiny decoder stub that unpacks it in memory at runtime. msfvenom’s x86/shikata_ga_nai is the famous polymorphic encoder.
Worth knowing this is also how malware evades signature detection — the same technique, different purpose. On the defensive side, this is why signature-based detection struggles and behaviour-based detection exists → security operations.
Staged vs stageless
- Stageless — the whole payload is delivered at once. Simple, larger
- Staged — a tiny first stage connects back and pulls the real payload over the network. Smaller initial footprint, needs a listener
Metasploit’s meterpreter is the well-known staged payload — a small stub that loads a full post-exploitation agent in memory → post-exploitation.
The modern reality
On a current system you usually cannot execute shellcode on the stack at all, because NX/DEP marks the stack non-executable. So modern exploitation often:
- Doesn’t inject shellcode — it reuses existing executable code via ROP, and the “shellcode” becomes a ROP chain that calls
execve, or - Uses ROP to make a region executable first (
mprotect), then jumps to injected shellcode, or - Just calls
system("/bin/sh")directly viaret2libc— no shellcode needed at all
So shellcode is foundational to understand and decreasingly delivered as raw bytes. The goal (a shell via execve/system) is constant; the delivery moved from “inject and run” to “assemble from what’s already there.”
Key insight
Shellcode is machine code written under three constraints that ordinary code never faces: no null bytes, no known load address, minimal size — and those constraints, not the logic, are what make it look like a dark art. In practice you generate it with pwntools or msfvenom and spend your effort on bad-character analysis. And on modern targets, the payload increasingly isn’t injected bytes at all but a ROP chain that calls the same syscalls.
Related
- buffer overflows — how it’s delivered
- bypassing mitigations — ROP, the modern replacement
- syscalls and the ABI — what
syscallactually does - assembly
Source: [reference] — Aug 2026. Generate and run against binaries you own.
Software engineering
Infrastructure & security
Computer science
Hardware & engineering