[Advanced] — how malware hides from detection and from you, and why the whole game is an arms race that signatures keep losing.

The kid version first

Malware authors know analysts and antivirus are coming for them, so they build in defences: hide the code so scanners can’t read it (packing), change it so no two copies look alike (polymorphism), and detect when they’re being watched and play dead (anti-analysis). Every one of these makes your job harder — and understanding them is the difference between an analysis that stalls and one that succeeds.

Two audiences it evades

Malware hides from two different watchers, and the techniques split accordingly:

  • Detection — evading antivirus, EDR, and network monitoring so it isn’t caught
  • Analysis — evading you, the analyst, so its behaviour and purpose stay hidden even once caught

Packing and obfuscation — hiding the code

A packer compresses or encrypts the malware and wraps it in a small stub that unpacks it at runtime. On disk, there’s nothing readable:

   [ small unpacker stub ][ encrypted blob of the real malware ]
        │ runs first
        ▼ decrypts the blob in memory, jumps to it
   [ the real code — only ever exists unpacked in RAM ]

Why it works against static analysis: the disassembler sees the stub and a wall of high-entropy garbage. No strings, no meaningful imports, no readable code. High entropy is itself the tell — a section that looks like random noise is packed or encrypted.

The counter is dynamic: run it, let it unpack itself, dump the real code from memory → unpacking in memory. UPX (a legitimate packer) is trivial to reverse (upx -d); custom and commercial packers (Themida, VMProtect) are genuinely hard.

Obfuscation is subtler than packing — the code runs as-is but is deliberately confusing: dead code, opaque predicates (branches that always go one way but look conditional), control-flow flattening (turning clean logic into a giant switch statement), and string encryption (strings built or decrypted at runtime so strings finds nothing).

Virtualisation-based obfuscation (VMProtect, Themida) is the hardest: the code is compiled into a custom bytecode run by an embedded interpreter, so you must first reverse the interpreter, then its made-up instruction set. This is state-of-the-art protection, used by serious malware and DRM alike.

Polymorphism and metamorphism — defeating signatures

Traditional antivirus matches known byte-patterns (signatures). Malware defeats this by changing its own bytes while keeping its behaviour:

  • Polymorphic — encrypts the payload with a different key each time and mutates the decryptor stub, so every copy has different bytes. The decrypted payload is identical, but the on-disk file never matches a byte-signature
  • Metamorphic — rewrites its own actual code each generation (reordered instructions, swapped equivalents, inserted junk), so even the unpacked code differs copy to copy. Rare and hard to write, but signature-proof

This is why signature-based detection is losing. A signature catches known bytes; polymorphism guarantees the bytes are never the same twice. The defensive answer is to match on behaviour (what it does) rather than bytes (what it is) — which is why the industry moved to EDR and behavioural detection → security operations, and why YARA rules target decrypted content and behaviour rather than the packed file.

Anti-analysis — detecting the watcher

Malware actively checks whether it’s being examined and changes behaviour if so — sleeping, exiting, or running harmless decoy code to waste your time:

Anti-debugging — detect a debugger and evade:

  • IsDebuggerPresent, CheckRemoteDebuggerPresent, checking the PEB debug flag
  • Timing checks — a debugger makes execution slow, so measuring elapsed time between two points reveals single-stepping
  • Detecting breakpoints (looking for the 0xCC byte software breakpoints insert)

Anti-VM / anti-sandbox — detect a virtual or analysis environment:

  • VM artefacts — VMware/VirtualBox drivers, registry keys, MAC address prefixes, device names
  • Sandbox evasion by patience — sandboxes run samples for a minute or two, so malware that sleeps for 20 minutes or waits for real user activity (mouse movement, a reboot, several document opens) never reveals itself in the sandbox window. A very common, very cheap evasion
  • Checking for too few CPU cores, too little RAM, or recently-created files (real machines have history; fresh VMs don’t)

Environmental keying — the payload only decrypts/runs on the intended target: a specific domain, machine, language, or geography. On any other machine — including your analysis box — it does nothing or can’t even be decrypted. This is both evasion and precision targeting, and it’s why some samples are near-impossible to analyse without the target context (Stuxnet is the famous example).

The counter-moves

Analysis fights back, and knowing the moves is the craft:

  • Anti-anti-debugging — patch out the checks (NOP the IsDebuggerPresent call), use debugger plugins (ScyllaHide) that hide the debugger, or hardware breakpoints that evade software-breakpoint detection
  • Anti-anti-VM — harden the VM to look like a real machine (change the artefacts the malware checks), or use bare-metal analysis for the stubborn ones
  • Patch time bombs — find the sleep(1200000) and set it to zero, or set the environment the malware is keyed to
  • Just unpack in memory — most obfuscation must resolve itself to run, so dumping post-unpack sidesteps a lot of it → memory dumping

Key insight

Malware hides from scanners by making its bytes unreadable (packing) and never repeating (polymorphism), and hides from analysts by detecting the debugger, VM, or sandbox and playing dead — so signature detection is a losing game and analysis is an arms race. The two structural lessons: defence had to move from matching bytes to matching behaviour (because behaviour is what malware can’t change without changing its purpose), and the analyst’s most reliable move is that obfuscation must undo itself to execute — run it in a hardened lab and dump the real code from memory.

Source: [reference] — Aug 2026.

↑ Back to Malware Analysis and Reverse Engineering