Unpacking malware like a pro

Malware authors deploy a variety of techniques to avoid Anti-Virus products and the most common of these techniques is malware packing. In this method, almost all parts of malware are obfuscated apart from an unpacking stub(Small piece of code loaded by the OS just as a normal program, which de-obfuscate the packed code during run-time )

Fundamentally, a packed program is a program that follows this pseudo-code:


E = […encoded malware …]
K = extract_key()
M = decode_malware(E, K)
address = load_in_memory(M)


Unpacking of malware is the first step to start analysis of a malware. Until we unpack a malware, we can not enumerate even the basic functionalities and attributes of the packed file like strings, API’s used, imports and export of a file.

In this blog we’ll discuss 3 step unpacking method, that can unpack almost all types of malware:

We need to place 3 breakpoints on the following Microsoft API’s:




  1. VirtualAlloc:

Reserves, commits, or changes the state of a region of pages in the virtual address space of the calling process. Memory allocated by this function is automatically initialized to zero.

LPVOID VirtualAlloc(
LPVOID lpAddress,
SIZE_T dwSize,
DWORD flAllocationType,
DWORD flProtect

The most important attribute of this function is lpaddress, which returns the starting offset of the reserved memory. This api is critical in analyzing space reserved/allocated by malware in cases of process injection, wherein it will extract the malware and then dump it in some other process.

2. VirtualProtect:

Basically this API changes the protection on a region of committed pages in the virtual address space of the calling process.

Malware needs to change permission of the region that was reserved by VirtualAlloc before injecting it into some other legitimate executable. Thus VirtualAlloc is always used as a complimentary API to change permission of allocated memory to read-write-execute.

BOOL VirtualProtect(
LPVOID lpAddress,
SIZE_T dwSize,
DWORD flNewProtect,
PDWORD lpflOldProtect

The most important attribute of this function is flNewProtect which changes permission of the reserved region to either PAGE_EXECUTE_READWRITE (0x40) or PAGE_EXECUTE_READ (0x20).

3) CreateProcessInternalW

Sequence of API’s for Creating a new process when invoking shell Execute [1 ]

CreateProcessInternalW is not a well documented API, even searching it on microsft msdn doesn’t show any result. But this API holds great value for us.

Whenever a malware unpacks by using self-injection method, it always needs to invoke a new process for which almost uncountable number of API’s can be used ranging from ShellExecute to CreateProcessW. Putting a breakpoint on any one of them can easily be defeated if malware uses some other API’s, but a trend was observed, which showcased that all these Process executing API’s converge to a single API during their execution cycle. This magic API was found out to be CreateProcessInternalW. Putting a breakpoint on this API would help us to detect and trace execution of almost all types of Self-injection malware techniques.

Place a breakpoint on:

CreateProcessInternalW : detecting self-injection techniques

VirtualAlloc , VirtualProtect : detecting all process injection techniques

or for safety put breakpoint on all the 3 API’s

Demo: { Process Injection }

For this demo, we take a 3 month old trick-bot sample, which uses process injection to drop malware into explorer.exe for further execution

PE sections of the packed malware
Imports of the packed malware

The above 2 images show the imports and the PE section of the packed malware. Having high entropy, less imports and big non-alignment in PE section table are a big indication of packed malware. Small fluctuation in the virtual size and raw size is acceptable and bound to happen, but a big or appreciable amount of difference in raw and virtual size gives indication that malware is packed. Furthermore, running Floss by Fireeye also doesn’t show up many recognizable strings, which serves as another indication of packer or crypter used by the malicious executable.

Putting breakpoint on VirtualAlloc

As explained above, our first step is to place breakpoints on the 3 API’s → VirtualAlloc, VirtualProtect and CreateProcessInternalW ( if you want even more protection, you can also insert breakpoint on CreateRemoteThread API).

This can be easily done in x64dbg by typing

bp VirtualAlloc

bp VirtualProtect

bp CreateProcessInternalW

in the command box in the bottom of the debugger.

Press step Over until you reach ret

After hitting first VirtualAlloc breakpoint, press step over button until you reach first return( ret or retn) instruction. On reaching ret instruction, see the EAX register, which holds address of the the allocated buffer of malware(on which malware will write executable that it wants to inject in pristine software).

Right click on the EAX → Follow in dump

Then run the program, until you keep hitting breakpoint on VirtualAlloc. Everytime you hit VirtualAlloc, follow the address in EAX register on different dump( you have 5 dumps in x64 dbg).

Everytime on hitting breakpoint, notice the previous dumps. After a few hits on VirtualAlloc, you’ll notice one of the previous dumps would have a MZ header visible in dump window.(means unpacked executable has been written)

Output after a few VirtualAlloc hit

When you see an executable dumped in one of the dumps, click the address in dump window and click follow in memory map.

Memory map location of dump which has unpacked malware

Right click on the section and click dump the file.

Open dumped file in a PE viewer like PE Bear and notice if it has been unpacked properly or not.

Imports of Unpacked Malware
Section Table of the unpacked malware

As we can see the dump has all imports and section table aligned. This means we have our unpacked malware, on which we can do our further analysis.

Similarly, for self-injection malware, we can use CreateProcessInternalW. I’ll cover the demo for that in next part of blog. Stay tuned ………

Hi! I am a B.Tech student, whose enjoys reverse engineering and digital forensics. I relish reading anything and everything about cybersecurity.