Reverse Engineering - VM Setup - Windows - How to set up a virtual machine for your reverse engineering work on Windows.

Intro

In this blog post I will share my personal preferences for setting up a virtual machine (or VM) for my reverse engineering work (or RE). I admit that there are many ways one can set it all up. These are my personal choices for the software and configuration, based on years of experience in software reverse engineering.

I'm putting it all together in one place so that I can refer to it later from other posts.

Note that the nature of the software that you will be reverse engineering slightly dictates the different configurations for your setup. For instance, if you are mostly RE'ing malware or unwanted, or gray-zone software, then you should probably focus on the following (additionally to what I will outline below):

Disable network adapter in a guest VM, or make sure to route it through a secure VPN.

Do not share networks between VM guest and your host.

Disable clipboard sharing, or maybe even disable the drag-and-drop functionality between the guest VM and the host.

Ensure to install timely updates to the VM host itself to prevent any possible VM-escape by the malware.

Do not let the guest VM (that may contain malware) from running unattended in the background.

Upload and download files to the guest VM using some shared cloud account. I personally prefer mega.nz as they do not seem to run uploads through any antivirus checks, which may kill the upload if you're exchanging malicious binaries. Additionally, to ensure that cloud server scans don't block your files, you may use WinRAR to encrypt your upload with a password.

But overall, here I will not concentrate on RE malware. This post will cover general RE work.

This post briefly touches upon the following tools:

Virtual Machine (VM):
Guest Operating System
Native Disassembler/Debugger/Decompiler:
Managed (.NET) Debugger/Decompiler:
Process Hacker
Process Monitor
Far Manager
HxD
Microsoft Visual Studio (C++)
Detect-It-Easy
CFF Explorer
WERSetup
WinAPI Search
Spy++
WinID
Resource Hacker
Calculator
Notepad
Snipping Tool

Additionally, if you're not into reading blog posts and would rather watch a screencast click here.

Reverse Engineering Tools

It all boils down to tools. There are quite a lot of options to choose from. These are the ones that helped me out. If you feel that you want to add to this list, don't hesitate to leave a comment below.

In no particular order:

Unless specified in the description, all software that is shown below is free to download and use. Moreover, at the time of this writing the installers for the aforementioned software did not contain any junkware that was not advertised on the package.

NOTE though that since the software listed below is not developed by the author of this blog post, I CANNOT guarantee that it will remain free and untainted by unwanted additions. Always exercise caution when downloading and installing software! My advice is to always install new software in a VM.

Virtual Machine (VM) - doing your RE work in a VM is very important. And this does not only apply to RE'ing malware. Any software that you install and run can "gunk up" the OS that it runs on: anything from installing and failing to remove files and System Registry entries, to adding callbacks and hooks that may slow down your system or render it in a faulty state.
Besides keeping an overall stability of your host OS, VMs provide a very handy feature, called "Snapshot", that saves the state of the guest OS and allows you to revert to it at some later time. This is indispensable for both the RE work and just for the general software development.

My choices of the virtualization software:
- VMWare Workstation Pro - it's a paid product. At the time of this writing it was a one-time payment to buy a license for a limited number of installations. I like this option versus a monthly subscription.
  VMWare Workstation 16 Pro, running Windows 11 guest OS.
  
  The reason I prefer VMWare is the following:
  - Their shared Clipboard between the host and guest VMs. And not only for the text, but also for copying files. Surprisingly, this seems to be an issue with other companies.
  - Their support for drag-and-drop of files from the host to the guest and back. This is a very handy feature.
  - Overall ease of setup and the speed with which the guest VM runs.
  - Support for wide range of guest operating systems. (Although lacking support for macOS.)
- VirtualBox from Oracle - it's a free solution, which explains its wide popularity. In my view though, it is lacking some features of the VMWare Workstation that I listed above and is somewhat more cumbersome to set up. But overall, it can serve the same function. It is also cross-platform, which is really nice. I used it in the past and it works.
- Parallels for macOS - it's a paid product as well. And as VMWare Workstation, at the time of this writing, it was a one-time payment.
  When I am writing this, Parallels seems to be the only reliable way to install and run not only a Windows OS, but also a macOS in a guest VM on an ARM-based Mac hardware. This comes with painful limitations though. The main one being the lack of snapshots for a guest macOS.
Note that there are other VM providers that are both paid and free. But in my view they lack in usability or ease of use when compared to the options that I listed above.

Note that all other software mentioned below should be installed in a guest OS in the VM.
Guest Operating System - at the moment I'm using Windows 10 Pro (or Pro N to eliminate some of the bloatware) as my guest OS. But this will depend on the specific requirements. Some software that you are RE'ing may run only on a server-type OS, or other may require newer OS, like Windows 11.
Make sure to download the installation ISO for Windows 10 or Windows 11 from a Microsoft server, and don't use third-party ISOs for the operating system.

Most virtualization products though support installation (and running) of multiple guest OS'es at the same time. So you are only limited by your available RAM and disk drive space on the host computer.

To prevent unexpected reboots and changes to Windows system files I would strongly recommend disabling automatic updates in the guest OS, like I described here.

If you have an OEM Windows license, or if the company that you work for can provide one for your guest VM, then use it to activate the OS. Otherwise, for just a freelance reverse engineer, if you don't activate Windows 10 it will display an overlay UI over the desktop and will prevent you from changing the desktop background, but it will continue working. So hypothetically this should not stop you from using an inactivated guest Windows 10 OS for your RE work.
Native Disassembler/Debugger/Decompiler - these tools are essential for any form of the software RE work. They all usually come in one package or suite.
A disassembler converts the machine code of a binary app that you are RE'ing into a human readable form, and a debugger lets you step through and analyze the workings of the RE'ed app. A decompiler is a relative newcomer to the scene. It attempts to convert the machine code of a binary app to a pseudo-C type language that was used to compile it. If done properly, a decompiled code is easier to analyze and to work with for the reverse engineer.

Note that relying solely on decompilers is a bad practice for RE'ing. The reason being, is that decompilers often produce an erroneous or confusing output and thus may be misleading. It is always advisable to defer to the original disassembled code for verification.
Moreover, decompilers from commercial products seem to be very overpriced.

By saying debugging, we assume the following operations that one can do with the binary code:
- Attach to a running process, or start it on demand.
- Place breakpoints and stop the process at will, or upon a certain condition.
- Step through the process instruction-by-instruction.
- Read the virtual memory of the process.
- Read threads in the process as well as its other meta-data.
- Read the state of the CPU(s) and change it.
- Change the virtual memory of the process and to redirect its control flow.
- Terminate and restart the process.
Debugging and disassembling tools also tend to fall under two main categories:
1. Dynamic (or Live) Debugging Tools - these allow you to actively debug the binary file by running it under a debugger, providing all of the options that I listed above.
2. Static Analysis Tools - these let you disassemble (and sometimes decompile) the binary file and to analyze its structure without the ability to run and debug it.
My choices of the debuggers/disassemblers are:
- Ghidra - is a static disassembler and a decompiler. In recent releases it also claims to be a live user-mode debugger.
  Ghidra during static analysis.
  
  PROs:
  - It is free.
  - It is open source.
  - It has a somewhat OK decompiler, which is also free.
  - Ghidra is multi-platform and runs on Windows, Linux or macOS, to name just the most popular platforms.
  - It can disassemble binaries from different CPU architectures.
  - It now supports dynamic user-mode debugging. (Previously Ghidra was only a static analysis tool.)
  CONs:
  - It is quite slow and uses up a lot of resources (RAM). This is mostly due to the fact that it is written in Java.
  - Ghidra operates in a kinda unusual way. Its UI and everything else doesn't follow general OS design guidelines.
  - Its disassembler and other windows are not easy to read and navigate.
  - Setting up custom structs and types is poorly supported.
  - Built in decompiler often produces misleading and confusing pseudo-code that is hard to read.
  - Some people may not like that Ghidra was released and maintained by a three-letter government agency in the US. I personally have no issues with that.
- WinDbg - this is the oldest debugging tool in the list. It is also developed by Microsoft and is most certainly used by their developers to debug Windows as well as other apps that Microsoft ships.
  The link above lets you download the new version of WinDbg. In case you prefer an old-school classic WinDbg, you can get it from the WDK, or SDK, as described here.
  
  WinDbg Next (released in 2023) with a crash dump opened.
  
  PROs:
  - It is free and included with the Windows SDK or WDK.
  - Supports debugging kernel-mode binaries. (It seems to be the only debugger that does this.)
    Read my other blog post for details on how to prepare for the kernel mode debugging.
  - Supports crash dump analysis.
  - Can interpret and work with the callstack specific for .NET binaries.
  - Has multiple extensions that greatly improve the debugging workflow.
  - WinDbg binaries are quite small and fast. It is also portable.
  CONs:
  - It is mostly command-based, lacks good UI. (Although the new version has greatly compensated for it.)
  - Does not have a decompiler.
  WinDbg still remains a de-facto debugger for most Microsoft-specific binaries.
- x64dbg/x32dbg - is a third-party debugger and disassembler. Technically there are two flavors to it: x64dbg and x32dbg, but in reality all you need is just x64dbg.
  x64dbg during a debugging session.
  
  PROs:
  - It is free.
  - It is open source.
  - It has a somewhat decent user-mode debugger.
  - It has extensive UI that helps in the RE workflow.
  - It has some plug-ins that can improve your work.
  CONs:
  - Is is somewhat slow. (Due to the fact that it's written with QT.)
  - It is Windows-only. (I'm curious, what was the point of using QT then?)
  - Disassembler window does not support a more friendly graph view.
  - Does not have a decompiler.
  I can't really explain it but there's something about x64dbg that doesn't make me want to use it.
- IDA (Free/Pro) - it is probably one of the oldest commercial tools that survived to this day.
  I remember during its earlier days IDA was an inferior cousin to another great debugger of that time, called SoftICE. Unfortunately though, SoftICE did not make it through the sands of time, but IDA did.
  
  To download a pro-version you will need to check their website.
  
  PROs:
  - Support for wide range of CPU architectures.
  - It is cross platform.
  - Pro version has a built-in decompiler.
  - It has extensive library of plug-ins. For instance, a popular BinDiff tool only works with the IDA PRO.
  CONs:
  - Outrageously expensive. The cost also involves yearly subscription.
  - Very slow (just as x64dbg, IDA is developed using a slow QT cross-platform engine.)
  - It is quite buggy.
  - It has very quirky UI and shortcut keys that require extra time to learn.
  - No support for a basic undo operation.
  - Overall, it has quite steep learning curve, or a barrier to entry.
  IDA would be my least favorite debugging tool. I'm including it here only because it is one of the most popular solutions these days, used primarily by large RE companies.
Unfortunately, in my view, there's currently no single debugger/disassembler that would 100% satisfy my RE needs, thus I usually resort to installing and using several of the aforementioned products at once.

My personal picks are:
- Ghidra
- WinDbg
- IDA Free
Managed (.NET) Debugger/Decompiler - if you're dealing with managed (or .NET) binaries in Windows, RE'ing them using a native debugger/disassembler would be akin to writing a backend web server in Assembly. There are special tools for dealing with .NET binaries instead.
Let me review some of the tools that I find useful for RE'ing managed Windows binaries:
- dnSpyEx - is a third-party debugger and .NET assembly editor, that can not only let you decompile a .NET binary into an approximation of the original .NET source code, but also to debug it (attach to a running .NET process, step through it, view states of internal variables, set breakpoints, and so forth.) It also allows to modify .NET binaries and recompile them with your modifications.
  The initial dnSpy tool was abandoned by its original creator and is now maintained as a GitHub fork.
  
  dnSpyEx after attaching to a .NET process.
  
  dnSpy is indispensable for debugging, or RE'ing .NET binaries in Windows.
- IL Spy - is another .NET assembly browser and decompiler. It can retrieve the wealth of information about a .NET binary.
  In my view, the IL Spy is not as user-friendly as dnSpy. But it is a good tool to keep, in case other tools don't produce a desired result during your RE work.
  
  Just as the initial dnSpy, the original IL Spy was also abandoned and is now maintained as a fork.
- de4dot - is a tool to remove packing in a .NET binary. It may be handy for RE'ing malware that could have been obfuscated, or packed.
- MegaDumper - is another tool to de-obfuscate a .NET binary. This one differs from de4dot in that it can unpack running .NET binary from memory.
- Other .NET Tools - there are some other (optional) .NET tools that are handy for RE. You can view them at this repository, that were conveniently collected in one place.
- .NET Source Code Reference - Microsoft had open-sourced their .NET framework and it is now conveniently available online via a handy lookup resource. It is my definite go-to resource when I need to understand the internals of the .NET framework.
Process Hacker - is a replacement for the Windows Task Manager. While the latter one is a good app for general users, any reverse engineer needs a better tool to terminate processes, as well as to retrieve all kinds of important (and technical) information about running programs and threads.
Process Hacker - current view of running processes + properties for a specific process.

Some people may say that Sysinternal's Process Explorer is a decent replacement for the default Task Manager, but I will disagree. It was indeed a good alternative in the past, but it has been greatly surpassed by the Process Hacker these days.

The Process Hacker has been lately receiving a lot of false-positive detections from the Windows Defender. To the best of my knowledge there's no malicious code in the Process Hacker and those reports are just false-positives.
Process Monitor - the tool from Sysinternals that is still quite handy. It allows for a reverse engineer to record multiple events that were (and are) happening in the system: anything from opening, closing and reading/writing files, to operations with the system registry, to network access. After that a reverse engineer can filter recorded data using many filters, including narrowing it down to a specific process.
Process Monitor, while capturing info about running processes in the system.

There's a slight learning curve to using Process Monitor, but it is worth it for your RE work.
Far Manager - is a small text-based file manager that is irreplaceable if you want to know what is happening in the file system. The issue with the default File Explorer in Windows is that it doesn't show a good deal of information about the file system. Far, on the other hand, does!
Far Manager - two-pane file system view.

Aside from clearly showing all available files and directories, Far can also retrieve their attributes, permissions, as well as to act as a text and hex viewer/editor of file contents. This is very handy when you need to quickly view the contents of a file, or to check the basic PE file structure.

I also use Far to search for files, not only by their name and extension, but also by using specific file contents. This comes very handy when Microsoft's File Explorer is atrocious at its search capabilities.

Far can be also used to run/open various files, pretty much like File Explorer does, with the exception that you can run Far elevated, which helps you to avoid clicking through the UAC prompts during your RE work.
HxD - is a super light and easy-to-use hex editor for files. It also allows to apply binary patches to existing files with ease. It also has quite a nifty search of the binary contents of a file using a sequence of hex bytes, which is perfect for binary patching.
HxD with an open PE file.
Microsoft Visual Studio (C++) - don't confuse it with the VS Code. I'm talking about a full-fledged Visual Studio with the C++ compiler and IDE.
Microsoft Visual Studio (C++) - Looking up value for the WM_PAINT constant.

You can download and install its "Community Edition", which is free. After that install only the features needed for the "Desktop C++ Development" to save space in your VM. Optionally, you can install or re-use an existing installation of the Visual Studio, on your host machine.

Keep in mind that the "Professional" or "Enterprise" editions of the Visual Studio are not free. Although I see no reason to use them.

My reasoning behind installing the Microsoft C++ compiler and the IDE is to be able to look up functions and symbols that are used in Windows programs. Make sure to install the SDK (and maybe even WDK) alongside the Visual Studio.

Additionally you may need to run some simple tests, that you may write in C or C++. All that will require a compiler, in this case Microsoft Visual C++ compiler.

Unfortunately the installer for the Visual Studio is pretty large. Expect to use up 8-10 GB of space just for that.
Detect-It-Easy - is a tool used to determine how a binary (PE) file was built. It gives a general outline of the binary and its type: native, .NET, etc. It also shows a compiler that was used to build it.
Detect-It-Easy tool with the analysis of a binary PE file.

This tool is useful on the first step of the RE process to determine which tools to use to RE the binary further. For instance, if you see that the binary executable was built with a C/C++ compiler, you will know that you need to use a native debugger/disassembler to RE it. But if it was built as a managed .NET application, you will use a totally different managed tool, such as dnSpy.
CFF Explorer - is a viewer/editor for binary PE files. It can be used to quickly analyze a PE file without resorting to running it via a debugger, or disassembling it.
CFF Explorer with an open PE file.

This tool can be also used for a binary patch to add, or remove sections from a PE file.

Another tool that was quite handy for analyzing the structure of PE files was PE Internals. Unfortunately it doesn't seem to be available from the developer's website andreybazhan.com. Use this link to download the archived copy of that tool.
WERSetup - is one of my own tools that can be used to set up Windows Error Reporting functionality. I use it to enable Windows to generate automatic crash dumps in case any, or specific user-mode process(-es), crash.
WERSetup tool with the C:\CrashDumps folder set up to collect up 10 crash dumps in the system.

I use this tool both for my development and RE work.
WinAPI Search - is my tool as well. It allows you to search system directory for specific Win32 APIs, as well as to analyze PE files for dependencies (akin to a discontinued Dependency Walker from Microsoft).
WinAPI Search tool while looking for exported functions with CreateWindow in their name.

Another application for this tool is that it can be used to search for the meaning of Windows error codes, and to undecorate C++ linker symbol names, among others.
Spy++ - this tool comes bundled with the Microsoft Visual C++. It will be available as a part of the utilities in the Tools main menu in any Visual Studio C++ project.
Spy++, showing current window layout in a desktop.

Spy++ is used to analyze window structure of the current desktop when RE'ing a GUI app. It can also view in real time window messages and notifications that are broadcast for a specific control.
WinID - is my tool as well. It works similarly to Microsoft's Spy++, with the exception that it outputs to its screen information about windows and classes (in the current desktop) that a user points to with a mouse. It may be more handy in a sense that you just point a mouse to a window, or to a control, to get information about what process it belongs to, or what window class it represents.
WinID tool, displaying information about a window with the TOOLBARCLASSNAME class.

This tool can be handy while RE'ing GUI apps in Windows.
Resource Hacker - is a third-party tool that allows to view and edit resources in Win32 binaries.
Resource Hacker tool with an open binary file.
Calculator - is a new Windows calculator that supports easy conversion from hexadecimal to binary and to decimal numbers. This is especially handy during live debugging. Make sure to switch it to the Programmer mode though.
Windows Calculator, while displaying a conversion from 1000 hex.

One such conversion could be required when working with process IDs (or PIDs.) Some apps may represent them in a decimal form (like Process Hacker, for instance), while others may require them in a hex form (like a debugger.)

Make sure to pin the calculator to the taskbar for a quick access.
Notepad - is also a part of the Windows OS. Notepad is a handy tool to jot down and to copy-and-paste quick notes during your RE workflow. Make sure to pin it to the taskbar for a quick access.
Windows Notepad with some random RE'ing notes.
Snipping Tool - is a quick and easy screenshot making tool for Windows. It will be handy to make a quick screenshot of a screen (for a later analysis) in case of a fast-flowing debugging session.
Windows Snipping Tool.

This tool has been deprecated by Microsoft in Windows 11. But you can still find it on Windows 10, or an earlier OS.

After you fully set up your VM make sure to make a snapshot to capture the state of the guest OS. You would want to do it to be able to revert back to it after you finish your RE work to reset the guest OS to its clean state, or in case of some unrecoverable malfunction.
Snapshots are a great feature of most of the virtualization software. So make sure to learn how to use them, to take a full advantage of their benefits.

Video Overview

I recorded a video where I demonstrated the installation and a quick demo of most of the tools that I mentioned above:

Play video fullscreen

Conclusion

The list of tools above is far from being complete. If I come across some other handy tool, or if existing tools change, I will try update it here for future reference.

And in case you have your own suggestions for your favorite RE tools, that I forgot to mention above, don't hesitate to leave a comment below.

Happy RE'ing!

Blog Post

Reverse Engineering - VM Setup - Windows

How to set up a virtual machine for your reverse engineering work on Windows.

Intro

Contents

Reverse Engineering Tools

Video Overview

Conclusion

Social Media

Contact

Related Articles