Programming, graphics, games, media, C++, Windows, Internet and more...
Mon
29
Apr 2024
If you are a programmer coding mostly in C++ for Windows, as I do, you likely use Microsoft Visual Studio as the IDE, including its code editor and debugger. When using Visual Studio for development, we typically compile, launch, and debug the code we developed. However, Visual Studio can also be used for debugging third-party executables. Having neither debug symbols (.pdb files) nor the source code of the application will result in a limited debugging experience, but it can still be useful in some cases. In this article, I will share a few tips and tricks for using the Visual Studio debugger to debug external .exe files.
I think this article is suitable for programmers with all levels of experience, also for beginners. We won't be looking at the x86 assembly code, I promise! 😀 All the screenshots are made with Visual Studio 2022 version 17.9.6 on Windows 10.
You can attach the debugger to an existing process using Debug > Attach to Process... command, but I think often we want to launch the process with the debugger attached from the very beginning. You can launch a selected .exe file under the Visual Studio debugger without having its source code available and building it. To do this:
This creates a special, dummy solution with 1 project inside that is not a C++ or C# project building any source code, but just a command to launch and debug the selected executable file. You can actually save this solution as an .sln file and go back to it later.
Once this is done, you control a debug session with the following commands. These commands function just like they normally do in the Visual Studio debugger.
In the Solution Explorer panel, when you right-click on a project and select Properties, you can change settings to be used for launching the process under the debugger, such as:
This works regardless of whether you use a normal project that compiles a source code or a project that just launches an .exe file. The screenshot illustrates the settings for an .exe project. Similarly, in case of a C++ project, you can find those settings in the project properties window in Configuration Properties > Debugging tab.
Editing command-line parameters as a single-line string can be inconvenient when using a complex CLI syntax with many parameters and testing their various combinations. Fortunately, there is a great Visual Studio extension that helps with this: Smart Command Line Arguments. After installing it, you can open a new panel with command: View > Other Windows > Command Line Arguments. Within the panel, you can conveniently edit parameters to pass to the current startup project in form of a list, with a possibility to enable/disable individual items. There is also a button to "Copy whole command line to clipboard" in case you need it.
Unfortunately, this extension seems to work only with solutions that build a source code and not the ones that launch an .exe file.
Among the project settings shown above, there is also Command parameter, which points to the .exe file to be launched. The possibility to change it means that we can work on a project that builds some executable but launch a completely different executable for a debugging session.
This capability also extends to static (.lib) and dynamic (.dll) libraries. While it's not possible to "'launch" a library directly, you can still set a library project as the startup project. By changing the Command parameter to point to some .exe file, you can enjoy a full debugging experience of your library, including breakpoints, watches, etc., as long as that executable links or loads this library.
What if the software we want to debug is comprised of multiple processes? There might be, for example, one "launcher" or "watchdog" process which then starts another process that does the actual work. Visual Studio debugs only the single process we start.
However, there is a nice extension that solves this problem: Microsoft Child Process Debugging Power Tool. After installing, it needs to be enabled. To do this:
When this is done, Visual Studio will debug the process we start and all the processes created by it.
When we debug our source code, we typically setup breakpoints, look at the call stack, inspect local variables, or use the Watch panel to enter custom expressions to evaluate. We usually know what processes and threads we have and what .dll libraries we load. When working with third-party executables without having the source code, we need a more general picture of the software we debug. For this, following panels may be useful:
Finally, the panel available under Debug > Windows > Output offers a text log that can be invaluable in some cases. It shows all the key events like process start, process exit (with its exit code), thread exit, .dll libraries loading, plus custom debug messages that the app prints using OutputDebugString function. To find out more about this function, see my old article "Ways to Print and Capture Text Output of a Process", section "OutputDebugString".
When debugging some application, in order to see function names on the call stack and other useful information, we need "debug symbols". They have a form of a .pdb file that accompanies the .exe file. When developing our own app and compiling it from the source code, this happens automatically. To be more specific:
When debugging a third-party executable, we likely don't have debug symbols for it, as .pdb files typically are not distributed with binaries packaged for end users. However, all hope for obtaining useful information on the call stack is not lost. Some threads may execute a code from Windows system libraries, for which Microsoft provides debug symbols. Furthermore, Visual Studio can automatically download these symbols. All you need to do is to enable their symbol server:
There is one caveat: Downloading these symbols can take a long time. When you enable symbol servers and Visual Studio attempts to download symbols, you'll encounter a window like the one shown below. Unfortunately, this process is fully synchronous, meaning it blocks Visual Studio as the window is modal. Additionally, it also blocks the application being debugged. I wish Microsoft would improve this process by loading these symbols in the background to prevent interruption of workflow.
Fortunately, the lengthy download process typically occurs only on the first launch or when a new .dll file is loaded that wasn't previously used. This is because for subsequent launches, the symbols (.pdb files) for the loaded .dll will already be cached on the local drive, resulting in faster debugging sessions. However, it is important to note that after a Windows Update, symbols may need to be downloaded again, as system libraries could have changed, necessitating an update of the cached symbols.
After loading debug symbols for system libraries, it becomes:
If you have obtained a .pdb file for an executable or a dynamic library that you want to debug, you can manually point Visual Studio to that file. Simply right-click on the module in the Modules panel and select "Load Symbols" from the context menu.
Finally, the last tip is about catching exceptions. When an unhandled exception occurs in a process that is not running under a debugger, the process simply terminates. However, when running under a debugger, the debugger catches the exception and provides an appropriate message showing what went wrong, such as a memory access violation.
We can configure Visual Studio to pause the application and break into the debugger when additional types of exceptions occur, even if they are caught and handled by the application, which will prevent them from going unnoticed. To enable this feature:
When an exception like this occurs, you can simply hit Debug > Continue to resume application execution.
To learn more about catching various types of exceptions (C++ exceptions, SEH exceptions), see my old article: "Why I Catch Exceptions in Main Function in C++".
When learning about using a debugger, we typically focus on setting up breakpoints, observing call stack, inspecting values of variables and data structures. In this article, I explained some of the higher-level features of the Visual Studio debugger that deal with entire processes, their launch parameters, modules (.dll libraries), and debug symbols.
Comments | #c++ #visual studio Share