x86 Disassembly/Debugger Detectors

Detecting Debuggers

It may come as a surprise that a running program can actually detect the presence of an attached user-mode debugger. Also, there are methods available to detect kernel-mode debuggers, although the methods used depend in large part on which debugger is trying to be detected.

This subject is peripheral to the narrative of this book, and the section should be considered an optional one for most readers.

IsDebuggerPresent API

The Win32 API contains a function called "IsDebuggerPresent", which will return a boolean true if the program is being debugged. The following code snippet will detail a general usage of this function:

 if(IsDebuggerPresent())
 {
   TerminateProcess(GetCurrentProcess(), 1);
 }

Of course, it is easy to spot uses of the IsDebuggerPresent() function in the disassembled code, and a skilled reverser will simply patch the code to remove this line. For OllyDbg, there are many plugins available which hide the debugger from this and many other APIs.

PEB Debugger Check

The Process Environment Block stores the value that IsDebuggerPresent queries to determine its return value. To avoid suspicion, some programmers access the value directly from the PEB instead of calling the API function. The following code snippet shows how to access the value:

mov eax, [fs:0x30] 
mov al, [eax+2] 
test al, al 
jne @DebuggerDetected

Kernel Mode Debugger Check

On Windows 32 and 64-bit Win <XP?, 7,8.1 and 10.

There is a structure called _KUSER_SHARED_DATA at offset 0x2D4 it contains the field named 'KdDebuggerEnabled' which is set to 0x03 if a KDM is active or 0x00 if not.

Base address of the structure is static (0x7FFE0000) across different Windows versions even < XP.

The field is updated constantly with the last 2 bits set to '11' by the kernel.

The following assembly instruction will work in both 32 and 64-bit applications:

cmp byte ptr ds:[7FFE02D4], 3
je @DebuggerDetected

This has quite a few advantages. Known Source of information.

Timeouts

Debuggers can put break points in the code, and can therefore stop program execution. A program can detect this, by monitoring the system clock. If too much time has elapsed between instructions, it can be determined that the program is being stopped and analyzed (although this is not always the case). If a program is taking too much time, the program can terminate.

Notice that on preemptive multithreading systems, such as modern Windows or Linux systems will switch away from your program to run other programs. This is called thread switching. If the system has many threads to run, or if some threads are hogging processor time, your program may detect a long delay and may falsely determine that the program is being debugged.

Detecting SoftICE

SoftICE is a local kernel debugger, and as such, it can't be detected as easily as a user-mode debugger can be. The IsDebuggerPresent API function will not detect the presence of SoftICE.

To detect SoftICE, there are a number of techniques that can be used:

1. Search for the SoftICE install directory. If SoftICE is installed, the user is probably a hacker or a reverser.
2. Detect the presence of int 1. SoftICE uses interrupt 1 to debug, so if interrupt 1 is installed, SoftICE is running.

Detecting OllyDbg

OllyDbg is a popular 32-bit usermode debugger. Unfortunately, the last few releases, including the latest version (v1.10) contain a vulnerability in the handling of the Win32 API function OutputDebugString(). [1] A programmer trying to prevent his program from being debugged by OllyDbg could exploit this vulnerability in order to make the debugger crash. The author has never released a fix, however there are unofficial versions and plugins available to protect OllyDbg from being exploited using this vulnerability.