A debugger is probably the most commonly-used tool when reverse-engineering (a disassembler tool such as the Interactive DisAssembler (IDA) being the next most common). As a result, anti-debugging tricks are probably the most common feature of code intended to interfere with reverse-engineering (and anti-disassembly constructs being the next most common). These tricks can simply detect the presence of the debugger, disable the debugger, escape from the control of the debugger, or even exploit a vulnerability in the debugger. The presence of a debugger can be inferred indirectly, or a specific debugger can be detected. Disabling or escaping from the control of the debugger can be achieved in both generic and specific ways. Exploiting a vulnerability, however, is achieved against specific debuggers. Of course, the debugger does not need to be present in order for the exploit to be attempted.
Typically, when a debugger loads, the debuggee's environment is changed by the operating system, to allow the debugger to interact with the debuggee (one exception to this is the Obsidian debugger). Some of these changes are more obvious than others, and affect the operation of the debuggee in different ways. The environment can also be changed in different ways, depending on whether a debugger was used to create a process, or if the debugger attaches to process that is running already.
What follows is a selection of the known techniques used to detect the presence of a debugger, and in some cases, the defences against them.
Note: This text contains a number of code snippets in both 32-bit and 64-bit versions. For simplicity, the 64-bit versions assume that all stack and heap pointers, and all handles, fit in 32 bits. They also rely on the fact that the PEB is always located in low memory.