Exposing a Resource Leak in Yoda Protector

There are many reasons to wrap your product inside a program protector or packer - some of which are even beyond reproach. But you can't blindly entrust your code to the operations of code encryptors and obfuscators. Unless you perform some type of code quality review, you may be inadvertently destabilizing your customer's or target's system. Resources may not be disposed of properly; the program stack may be corrupted; the exception handling chain that you so carefully constructed may have an extra link or two. Since access to the source code for these packer programs is in most cases limited and traditional debugging tools such as Compuware's BoundsChecker may not function properly alongside these programs, one avenue open to you is to reverse engineer what the packer is doing. The packer that I will be examining for this article is one called "yoda's Protector" (version 1.03.2) and can be found at http://protools.reverse-engineering.net. Source code, which appears to be out of date, can be found at https://sourceforge.net/projects/yodap. (Building the source from the project files produces an executable with a version number of 1.0. but test programs "protected" by this version either crashed or blue-screened my system when a debugger was attached.)

For the purposes of this investigation I took a copy of CALC, the Windows calculator program (version 5.1.2600.0 (xpclient.010817-1148)) and "protected" it using Yoda's Protector. The options I selected were:

• Anti-SoftICE protection
• Checksum Protection
• API Redirection
• Anti-Dump Protection
• Clear Import Information
• Remove .reloc section
• Remove debug information
• Compress Option - 10
• Create backup copy
• Section's Name - .yP

(The packed version of CALC is included in the supporting files for this article.) If you compare file sizes both before and after this operation, you will see that the packed version is much smaller than the original --112Kb shrinks down to 81Kb. Launching the compressed version of the executable brings up the calculator program in all its glory. Let us see what happens when I start CALC using a debugger.

As I expected, nearly all the debuggers I tried experience problems. OllyDbg, WinDbg, and Visual Studio.NET 2003 crashed and burned with only one clue remaining: the desktop taskbar was unresponsive. My debugger, PEBrowse Professional Interactive (available at www.smidgeonsoft.com), locked up, but the taskbar was still disabled. SoftICE appeared to handle the program with no problem at all - the reason for this is mysterious, as I'll discuss later.

There is, however, one common thread running through the usermode debuggers - inability to handle this beast - the taskbar has been disabled. I can still use the three-fingered-salute (Ctrl-Alt-Delete) to bring up Task Manager and restart the system. Now it's time for me to roll up my sleeves and dive into the code behind the packed calculator program. (I will be using my own debugger during this discussion - the others should work well using the hints and addresses that I will be providing.)