Reversing a DLL in order to use it.

[Li_]

I have a dll that i would like to use, however i'm not quite sure of how to use the functions provided by it.

Using PE Explorer i find there there are 3 functions.

The DLL is not packed.

What is the best way to find out what arguments the functions take and return? more or less how can i find out how to use it without having the documentation?

I also have a program that uses the dll properly but i think it might take more time to reverse the app to see how it uses the dll?

Thanks.

Edited April 19, 2011 by Li_

[Hyperlisk]

Use IDA. IDA is definitely one of the best, if not THE BEST, tools for static analysis. And if you have any programs that DO use that DLL, observe how they interact with it. Using that basic technique, I reversed a DLL for a Chinese MP4 player I own so I could create my own updates for it.

Edited April 19, 2011 by Hyperlisk

[Li_]

Use IDA. IDA is definitely one of the best, if not THE BEST, tools for static analysis. And if you have any programs that DO use that DLL, observe how they interact with it. Using that basic technique, I reversed a DLL for a Chinese MP4 player I own so I could create my own updates for it.

Do you have any other pointers on seeing what i should look for when i debug using IDA? Also there isnt a way to see what args a function would take just via the dll?

[Hyperlisk]

Unfortunately not. Types of parameters only make sense in programming language, not so much in machine code. Assembly doesn't know if it's modifying a structure or a variable. It just does what it's told. One thing that can give you clues is the exports of your DLL. If it exports something like OpenSomeResource(), you can tell the return value will probably be a resource of SomeResource type. Then, coupled with applications that natively use your DLL, you can see how it uses the SomeResource object/type, so you can implement the same functionality yourself. Once you find it is using the SomeResource object/type, you can begin decoding each field of the structure. Reversing this way is a very incremental process.

Edited April 19, 2011 by Hyperlisk

[Li_]

After further studying of the DLL PE Explorer states that the dll's subsystem is Win32 GUI...ive search a bit on this via google, but i havent come across something that explains exactly what this means.

[Hyperlisk]

Simply put, it uses WinMain(), instead of a plain old, C-style, main(). So it is ready to recieve messages from Windows regarding its GUI components, etc.

Edited April 20, 2011 by Hyperlisk

[Li_]

Simply put, it uses WinMain(), instead of a plain old, C-style, main(). So it is ready to recieve messages from Windows regarding its GUI components, etc.

So does this mean, the dll itself can be "pulled" up as a window?(as GUI)

[Hyperlisk]

More like it is ready to use in a Windows GUI system.

[Li_]

More like it is ready to use in a Windows GUI system.

Could you elaborate on that please.

[Hyperlisk]

It's hard to without just saying "It's a DLL". It's just saying that this particular PE file (DLL) is targeting a PC platform.

[Killboy]

Technically speaking all Windows binaries use WinMain, although when coding console apps you 'apparently' use main.

The subsystem flag just tells Windows what subsystem is required to run this app, so if it says Win32GUI this exe/dll requires the graphical user interface subsystem of Windows.

There is no equivalent for console apps so I guess they use IMAGE_SUBSYSTEM_WINDOWS_GUI too

Long story short, it runs on any Windows with a GUI.

[chickenbutt]

Technically speaking all Windows binaries use WinMain, although when coding console apps you 'apparently' use main.

The subsystem flag just tells Windows what subsystem is required to run this app, so if it says Win32GUI this exe/dll requires the graphical user interface subsystem of Windows.

There is no equivalent for console apps so I guess they use IMAGE_SUBSYSTEM_WINDOWS_GUI too

Long story short, it runs on any Windows with a GUI.

Also most people don't grasp the concept of PE and ELF early enough. It's a wrapper for kernel-managed machine-code, resources, imports/exports, and allocation data like .reloc and alignment headers. In RCE rarely does it give insightful pointers to the algorithms contained.

The easy way in this case is reversing the app that uses it by simply break pointing on call stacks, or patching the DLL with inline hooks if anything is obfuscated and you don't want to unpack. If it's .NET you can manually reverse off the msil partitons or use a tool.

IDA is pointless unless you need a half working decompiler or you work with other instruction sets in different wrappers. It's like two thousand dollars and has a ****ty upgrade plan. People who actually use it are too boring to leak it.

Edited April 20, 2011 by chickenbutt

[Li_]

Also most people don't grasp the concept of PE and ELF early enough. It's a wrapper for kernel-managed machine-code, resources, imports/exports, and allocation data like .reloc and alignment headers. In RCE rarely does it give insightful pointers to the algorithms contained.

The easy way in this case is reversing the app that uses it by simply break pointing on call stacks, or patching the DLL with inline hooks if anything is obfuscated and you don't want to unpack. If it's .NET you can manually reverse off the msil partitons or use a tool.

IDA is pointless unless you need a half working decompiler or you work with other instruction sets in different wrappers. It's like two thousand dollars and has a ****ty upgrade plan. People who actually use it are too boring to leak it.

I can use IDA if need be, i also have olly, immunity..etc.

The dll and the app are created with vc++ 6

Ive already unpacked the app.

Do you have any insight on my problem chickenbutt?

Thanks.

[chickenbutt]

I can use IDA if need be, i also have olly, immunity..etc.

The dll and the app are created with vc++ 6

Ive already unpacked the app.

Do you have any insight on my problem chickenbutt?

Thanks.

BP where the stack for each function is pushed and you have your arguments, if it's crypto data you can trace. Olly usually helps with symbols and stacks.

If the symbols are memory loaded bp+trace or inline hook and dump stacks.

Edited April 21, 2011 by chickenbutt