A recompiler

[xSRTsect]

I have  long term project, unfortunatly the time I can afford to RCE is not as much as I would like it to be. It would consist of something like a compiler with JIT optimizer such that the new generated assembly code (and executable) would be slightly less obfuscted, the original recompiler idea would have some features like:

 

-> constant folding

-> codeflow rebounder

-> some features that would allow the comunity to add more features to it (plugins or whatever)

 

 

Please dont be fornicating retarded, I have already tried what the internet has to offer in this field and it DOES NOT WORK when you try serious stuff. The best option may be IDA's deobfuscator plugin which crashes for a large ammount of code. In your opinion how far can code optimization/deobfuscation go?

[metr0]

What would "codeflow rebounding" be? Anyway, there have been attempts that apply compiler optimizations for deobfuscation and they seem to work well enough for the targets they were written for. DeCV by pa_kt and that Oreans deobfu over at Woodmanns (by orange-bat, perhaps?) come to my mind. More ambitious projects try to make use of well-known IRs like LLVM or VEX, you might want to check them out.

 

There are already loads of different static binary analysis frameworks out there, don't add yet another one. None of the above mentioned tools work on everything you throw at them -- if that's what you mean when talking about serious stuff. But focus your effort on fixing an existing tool anyway.

[xSRTsect]

Codeflow rebounder I mean to try to robound the broken code. Its very tipicall to see instructions like:

 

mov eax, 0xbadc0de

xor eax, ebx

jmp @next_chart

 

and next_chart looks like:

 

mov eax, 2

xor eax, 2

jmp @next_...

 

DeCV works for devirtualization for that particular VM. What I am trying to accomplish here is a generic deobfuscation (notice that I am just trying to make the code a little bit more readable) tool that might work better than the currently available tools. I.E: work for larger amount of code lines in a decent time. My experience with IDA deobfuscation plugin is that it doesn't work when the code is too large.

 

EDIT: And also let me just add this detail: I have worked with LLVM before for this purpose. The problem is that its internal IR is very difficult to deal with if we wish to represent a stack based machine (like x86). I used it for some "practical" purposes but its very limitative. I really think it would be better to start by 0 and build a whole new compiler. But the more feedback I have from other people, the better this will be.

 

So, bottom line, you think this is a bad idea?

Edited May 21, 2014 by xSRTsect

[kao]

In general, it's definitely a good idea.

I see 2 problems here:

1) as metr0 said - reinventing the wheel (IR/compiler/optimizer) is not a good idea. You'll spend lots of time on it, it may or may not work, and then everyone who wants to contribute will have to learn this new complex system before doing anything.

2) community needs the initial working prototype to build upon. If you look at the most successful open-source projects here, both x64_dbg and scyllahide were initially made by one person, then presented to community, and then the community started helping. If you don't have time/resources to make the first version, I doubt that the project will succeed here.

P.S. x86 is not a stack-based machine.

[xSRTsect]

I would do the first version yes, I was just trying to understand what others think. But like I said, it may take some time untill I have the time to work in it and I may realise in the meantime that it may be computationally untreatable (?) I don't know lets just see how far the rabbit hole goes.

Edited May 22, 2014 by xSRTsect