PE file scanner with collaboration system

[Aguila]

I have an idea for a new project. Please tell me what you think about it.

You all know PEiD, DIE, exeinfope, ProtectionID, Fast Scanner, DNiD, and so on. File scanner help to detect a protection system. But they all have a problem: no updates, not open source.

Here is probably a solution to create the best tool.

Client:

A C/C++ program with easy GUI similiar to PEiD, without any crap features you don't ever need (like you can find in PID). The signatures are stored in a real database (not some stupid txt file!): SQLite. The program can download updates from a web portal. Should support every file type.

Web portal:

A collaboration system. Every user can add signatures to the database. If only a few user add once a week a signature, the database will grow and the tool is always up to date. There is a easy login system with rating system to ensure some quality. The web portal should be coded in PHP with a mysql database, so it can be hosted everywhere even with free hosters. I already created a simple database draft (see attachment). Doesnt need much explaination I think. Signatures FileType can be ANY_FILE, DLL_32BIT, ARM, DOTNET, EXE_32BIT, and so on. Just a simple web portal with CRUD operations.

Of course everything should be open-source. I can help with the client, but I can't code the web portal (lack of php/html/css skills). What do you think?

Edited April 1, 2012 by Aguila

[diablo2oo2]

i like your project idea. lets do it

[mrexodia]

Seems like a good project!

Problem with such auto-update web portals is that it's gonna need lots of bandwidth (if the project is a success ofcourse)

I can help a bit with MySQl if you like (though my skill is minimal)

[Aguila]

I don't think bandwith will be a problem. There are plenty free hosters that offer "unlimited traffic". You can set up duplicate websites without any problems. The client should be able to check different websites for signature updates. Can be easily done by checking the "CreationDate" and "ModificationDate" property in the database.

Thanks for offering help. I hope somebody can do this web portal.

I detected a flaw in the database schema, here is a fix.

[diablo2oo2]

Hey Agila,

do you know a good database modeling tool (maybe multiplatform)? Another thing, most free hoster do not allow database connections from outside, only localhost.

[evlncrn8]

you know, i was going to help, until you decided to critique other tools... wasnt needed, and its not all signatures, signatures are weak and often prone to false positives...

also there was an open source scanner called burnout... go google, next time do some research instead of making pretty little designs

/>http://burnout.sourceforge.net/

Edited April 1, 2012 by evlncrn8

[Killboy]

So criticism is bad now?

Edited April 1, 2012 by Killboy

[Aguila]

do you know a good database modeling tool (maybe multiplatform)? Another thing, most free hoster do not allow database connections from outside, only localhost.

Yes there is http://www.mysql.de/...ucts/workbench/ but I used Visio to do this quick database schema. Doesn't matter, because I don't want to access the database directly. PHP must offer an interface.

you know, i was going to help, until you decided to critique other tools...

What did I say? PID has too many useless features? no update? not open source? well it is true. But they are still great tools.

also there was an open source scanner called burnout... go google, next time do some research instead of making pretty little designs

are you kidding? Did you look at the source code of this scanner?

have a look:

	    AddPercentEXE(BurnOut.AddScaninEXE)
	    position = InStr(FileContent, "AddD" & xxx(3) & xxx(0) & xxx(0) & xxx(0), CompareMethod.Binary)
	    If position Then
		    position -= 1
		    Return "SecuROM " & GetSecuROM4Version(file, position)
	    End If
	    AddPercentEXE(BurnOut.AddScaninEXE)
	    position = InStr(FileContent, xxx(&HCA) & xxx(&HDD) & xxx(&HDD) & xxx(&HAC) & xxx(&H3), CompareMethod.Binary)
	    If position Then
		    position -= 1
		    Return "SecuROM " & GetSecuROM4and5Version(file, position)
	    End If
	    AddPercentEXE(BurnOut.AddScaninEXE)
	    If BurnOut.PrefixInStr(FileContent, ".securom", xxx(&HE0), xxx(&HC0), CompareMethod.Binary) Then ' SecuROM new 7
		    Return "SecuROM " & GetSecuROM7Version(file)
	    End If

[evlncrn8]

of course i looked at it, its crap.. and some detections are a lot tougher than others to detect. especially extracting version numbers, and often require a lot of research...

as for criticism, theres a time and place, it simply wasnt needed

updates, pid is actually updated, cdkiller is however awol, and handled all the beta testers and releases.. i said in another thread i wanted to do a release but without the access to the ftp its a bit tricky, i will msg empire and see what can be done but cant promise anything...

as for pid going open source, it wont happen, its a nice concept, but with it comes a problem, that being that some companies who would rather their protection is not detected will be able to see how the detection is done... much in the same way some tools exist but no source

Edited April 1, 2012 by evlncrn8

[Aguila]

as for pid going open source, it wont happen, its a nice concept, but with it comes a problem, that being that some companies who would rather their protection is not detected will be able to see how the detection is done... much in the same way some tools exist but no source

so you are saying that e.g. securom or themida developers cannot reverse engineer PID? Does your tool have some superhuman protection or what?

I agree that PID is a different tool, not compareable to the other tools. It is more accurate because you are not relying only on signatures. But still, a tool without updates is not "the best".

[evlncrn8]

no, try thinking... im saying it makes their task easier, no source = they have a bit more work to do....

for example, why is the rdg packer detector (among others) crypted / anti debug? maybe to prevent other detectors ripping off the detection method (we also found this happens with pid too, not as much now as before though)... either way, it shows that people want to protect their work and research, which is why open source for such things probably wont work.... even look at the public info out there for commercial prots, its pretty out of date, locked down in teams and some communities... i know stuff on prots i researched and can do no-cd / bypass license checks and such... which took a lot of time, and i wouldnt hand that over or make it public (wheres the benefit?), and im definately not alone in that sentiment

[Departure]

Too bad its in C/C++ because there is an open source Delphi project going on, and in fact its current state is just as good as PeID. Will this new project support plugins? I like the online database idea, but I agree with Mr Exiodia it will chew a good amount of band width, I have done a similar database before for an application that was only used by around 10,000 to 15,000 people tops for coloring Paltalk nicknames, it chewed over 2 terabytes per month of bandwidth, the database at that time was a simple text file on line that was updated through a php script, the text file was around 20kb or less, So I could only imagine what a common 600kb text file for signatures would use in bandwidth each month with alot more users.

[Teddy Rogers]

Did the entire database get downloaded on each update or did it only retrieve actual updates? Possibly database structure and use would be good to think about logically to better optimise and use it...

Ted.

[Aguila]

There must be some interface offered by PHP. You are right there is no need to download the database each time. But as I said, there can be multiple websites and the tool should check them all. I really think that bandwith is the smallest problem.

Maybe something like this:

/download.php?lastupdate=<date>

/download.php?getgroup=<groupname>

...

The downloaded file should be XML formatted.

Too bad its in C/C++ because there is an open source Delphi project going on, and in fact its current state is just as good as PeID.

Never heard of it. But you are free to use this concept to implement it in your tool. I don't know delphi, so I can't help there.

Will this new project support plugins?

Yes it should support the PEiD plugins. e.g. KANAL is still a often used plugin.

@evlncrn8

I don't think "protection from rippers" is a valid reason. Because there are only a few people that use such tools and they know if someone ripped it and released a similiar tool. Also rippers lack some programming knowledge and stuff. They can't create better tools, so the ripped stuff will disappear soon...

[Loki]

Happy to help with the web portal part..... I'm no world beating php genius, but I have experience doing this stuff.

If its hosted in a few places then it should be easy to propagate the data too.

Your schema should work quite nicely. Not sure what IsBanned is on the signitures though? Some sort of validity check?

On creation, if ModificationDate is set to CreationDate then a simple SQL query like you describe using lastupdated should work nicely

[Aguila]

Nice Loki, thanks for your help.

Some more details:

There are 3 roles in the web portal: admin, "trusted user" and "normal user".

admin: can edit, delete and post everything + edit the users

trusted user: can edit, delete and post everything he want

normal user: can post stuff, edit his own signatures, but new signatures need to be approved by a trusted user or admin. I think this is necessary to stop spam or wrong signatures.

Not sure what IsBanned is on the signitures though? Some sort of validity check?

Maybe IsApproved is better. We need something to control which signature is distributed to the clients.

On creation, if ModificationDate is set to CreationDate then a simple SQL query like you describe using lastupdated should work nicely

yes. I added the ModificationDate to the other 2 objects too. I think this will work out.

But there is one difficult thing: the timezones. Every date must be in one specific timezone. It should be possible to set up the web portal in different countries. Also the client must convert the date into one specific timezone. Or there need to be an extra interface to ask the web portal which date now is. Something like /getCurrentDate.php. Then the client can calculate the difference and ask for the correct modification date.

Edited April 2, 2012 by Aguila

[deepzero]

modern detectors, such as pid, go beyond simple pattern matching. What would be the approach there?

Maybe it would be useful to assign a "trust" value to a signature, depending on how "trusted" the user is (various trust levels would be required, maybe based on supplied sigs, quality, etc)?

[Aguila]

modern detectors, such as pid, go beyond simple pattern matching. What would be the approach there?

I think pattern matching is still a working technique. Because of your point I added an extra attribute to the signature table: ScanType. ScanType can be FILESCAN, MEMORYSCAN or BOTH. Most protectors can be easily identified in a running process. Maybe detection for compilers can be build in the client. And there is still plugin support to extend the detection rate.

Maybe it would be useful to assign a "trust" value to a signature, depending on how "trusted" the user is (various trust levels would be required, maybe based on supplied sigs, quality, etc)?

yes thx this is a good idea. I changed Rating to TrustLevel.

[Loki]

Timezone shouldnt be an issue as I'm assuming it isnt viewable.

Can use a unix timestamp from the server and pass it back with the updates to be saved by the client as "lastupdated" (to be returned on next update)

Keeps it simple.

[diablo2oo2]

The pattern should support regular expressions (PCRE) and not just simple wildcard rules.

[s0me0ne]

@Departure

whats this open source Delphi project called?

[Departure]

PE Sniffer, Actually PE Tools seem to use this open source in there project with a couple GUI mods

[Aguila]

I did a quick google search and you mean this?

//  * Unit Name : PE_Sniffer
//  * Purpose   : Óòèëèòà äëÿ îòîáðàæåíèÿ êðàòêîé èíôîðìàöèè î PE File
//  * Author	: Dik0n
//  * Copyright : © 2009 - 2011
//  * Home Page : http://my-soft.ucoz.ru
//  * E-Mail	: dima_yar@mail.ru
//  * Version   : 0.1

This is really a PEiD clone without any advantages. PEiD is still better than this tool. Ok it is open-source but there is no "value-added" stuff.

[mrexodia]

@offtopic: Loved the fact everything is english except "Purpose"

Edited April 16, 2012 by Mr. eXoDia

[mentebinaria]

Hey, I have the same feeling about protection detection tools and general PE analysis and editing tools. So, I started a project called pev. I wrote my own PE library (called libpe) in C and some tools that use it, including a PEiD scanning clone called pepack (that uses userdb.txt database). We can improve pepack to do better detections, use SQLite, support updates (we have full support from SourceForge since it's an open source project, but we also have a VPS).

All tools of pev toolkit are designed to be fully scriptable, but we can write a GUI with GTK+, for example. It's multiplatform, like pev is.

What do you think?