How do AV systems find packed Malicious Software

[GoJonnyGo]

Hi there,

i am wondering, how antivirus systems can find viruses in packed software. Do they know every unpacking routine and first look at with with protector it is packed and unpack it then to perform a search or do they wait till the exe unpacked itself and is on oep or how does this happen?

[cyb3rl0rd1867]

I'm definitely not an expert in the field, but (I believe) they can unpack some packers. Also keep in mind that they can't detect everything, because they can't unpack everything. If you would write your own packer, chances are that it won't be detected, at least not for a little while. Again, this is to the best of my (limited) understanding. Hope this helps!

[NullPointerException]

depending on which antivirus you are dealing: some are honest because they first detect the packer,unpack it and scans for known virus signature. others are lazy: scan for the packers and delete files just because they use that packer (thus producing a lot of false positive)

let's say we have an X protector (very difficult to unpack): the honest av will unpack it and check the real code while the lazy will say "it's too difficult hence it must have some evil code inside".

the end user cant understand the difference between those two and eventually would say "the av which detects more viruses (even false ones) is the best". So he pays for an av coded by lazy developers that unjustly sells more than good ones.

of course commercial protectors care about this fact and they suggest av developers how to check code inside. if they dont do so av will blacklist every app which uses that particular protector (that's why, for example, themida dismissed its ring-0 protection or vmprotect asked av to contact them).

lol i know that this may not interest you...

back to your question: if you are sure that your av UNPACKS file in memory you can break when it reach the oep of your app but it is very hard and not always possible.

As for detection each av has its private signature: it scan for protectors that are mostly used for virus or scans for byte patterns in unpacked files.

[kimbo]

i think they have generic unpacker or emulator, then they scan the unpacked bytes.

[cipher]

the anti virus uses sandboxie technique to analyse the packed application , one such av is kaspersky which is much more intelligent in handling packed applications.

[trimitu]

the anti virus uses sandboxie technique to analyse the packed application , one such av is kaspersky which is much more intelligent in handling packed applications.

I think the AV software uses the entropy to detect a packed software!

[diablo2oo2]

yes i also think they use sandbox technique. they just run the file. it unpack itself before its executed. then the AV scanns the running file.

but its easy to fool the AV. you can crypt every procedure and just decrypt the procedure/function when it needs to be executed. after the procedure/function was executed you crypt it again.

Edited March 20, 2011 by diablo2oo2

[Blue]

yes i also think they use sandbox technique. they just run the file. it unpack itself before its executed. then the AV scanns the running file.

but its easy to fool the AV. you can crypt every procedure and just decrypt the procedure/function when it needs to be executed. after the procedure/function was executed you crypt it again.

Thanks for your suggestions diablo2oo2,

But the way of fooling A.V. you are talking about is nearly impossible today. I think that you are talking about some FUD crypting and decrypting routine on runtime. The A.V. coming today are more advanced but yet back by one step from latest malwares

As AVAST 5 PRO is a best example, use avast and then try to fool it by using any runtime routine.And Bam!!! the exe got detected

As far as unpacking for antivirus is very limited one. Because making an unpacking routine of all packers will make AV core heavy.Also as we know that there are also some custom packers so it is impossible to detect that packers and then unpack it. If AV detects a packer and have its unpacking routine then it's fine, if a new packer got detected then AV uses a general sandbox system like AVAST/NORTON/F-SECURE and kaspersky uses, in that sandbox the exe will be tested and if at the runtime any single line of malware got detected then the whole exe will be flagged.

But all of this process will be done in some top av's only like Quick heal,Kaspersky,Norton,AVAST and some more.

Hope this will help and sorry for my bad english

[Malaya2100]

Hey Blue, not so bad, it's understood.

[CodeExplorer]

AVs are quite stupid:

after is set name of .text section to 00 and characteristics to 0x40000040

the file was catloged as suspicious.

[0xFF]

Don't forget that AV companies have a little research room where they have hardcore reversers where their job is to sit and disassemble over 500 PE every day and report it to the coders or w.e to update the definitions...

[chickenbutt]

Don't forget that AV companies have a little research room where they have hardcore reversers where their job is to sit and disassemble over 500 PE every day and report it to the coders or w.e to update the definitions...

Last I checked they all had automated frameworks for samples, except special cases like TDL4 or Rustock, where they buy or rip info from independent and 3rd party researchers. Kaspersky has some good RCE people that they only use for management and tool-dev, I've actually seen RCE work being contracted out to freelancers by some big names before.

With the exception of managed-code PE obfuscators, all malware binaries have to have stubs for unpacking/decrypting.

Edited May 28, 2011 by chickenbutt