Malicious software - so called malware poses a major threat to the security of computer systems. The amount and diversity of its variants render classic security defenses ineffective, such that millions of hosts in the Internet are infected with malware in the form of computer viruses, Internet worms and Trojan horses. While obfuscation and polymorphism employed by malware largely impede detection at file level, the dynamic analysis of malware binaries during run-time provides an instrument for characterizing and defending against the threat of malicious software.
In this article, we propose a framework for the automatic analysis of malware behaviour using machine learning. The framework allows for automatically identifying novel classes of malware with similar behaviour (clustering) and assigning unknown malware to these discovered classes (classification). Based on both, clustering and classification, we propose an incremental approach for behaviour-based analysis, capable of processing the behaviour of thousands of malware binaries on a daily basis. The incremental analysis significantly reduces the runtime overhead of current analysis methods, while providing accurate discovery and discrimination of novel malware variants.