Yuval Elovici

Senior Academic

Trusted system-calls analysis methodology aimed at detection of compromised virtual machines using sequential mining

Nir Nissim, Yuval Lapidot, Aviad Cohen, Yuval Elovici

Most organizations today employ cloud-computing environments and virtualization technology; Due to their prevalence and importance in providing services to the entire organization, virtual-servers are constantly targeted by cyber-attacks, and specifically by malware. Existing solutions, consisting of the widely-used antivirus (AV) software, fail to detect newly created and unknown-malware; moreover, by the time the AV is updated, the organization has already been attacked. In this paper, we present a during run-time analysis methodology for a trusted detection of unknown malware on virtual machines (VMs). We conducted trusted analysis of volatile memory dumps taken from a VM and focused on analyzing their system-calls using a sequential-mining-method. We leveraged the most informative system-calls by machine-learning algorithms for the efficient detection of malware in widely used VMs within organizations (i.e. IIS and Email server). We evaluated our methodology in a comprehensive set of experiments over a collections of real-world, advanced, and notorious malware (both ransomware and RAT), and legitimate programs. The results show that our suggested methodology is able to detect the presence of unknown malware, in an average of 97.9% TPR and 0% FPR. Such results and capabilities can form the ground for the development of practical detection-tools for both corporates and companies.

Publication language English
Pages 147-175
Journal Knowledge-Based Systems
Volume 153
Publication status Published - 01.08.2018

Keywords

Machine learning
Malware detection
Memory dump
Private cloud
Ransomware
Remote access Trojan
Sequential mining
Virtual machine
Virtual server
Volatile memory

ASJC Scopus subject areas

Management Information Systems
Software
Information Systems and Management
Artificial Intelligence