Skip to main content

New Research Improves Ability To Detect Malware in Cloud-Computing Systems

Researchers from North Carolina State University have developed new software that offers significantly enhanced security for cloud-computing systems. The software is much better at detecting viruses or other malware in the “hypervisors” that are critical to cloud computing, and does so without alerting the malware that it is being examined.

Cloud computing is being hailed as a flexible, affordable way of offering computer resources to consumers. Under the cloud-computing paradigm, the computational power and storage of multiple computers is pooled, and can be shared by multiple users. But concerns exist about hackers finding ways to insert malware into cloud computing systems. A new program called HyperSentry, developed by researchers at NC State and IBM, should help allay those fears.

HyperSentry is security software that focuses on protecting hypervisors in virtual computing clouds. Hypervisors are programs that create the virtual workspace that allows different operating systems to run in isolation from one another – even though each of these systems is using computing power and storage capability on the same computer.

Specifically, HyperSentry enables cloud administrators to measure the integrity of hypervisors in run time – meaning that the administrators can check to see whether a hypervisor has been breached by a third party, while the hypervisor is operating.

“The concern is that an attacker could compromise a hypervisor, giving them control of the cloud,” says Dr. Peng Ning, professor of computer science at NC State and co-author of a paper describing the research. If a hypervisor is compromised, the attacker could do almost anything: access users’ sensitive information; use the cloud’s computing resources to attack other Internet entities; spread malware; etc.

“HyperSentry solves two problems,” Ning says. “It measures hypervisor integrity in a stealthy way, and it does so in the context of the hypervisor.” Context is important, Ning explains. To effectively identify hypervisor problems you need to look at the hypervisor program memory and the registers inside the central processing units (CPUs) that are actually running the program. (The registers are the internal memory of CPUs.) This is important because intelligent malware can conceal itself from security programs that look only at the memory where the hypervisor is supposed to be located – they can effectively make themselves invisible to such security programs by modifying certain registers of the CPU and thus relocating the infected hypervisor elsewhere. By ensuring in-context measurement, HyperSentry can successfully track where the infected hypervisor is actually located and thus defeat such intelligent malware.

The fact that HyperSentry can check the integrity of a hypervisor in a stealthy way – checking the hypervisor without the hypervisor being aware of it – is important too. If a hypervisor is aware that it is being scrutinized, and has already been compromised, it can notify the malware. The malware, once alerted, can then restore the hypervisor to its normal state in order to avoid detection. Then the malware effectively hides until the security check is over.

Once a compromised hypervisor has been detected, a cloud administrator can take action to respond to the compromise, such as shutting down the computer, performing additional investigations to identify the scope of the problem and limiting how far the damage can spread.

The research is being presented Oct. 5 at the 17th ACM Conference on Computer and Communications Security in Chicago, Ill. The research was a part of the thesis work of NC State Ph.D. student Ahmed Azab, and was co-authored by Ning; NC State Ph.D. student Zhi Wang; Dr. Xuxian Jiang, an assistant professor of computer science at NC State; and Dr. Xiaolan Zhang and Nathan Skalsky of IBM. The work was done with funding from the U.S. Army Research Office, the National Science Foundation and IBM.

NC State’s computer science department is part of the university’s College of Engineering.

-shipman-

Note to editors: The study abstract follows.

“HyperSentry: Enabling Stealthy In-context Measurement of Hypervisor Integrity”

Authors: Ahmed M. Azab, Peng Ning, Zhi Wang, Xuxian Jiang, North Carolina State University; Xiaolan Zhang, IBM T. J. Watson Research Center; Nathan C. Skalsky, IBM Systems & Technology Group

Presented: Oct. 5, 2010, at the 17th ACM Conference on Computer and Communications Security in Chicago, Ill.

Abstract: This paper presents HyperSentry, a novel framework to enable integrity measurement of a running hypervisor (or any other highest privileged software layer on a system). Unlike existing solutions for protecting privileged software, HyperSentry does not introduce a higher privileged software layer below the integrity measurement target, which could start another race with malicious attackers in obtaining the highest privilege in the system. Instead, HyperSentry introduces a software component that is properly isolated from the hypervisor to enable stealthy and in-context measurement of the runtime integrity of the hypervisor. While stealthiness is necessary to ensure that a compromised hypervisor does not have a chance to hide the attack traces upon detecting an up-coming measurement, in-context measurement is necessary to retrieve all the needed inputs for a successful integrity measurement. HyperSentry uses an out-of-band channel (e.g., Intelligent Platform Management Interface (IPMI), which is commonly available on server platforms) to trigger the stealthy measurement, and adopts the System Management Mode (SMM) to protect its base code and critical data. A key contribution of HyperSentry is the set of novel techniques that overcome SMM’s limitation, providing an integrity measurement agent with (1) the same contextual information available to the hypervisor, (2) completely protected execution, and (3) attestation to its output. To evaluate HyperSentry, we implement a prototype of the framework along with an integrity measurement agent for the Xen hypervisor. Our experimental evaluation shows that HyperSentry is a lowoverhead practical solution for real world systems.