NC State Research Shows Way To Block Stealthy Malware Attacks

The spread of malicious software, also known as malware or computer viruses, is a growing problem that can lead to crashed computer systems, stolen personal information, and billions of dollars in lost productivity every year. One of the most insidious types of malware is a “rootkit,” which can effectively hide the presence of other spyware or viruses from the user – allowing third parties to steal information from your computer without your knowledge. But now researchers from North Carolina State University have devised a new way to block rootkits and prevent them from taking over your computer systems.

To give some idea of the scale of the computer malware problem, a recent Internet security threat report showed a 1,000 percent increase in the number of new malware signatures extracted from the in-the-wild malware programs found from 2006 to 2008. Of these malware programs, “rootkits are one of the stealthiest,” says Dr. Xuxian Jiang, assistant professor of computer science at NC State and a co-author of the research. “Hackers can use rootkits to install and hide spyware or other programs. When you start your machine, everything seems normal but, unfortunately, you’ve been compromised.”

Rootkits typically work by hijacking a number of “hooks,” or control data, in a computer’s operating system. “By taking control of these hooks, the rootkit can intercept and manipulate the computer system’s data at will,” Jiang says, “essentially letting the user see only what it wants the user to see.” As a result, the rootkit can make itself invisible to the computer user and any antivirus software. Furthermore, the rootkit can install additional malware, such as programs designed to steal personal information, and make them invisible as well.

In order to prevent a rootkit from insinuating itself into an operating system, Jiang and the other researchers determined that all of an operating system’s hooks need to be protected. “The challenging part is that an operating system may have tens of thousands of hooks – any of which could potentially be exploited for a rootkit’s purposes,” Jiang says, “Worse, those hooks might be spread throughout a system. Our research leads to a new way that can protect all the hooks in an efficient way, by moving them to a centralized place and thus making them easier to manage and harder to subvert.”

Jiang explains that by placing all of the hooks in one place, researchers were able to simply leverage hardware-based memory protection, which is now commonplace, to prevent hooks from being hijacked. Essentially, they were able to put hardware in place to ensure that a rootkit cannot modify any hooks without approval from the user.

The research, “Countering Kernel Rootkits with Lightweight Hook Protection,” will be presented at the 16th ACM Conference on Computer and Communications Security in Chicago, Nov. 12. The study’s co-authors are Jiang, Dr. Peng Ning, associate professor of computer science at NC State, NC State Ph.D. student Zhi Wang and Weidong Cui of Microsoft Research.

-shipman-

Note to Editors: The presentation abstract follows.

“Countering Kernel Rootkits with Lightweight Hook Protection”

Authors: Zhi Wang, Xuxian Jiang, Peng Ning, North Carolina State University; Weidong Cui, Microsoft Research

Presented: Nov. 12, 2009, at the 16th ACM Conference on Computer and Communications Security in Chicago.

Abstract: Kernel rootkits have posed serious security threats due to their stealthy manner. To hide their presence and activities, many rootkits hijack control flows by modifying control data or hooks in the kernel space. A critical step towards eliminating rootkits is to protect such hooks from being hijacked. However, it remains a challenge because there exist a large number of widely-scattered kernel hooks and many of them could be dynamically allocated from kernel heap and co-located together with other kernel data. In addition, there is a lack of flexible commodity hardware support, leading to the so-called protection granularity gap – kernel hook protection requires byte-level granularity but commodity hardware only provides page-level protection. To address the above challenges, in this paper, we present Hook-Safe, a hypervisor-based lightweight system that can protect thousands of kernel hooks in a guest OS from being hijacked. One key observation behind our approach is that a kernel hook, once initialized, may be frequently “read”-accessed, but rarely “write”-accessed. As such, we can relocate those kernel hooks to a dedicated page-aligned memory space and then regulate accesses to them with hardware-based page-level protection. We have developed a prototype of HookSafe and used it to protect more than 5,900 kernel hooks in a Linux guest. Our experiments with nine real-world rootkits show that HookSafe can effectively defeat their attempts to hijack kernel hooks. We also show that HookSafe achieves such a large-scale protection with a small overhead (e.g., around 6% slowdown in performance benchmarks).