Detecting Kernel-Level Rootkits Using Data Structure Invariants

Rootkits affect system security by modifying kernel data structures to achieve a variety of malicious goals. While early rootkits modified control data structures, such as the system call table and values of function pointers, recent work has demonstrated rootkits that maliciously modify non control data. Most prior techniques for root kit detection have focused solely on detecting control data modifications and, therefore, fail to detect such rootkits. This paper presents a novel technique to detect rootkits that modify both control and noncontrol data.

The main idea is to externally observe the execution of the kernel during an inference phase and hypothesize invariants on kernel data structures. A root kit detection phase uses these invariants as specifications of data structure integrity. During this phase, violation of invariants indicates an infection. We have implemented Gibraltar, a prototype tool that infers kernel data structure invariants and uses them to detect rootkits. Experiments show that Gibraltar can effectively detect previously known rootkits, including those that modify non-control data structures.

Existing System:

Recent studies have shown a phenomenal increase in malware that use stealth techniques commonly employed by rootkits. The increase in the number and complexity of rootkits can be attributed to the large and complex attack surface that the kernel presents. The kernel manages several hundred heterogeneous data structures, most of which are critical to its correct operation. A rootkit can subvert kernel integrity by subtly modifying any of these data structures. In particular, kernel data structures that hold control data, such as the system call table, jump tables, and function pointers, have long been a popular target for attack by rootkits. However, recent work has demonstrated rootkits that achieve a variety of malicious goals by modifying non-control data in the kernel.

Proposed System:

We propose a novel approach that automatically generates kernel data structure integrity specifications. In our approach, these integrity specifications take the form of data structure invariants—properties that must hold for the lifetime of a data structure. The key idea is to monitor the values of kernel data structures during an inference phase in order to hypothesize invariants that are satisfied by these data structures. These invariants can encompass both control and noncontrol data structures. For example, an invariant could state that the values of elements of the system call table are a constant (an example of a control data invariant). Similarly, an invariant could state that all the elements of the running-tasks linked list These invariants are then checked during a rootkit detection phase, in which violation of an invariant is assumed to indicate the presence of a rootkit.


  • The Page Fetcher
  • The Data Structure Extractor
  • The Invariant Generator
  • The Monitor

Tools Used:

Front End : C#.Net
Back End : SQL Server 2005