Automatic Inference and Enforcement of Kernel Data Structure Invariants

Abstract

Kernel-level rootkits affect system security by modifying key kernel data structures to achieve a variety of malicious goals. While early rootkits modified control data structures, such as the system call table and values of function pointers, recent work has demonstrated rootkits that maliciously modify non-control data. Prior techniques for rootkit detection fail to identify such rootkits either because they focus solely on detecting control data modifications or because they require elaborate, manually-supplied specifications to detect modifications of non-control data.This paper presents a novel rootkit detection technique that automatically detects rootkits that modify both control and non-control data. The key idea is to externally observe the execution of the kernel during a training period and hypothesize invariants on kernel data structures. These invariants are used as specifications of data structure integrity during an enforcement phase; violation of these invariants indicates the presence of a rootkit.We present the design and implementation of Gibraltar, a tool that uses the above approach to infer and enforce invariants. In our experiments, we found that Gibraltar can detect rootkits that modify both control and non-control data structures, and that its false positive rate and monitoring overheads are negligible.

HyperCheck: A Hardware-Assisted Integrity Monitor

“…Furthermore, there is a plethora of research aimed towards protecting the Linux kernel [2,22,16,38,21,36,31]. Baliga [2] et al use a PCI device to acquire the memory and automatically derive the kernel invariance.…”

“…Baliga [2] et al use a PCI device to acquire the memory and automatically derive the kernel invariance. Currently, we discover the kernel invariance manually but we could employ their techniques directly and without modifications.…”

“…Protecting software from integrity attacks using hardware-assisted techniques is not new: researchers used a special-purpose PCI device to acquire the physical memory either for rootkit detection [30,2] or for forensic purpose [8] in the past. The closest system to our work is Copilot [30].…”

“…However, having a machine equipped with trusted boot can prevent attacks against HyperCheck that simulate a hardware reset. 2 …”

SoK: Introspections on Trust and the Semantic Gap

“…3) Dynamic Learning: Rather than identifying code invariants from kernel source code, VMI based on dynamic analysis learns data structure invariants based on observing an OS instance [24,41,64].…”