Nested Kernel

Abstract

Monolithic operating system designs undermine the security of computing systems by allowing single exploits anywhere in the kernel to enjoy full supervisor privilege. The nested kernel operating system architecture addresses this problem by "nesting" a small isolated kernel within a traditional monolithic kernel. The "nested kernel" interposes on all updates to virtual memory translations to assert protections on physical memory, thus significantly reducing the trusted computing base for memory access control enforcement. We incorporated the nested kernel architecture into FreeBSD on x86-64 hardware while allowing the entire operating system, including untrusted components, to operate at the highest hardware privilege level by write-protecting MMU translations and de-privileging the untrusted part of the kernel. Our implementation inherently enforces kernel code integrity while still allowing dynamically loaded kernel modules, thus defending against code injection attacks. We also demonstrate that the nested kernel architecture allows kernel developers to isolate memory in ways not possible in monolithic kernels by introducing write-mediation and write-logging services to protect critical system data structures. Performance of the nested kernel prototype shows modest overheads: <1% average for Apache and 2.7% for kernel compile. Overall, our results and experience show that the nested kernel design can be retrofitted to existing monolithic kernels, providing important security benefits.

A Hardware Platform for Ensuring OS Kernel Integrity on RISC-V

“…Then, they inserted secure monitor calls (SMCs), which are special instructions to transfer the control from the normal world to the secure world, into the kernel code and verified the integrity of the kernel in the secure world when SMCs are executed. Some approaches [3,5,6,37] hardened the kernel itself without any supports of a higher privileged layer such as hypervisor and TrustZone. Instead, to build an SEE in the kernel address space, they utilized features of the processor architecture such as WP [6] in Intel x86 and TxSZ [5] in the ARM.…”

“…Some approaches [3,5,6,37] hardened the kernel itself without any supports of a higher privileged layer such as hypervisor and TrustZone. Instead, to build an SEE in the kernel address space, they utilized features of the processor architecture such as WP [6] in Intel x86 and TxSZ [5] in the ARM. They split the kernel into the secure domain and non-secure domain, then put the higher privilege to the secure domain than the non-secure domain by assigning sensitive kernel operations such as page table and system control register management to the secure domain.…”

“…To protect the kernel against kernel-level attacks and rootkits [2], many researchers have devised mechanisms to monitor the integrity of the OS kernel in a secure execution environment that is isolated from the monitored kernel. To implement an SEE, several studies have proposed software-based techniques that mandate either the instrumentation of kernel code [3][4][5][6] or the introduction of a higher privileged layer [7,8] (i.e., hypervisor).…”

SofTEE: Software-Based Trusted Execution Environment for User Applications

“…Virtual Ghost offers better performance than a hardware virtualizationbased approach [10], but Virtual Ghost is much slower than an ARM TrustZone based approach, TrustShadow [9]. Some studies [16], [34] provide trusted kernel execution environments without special hardware features. For example, in SKEE [34], A. M. Azab et al proposed a framework to support the trusted kernel execution environment.…”

“…The first way is to put the module and the kernel in the same address space. Instead, the module is hidden from the kernel via address space randomization (ASR) [32] or protected by write protection (WP) in x86 [16]. The system with ASR is less likely to be attacked, so the module is expected to be safe, but in practice, it is not secure.…”