rOOM: A Rust-Based Linux Out of Memory Kernel Component

Abstract

The Linux kernel has been applied in various securitysensitive fields, so ensuring its security is crucial. Vulnerabilities in the Linux kernel are usually caused by undefined behaviors of the C programming language, the most threatening of which are memory safety vulnerabilities. Both the software-based and hardware approaches to memory safety have disadvantages of poor performance, false positives, and poor compatibility. This paper explores the feasibility of using the safe programming language Rust to reconstruct a Linux kernel component and open-source the component's code. We leverage the Rust FFI mechanism to design a safe foreign interface layer to enable the reconstructed component to invoke other Linux functionalities, and then use Rust to reconstruct the component, during which we leverage Rust's type-safety and ownership mechanisms to improve its security, and finally export the C interface of the component to enable the invocation by the Linux kernel. The performance and memory overhead of the reconstructed component, referred to as "rOOM", were evaluated, revealing a performance overhead of 8.9% in kernel mode, 5% in user mode, 3% in real time, and a memory overhead of 0.06%. These results suggest that it is possible to develop key components of the Linux kernel using Rust in terms of functionality, performance, and memory overhead.