Scalable address spaces using RCU balanced trees

Abstract

Software developers commonly exploit multicore processors by building multithreaded software in which all threads of an application share a single address space. This shared address space has a cost: kernel virtual memory operations such as handling soft page faults, growing the address space, mapping files, etc. can limit the scalability of these applications. In widely-used operating systems, all of these operations are synchronized by a single per-process lock. This paper contributes a new design for increasing the concurrency of kernel operations on a shared address space by exploiting read-copy-update (RCU) so that soft page faults can both run in parallel with operations that mutate the same address space and avoid contending with other page faults on shared cache lines. To enable such parallelism, this paper also introduces an RCU-based binary balanced tree for storing memory mappings. An experimental evaluation using three multithreaded applications shows performance improvements on 80 cores ranging from 1.7× to 3.4× for an implementation of this design in the Linux 2.6.37 kernel. The RCU-based binary tree enables soft page faults to run at a constant cost with an increasing number of cores, suggesting that the design will scale well beyond 80 cores.

Parallelizing live migration of virtual machines

“…Actually, this is not a specific problem to PMigrate, but a general problem to most operating systems that use a per-process lock to serialize concurrent mutation to an address space. Though Clements et al [14] have demonstrated the effectiveness in parallelizing the process of read accesses to address space (i.e., page faults) with write accesses (e.g., mmap) using a RCU balanced tree, the mutation to the address space like mmap/munmap still requires acquiring the mmap sem in write mode. Hence, there is still only one mmap/munmap operation can proceed at a time.…”

“…In practice, there are a number of memory states that should be maintained consistently for an OS kernel. Further, current Linux kernel still uses a red/black tree like data structure to maintain both virtual memory areas and memory states, completely replacing it into a new data structure will be very resource-intensive due to a number of other correlated data structures such as reverse map and mmap cache [14]. As a result, PMigrate currently only uses range lock to parallelize the time-consuming parts inside the mmap/munmap/foreign map calls by removing them from the critical sections protected by the mmap mem and protecting them by the Range Locks, while still leaving the original data structure intact.…”

“…Scaling Operating Systems: There have been a number of studies on the scalability of commodity operating systems [10,14]. Among them, the concurrent address space using RCU balanced tree [14] is the closest one.…”

“…Among them, the concurrent address space using RCU balanced tree [14] is the closest one. However, the RCU balance tree focuses on parallelizing the process of read accesses to an address space (e.g., page faults) and mutation to the address space (e.g., mmaps), while the Range Lock abstraction parallelizes the mutation to an address space.…”

A scalable lock manager for multicores

“…A few years ago, studies [11,17] showed that many opensource and commercial database engines exhibit limited multi-core scalability, and even performance collapse, due to bottlenecks in their lock, log or transaction manager. This has triggered significant research activity [3,5,12,[14][15][16][17]19] aimed at reducing lock or latch contention.…”