Scalable address spaces using RCU balanced trees

Abstract

Software developers commonly exploit multicore processors by building multithreaded software in which all threads of an application share a single address space. This shared address space has a cost: kernel virtual memory operations such as handling soft page faults, growing the address space, mapping files, etc. can limit the scalability of these applications. In widely-used operating systems, all of these operations are synchronized by a single per-process lock. This paper contributes a new design for increasing the concurrency of kernel operations on a shared address space by exploiting read-copy-update (RCU) so that soft page faults can both run in parallel with operations that mutate the same address space and avoid contending with other page faults on shared cache lines. To enable such parallelism, this paper also introduces an RCU-based binary balanced tree for storing memory mappings. An experimental evaluation using three multithreaded applications shows performance improvements on 80 cores ranging from 1.7× to 3.4× for an implementation of this design in the Linux 2.6.37 kernel. The RCU-based binary tree enables soft page faults to run at a constant cost with an increasing number of cores, suggesting that the design will scale well beyond 80 cores.

A Comparative Analysis of RTOS and Linux Scalability on an Embedded Many-core Processor

“…Meanwhile, Linux has an extremely complex implementation for page table management with advanced features including demand paging and swapping [26]. Modifying page tables dynamically could be a bottleneck in some cases and has been actively studied and optimized over many years [27]. Consequently, if an application frequently requests the memory management services from kernel, it may have better performance on FMP than Linux.…”

RadixVM

“…The single read-write lock protects the OS index data structures that map virtual addresses to metadata for mapped memory regions, as well as invariants between the OS index data structure and the hardware page tables and TLBs. In our earlier work on the Bonsai VM system, we described a lock-free design for page faults in Linux, but that design still required the use of Linux's address space lock to serialize mmap and munmap operations [7].…”

“…Because of complex invariants in virtual memory systems, widely used kernels, such as Linux and FreeBSD, have a single lock per shared address space. Recent research shows how to run a page fault handler in parallel with mmap and munmap calls in the same address space [7], but still serializes mmap and munmap, which applications use to allocate and return memory to the operating system. However, if the mmap and munmap involve non-overlapping memory regions in the shared address space, as is the case in memory allocation and freeing, then in principle these calls should be perfectly parallelizable because they operate on Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page.…”

“…We have implemented RadixVM in a new research kernel derived from xv6 [9] because changing the Linux virtual memory implementation is very labor intensive owing to its overall complexity and how interconnected it is with other kernel subsystems [7]. An experimental evaluation on an 80 core machine with microbenchmarks that represent common sharing patterns in multithreaded applications and a parallel MapReduce library from MOSBENCH [6] show that the RadixVM design scales for non-overlapping mmaps and munmaps.…”

Efficient Fetch-and-Increment

“…A persistent augmented balanced binary tree [9,8], such as a red-black tree or an AVL tree, is used to implement T v . Each pair in the sequence I(v) is represented by a node containing the side and the size of the pair.…”