Thread Migration in a Replicated-Kernel OS

Abstract

Chip manufacturers continue to increase the number of cores per chip while balancing requirements for low power consumption. This drives a need for simpler cores and hardware caches. Because of these trends, the scalability of existing shared memory system software is in question. Traditional operating systems (OS) for multiprocessors are based on shared memory communication between cores and are symmetric (SMP). Contention in SMP OSes over shared data structures is increasingly significant in newer generations of many-core processors. We propose the use of the replicated-kernel OS design to improve scalability over the traditional SMP OS. Our replicated-kernel design is an extension of the concept of the multikernel. While a multikernel appears to application software as a distributed network of cooperating microkernels, we provide the appearance of a monolithic, single-system image, task-based OS in which application software is unaware of the distributed nature of the underlying OS. In this paper we tackle the problem of thread migration between kernels in a replicated-kernel OS. We focus on distributed thread group creation, context migration, and address space consistency for threads that execute on different kernels, but belong to the same distributed thread group. This concept is embodied in our prototype OS, called Popcorn Linux, which runs on multicore x86 machines and presents a Linux-like interface to application software that is indistinguishable from the SMP Linux interface. By doing this, we are able to leverage the wealth of existing Linux software for use on our platform while demonstrating the characteristics of the underlying replicatedkernel OS. We show that a replicated-kernel OS scales as well as a multikernel OS by removing the contention on shared data structures. Popcorn, Barrelfish, and SMP Linux are compared on selected benchmarks. Popcorn is shown to be competitive to SMP Linux, and up to 40% faster.

libMPNode

“…Previous works have studied migrating threads between multiple nodes. Operating systems like Kerrighed [21], K2 [19] and Popcorn Linux [15,16,25] provide a single system image between non-cache-coherent nodes interconnected via high speed links. In particular, they provide the ability to migrate threads between nodes by transplanting the thread's context.…”

“…libMPNode leverages Popcorn Linux [16], namely transparent thread migration and distributed shared memory, to allow application developers to use a well-studied and simple shared memory parallel programming model in multi-node systems. Cross-node thread migration is implemented similarly to OS continuations [15]; threads begin by making a system call into the thread migration service. The original thread is put to sleep on the source node, and a new thread is instantiated with the original thread's context on the remote node, returning to user-space to continue normal execution.…”

“…Because the DSM is implemented transparently by the OS, existing sharedmemory applications can execute across nodes unmodified. The complete details of Popcorn Linux can be found in past works [3,4,15,16,25].…”

Design methodologies for resource management of many-core embedded systems

“…The increased number of PEs on a single chip, creates new requirements regarding their inter-connection, since cache coherency scales well for a small amount of cores but becomes the bottleneck in high core-count processors [126,208]. Networks on Chip have been established as the proposed communication architecture for overcoming this bottleneck [41].…”

“…has not yet yielded the expected performance gains. One main reason lies in the scalability of traditional run-time resource management schemes [47,126]. Keeping caches coherent while managing a large number of cores requires customized approaches, especially in shared memory systems where the coreto-core communication is structured upon the underlying cache-coherency protocol [183,136].…”