The Nornir run-time system for parallel programs using Kahn process networks on multi-core machines—a flexible alternative to MapReduce

Abstract

Even though shared-memory concurrency is a paradigm frequently used for developing parallel applications on small-and middle-sized machines, experience has shown that it is hard to use. This is largely caused by synchronization primitives which are low-level, inherently non-deterministic, and, consequently, non-intuitive to use. In this paper, we present the Nornir run-time system. Nornir is comparable to well-known frameworks such as MapReduce and Dryad that are recognized for their efficiency and simplicity. Unlike these frameworks, Nornir also supports process structures containing branches and cycles. Nornir is based on the formalism of Kahn process networks, which is a shared-nothing, message-passing model of concurrency. We deem this model a simple and deterministic alternative to shared-memory concurrency. Experiments with real and synthetic benchmarks on up to 8 CPUs show that performance in most cases scales almost linearly with the number of CPUs, when not limited by data dependencies. We also show that the modeling flexibility allows Nornir to outperform its MapReduce counterparts using well-known benchmarks.

Scientific Workflows and Provenance: Introduction and Research Opportunities

“…For example, a parameter sweep experiment may require running a program thousands of times with slightly altered input parameters, thus consuming many compute cycles and producing so much data that manually managing it quickly becomes impossible. Distributed Grid or cloud computing technologies [24,86,94] and other parallel frameworks such as MapReduce [23] and dataflow process networks [54,3,82] can be used to scale workflow execution by exploiting parallel resources.…”

StreamDrive: a Dynamic Dataflow Framework for Clustered Embedded Architectures

“…The QUARK (QUeing And Runtime for Kernels) [61], TIDeFlow [38], and OpenStream [44], have been developed in the context of the High-Performance Computing (HPC) applications. YAPI [28] and Nornir [60] support the KPN execution model on workstation computers. These runtime environments come with heavy performance and memory footprint overheads.…”

P2G: A Framework for Distributed Real-Time Processing of Multimedia Data

“…However, in practice, very few general-purpose KPN runtime implementations exist. Known implementations include the Sesame project [34], the process network framework [28], YAPI [22] and our own Nornir [39]. These frameworks have several benefits, but for application developers, the KPN model has some challenges, particularly in a distributed scenario.…”

“…In our Nornir runtime system for parallel processing [39], we addressed many of the shortcomings of the batch processing frameworks. Nornir is based on the idea of Kahn Process Networks (KPN).…”