A programming model and foundation for lineage-based distributed computation

Abstract

The most successful systems for “big data” processing have all adopted functional APIs. We present a new programming model, we call function passing, designed to provide a more principled substrate, or middleware, upon which to build data-centric distributed systems like Spark. A key idea is to build up a persistent functional data structure representing transformations on distributed immutable data by passing well-typed serializable functions over the wire and applying them to this distributed data. Thus, the function passing model can be thought of as a persistent functional data structure that is distributed, where transformations performed on distributed data are stored in its nodes rather than the distributed data itself. One advantage of this model is that failure recovery is simplified by design – data can be recovered by replaying function applications atop immutable data loaded from stable storage. Deferred evaluation is also central to our model; by incorporating deferred evaluation into our design only at the point of initiating network communication, the function passing model remains easy to reason about while remaining efficient in time and memory. Moreover, we provide a complete formalization of the programming model in order to study the foundations of lineage-based distributed computation. In particular, we develop a theory of safe, mobile lineages based on a subject reduction theorem for a typed core language. Furthermore, we formalize a progress theorem that guarantees the finite materialization of remote, lineage-based data. Thus, the formal model may serve as a basis for further developments of the theory of data-centric distributed programming, including aspects such as fault tolerance. We provide an open-source implementation of our model in and for the Scala programming language, along with a case study of several example frameworks and end-user programs written atop this model.

A fault-tolerant programming model for distributed interactive applications

“…CPL [Bračevac et al 2016] is a calculus to create and combine multiple services in a type safe manner. Function passing [Haller et al 2018;Miller et al 2016] defines derived values of immutable remote data in a type safe way. Concurrent Objects [Filipović et al 2010] provides objects with serializability and linearizability by guaranteeing that concurrent implementations behave observational equivalent to sequential implementations.…”

An Executable Specification of Map-Join-Reduce Using Haskell

“…There are many benefits provided by functional programming that have won over framework designers, such as immutability, compositionality, powerful abstraction forms and lazy evaluation. As stated in [10], data transformation via higher-order functions makes it much easier, by design, to tackle concerns central to distributed systems such as concurrency. As one of powerful functional programming languages, Haskell has the benefits of fast prototyping capabilities, strong type checking and type inference.…”