Towards Systematic Parallel Programming over MapReduce

Abstract

Abstract. MapReduce is a useful and popular programming model for data-intensive distributed parallel computing. But it is still a challenge to develop parallel programs with MapReduce systematically, since it is usually not easy to derive a proper divide-and-conquer algorithm that matches MapReduce. In this paper, we propose a homomorphism-based framework named Screwdriver for systematic parallel programming with MapReduce, making use of the program calculation theory of list homomorphisms. Screwdriver is implemented as a Java library on top of Hadoop. For any problem which can be resolved by two sequential functions that satisfy the requirements of the third homomorphism theorem, Screwdriver can automatically derive a parallel algorithm as a list homomorphism and transform the initial sequential programs to an efficient MapReduce program. Users need neither to care about parallelism nor to have deep knowledge of MapReduce. In addition to the simplicity of the programming model of our framework, such a calculational approach enables us to resolve many problems that it would be nontrivial to resolve directly with MapReduce.

Accumulative Computation on MapReduce

“…The research on parallel skeletons [9], [15], [18] and list homomorphisms [23] illustrates a systematic and constructive way to high-level parallel programming, by which this work was inspired. In this paper, we have described how to implement and use the two parallel skeletons scan and accumulate, in MapReduce.…”

“…An input list is given as [11,15,8,9,20,25,12,23], the initial value of parameter c = −∞, and the list is split to two (with the offset 0 and 10, respectively). The processing is represented in the following tabular form, step by step.…”

“…But there is still a gap between a nontrivial problem and the MapReduce paradigm because MapReduce programming model is relatively low-level that leads many difficulties in piratical programming. Many studies such as Sawzall [25], PigLatin [12], and DryadLINQ [27] tried to provide more user-friendly domain specific languages to address the programmability problem, and our previous work [11], [23] introduced the approaches of calculation theorems for list homomorphisms into MapReduce, for the similar purpose but in different methodology.…”

“…As an example, consider the elimSmallers problem of eliminating all the smaller elements of a list to produce an ascending list (if an element is less than someone in the previous, it is smaller). For instance, given a list [11,15,8,9,20,25,12,23] , then 8, 9, 12 and 23 are smaller ones, and thus the result is [11,15,20,25]. A recursive function that solves this problem can be defined as follows, in Haskell [5].…”

Farms, pipes, streams and reforestation: reasoning about structured parallel processes using types and hylomorphisms

“…The third homomorphism theorem states that if a function can be written both as a left fold and a right fold, then it can also be evaluated in a divide-and-conquer manner [9]. This theorem has been widely used for parallelism [2,6,8,11,19,24,26]. The majority of this work enables suitable automation and derivation of efficient parallel implementations.…”