Mobile Pipelines: Parallelizing Left-Looking Algorithms Using Navigational Programming

Abstract

Abstract. Parallelizing a sequential algorithm-i.e., manually or automatically converting it into an equivalent parallel distributed algorithm-is an important problem. Ideally, the parallel algorithm should preserve the computational structure of the original sequential algorithm, display a high degree of parallelism, have low communication overhead, and be scalable. The difficulty of accomplishing this for a particular sequential algorithm depends on the nature of the algorithm and the specific model of parallelism under consideration. Generally, the so called "right-looking" algorithms are easy to parallelize because they tend to have a significant amount of data parallelism. In these algorithms, data is "eagerly" propagated to and consumed by subsequent computations immediately after it is produced, and so it does not need to be kept in temporary storage for extended periods of time. In this paper we consider the class of "left-looking" sequential algorithms [1], which cannot be easily parallelized using the message-passing or distributed shared memory models. These algorithms are characterized by "lazy" propagation of data. We show that these algorithms can be directly parallelized using the Navigational Programming model. We present performance data demonstrating the effectiveness of our approach.

Incremental Parallelization with Migration

“…In this case, performing the computation on the node that holds the read set is more efficient than the node that writes the final value. For a specific example see [41]. The two strategies frequently produce the same result, as they do with the examples of this paper.…”

“…NavP provides a different view of distributed computation from the classical SPMD (Single Program Multiple Data) view [40,41]. The SPMD view describes distributed computations at stationary locations, while the NavP view describes a computation following the movement of its locus.…”

“…NavP [40] is still considered SPMD (Single Program Multiple Data) in that different processes execute the same code with different data, and differs from the traditional SPMD view [40,41] in that processes are able to access remote data by migrating to the target node. Hence the locus of any given computation is not stationary but follows the distribution of data as appropriate for best performance.…”

NavP: Enabling Navigational Programming for Science Data Processing via Application-Initiated Checkpointing

“…And multiple DSC's are then synchronized to carry out parallel computation using the idea of a Mobile Pipeline (2005) [7]. Scalable performance and easy of programming has been demonstrated in this way, for notoriously hard-to-parallelize numerical algorithms.…”

NavP: Structured and Multithreaded Distributed Parallel Programming

“…It has been pointed out that send and recv are harmful today for much the same reason that unrestricted goto statements have been considered harmful since the "software crisis" of the 1960's [17]. Because of its structured programming, NavP allows us to parallelize certain programs that are generally considered unparallelizable using other approaches [31]. Approaches such as HPF or OpenMP allow the programmers to work with the original code structure, but they rely heavily on their compilers and give the programmers less flexibility than MPI or NavP does in controlling locality and parallelism.…”