Bulk synchronous parallel computing-a paradigm for transportable software

Cheatham, Thomas E.; Fahmy, Amr; Ştefănescu, Dan; Valiant, Leslie G.

doi:10.1109/hicss.1995.375451

Cited by 50 publications

(25 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…They are often used in conjunction with experimentation to account for aspects that are not considered by the models, such as local computation times. As such, they have been quite successful, leading to practical designs of various algorithms [3], [6], [14], [16], [17], [22], [26], [30], [38].…”

Section: Introductionmentioning

confidence: 99%

Accounting for memory bank contention and delay in high-bandwidth multiprocessors

Blelloch

Gibbons

Matias

et al. 1995

Proceedings of the Seventh Annual ACM Symposium on Parallel Algorithms and Architectures - SPAA '95

View full text Add to dashboard Cite

Abstract-For years, the computation rate of processors has been much faster than the access rate of memory banks, and this divergence in speeds has been constantly increasing in recent years. As a result, several shared-memory multiprocessors consist of more memory banks than processors. The object of this paper is to provide a simple model (with only a few parameters) for the design and analysis of irregular parallel algorithms that will give a reasonable characterization of performance on such machines. For this purpose, we extend Valiant's bulk-synchronous parallel (BSP) model with two parameters: a parameter for memory bank delay, the minimum time for servicing requests at a bank, and a parameter for memory bank expansion, the ratio of the number of banks to the number of processors. We call this model the (d, x)-BSP. We show experimentally that the (d, x)-BSP captures the impact of bank contention and delay on the CRAY C90 and J90 for irregular access patterns, without modeling machine-specific details of these machines. The model has clarified the performance characteristics of several unstructured algorithms on the CRAY C90 and J90, and allowed us to explore tradeoffs and optimizations for these algorithms. In addition to modeling individual algorithms directly, we also consider the use of the (d, x)-BSP as a bridging model for emulating a very high-level abstract model, the Parallel Random Access Machine (PRAM). We provide matching upper and lower bounds for emulating the EREW and QRQW PRAMs on the (d, x)-BSP.

show abstract

Section: Introductionmentioning

confidence: 99%

Accounting for memory bank contention and delay in high-bandwidth multiprocessors

Blelloch

Gibbons

Matias

et al. 1995

Proceedings of the Seventh Annual ACM Symposium on Parallel Algorithms and Architectures - SPAA '95

View full text Add to dashboard Cite

show abstract

“…That is, we assume p processors connected via an interconnection fabric where processors have t ypical workstation size local memories and concurrent access to a shared disk array. For the purpose of parallel algorithm design, we use the Coarse Grained Multicomputer (CGM) model 5,8,15,18,24]. More precisely, we use the EM-CGM model 6, 7, 9] which is a multi-processor version of Vitter's Parallel Disk Model 25,26,27].…”

Section: Parallel Computing Modelmentioning

confidence: 99%

Parallelizing the Data Cube

Dehne

Eavis

Hambrusch

et al. 2001

Database Theory — ICDT 2001

View full text Add to dashboard Cite

This paper presents a general methodology for the e cient parallelization of existing data cube construction algorithms. We describe two di erent partitioning strategies, one for top-down and one for bottom-up cube algorithms. Both partitioning strategies assign subcubes to individual processors in such a way that the loads assigned to the processors are balanced. Our methods reduce inter-processor communication overhead by partitioning the load in advance instead of computing each individual group-by in parallel as is done in previous parallel approaches. In fact, after the initial load distribution phase, each processor can compute its assigned subcube without any communication with the other processors. Our methods enable code reuse by permitting the use of existing sequential (external memory) data cube algorithms for the subcube computations on each processor. This supports the transfer of optimized sequential data cube code to a parallel setting.The bottom-up partitioning strategy balances the number of single attribute external memory sorts made by each processor. The top-down strategy partitions a weighted tree in which weights re ect algorithm speci c cost measures like estimated group-by sizes. Both partitioning approaches can beimplement e d o n a n y shared disk type parallel machine composed of p processors connected via an interconnection fabric and with access to a shared parallel disk array.We h a ve implemented our parallel top-down data cube construction method in C++ with the MPI message passing library for communication and the LEDA library for the required graph algorithms. We tested our code on an eight processor cluster, using a variety of di erent data sets with a range of sizes, dimensions, density, and skew. The tests show that our partitioning strategies generate a close to optimal load balance between processors. The actual run times observed show an optimal speedup of p.2

show abstract

“…Finally, several groups have been involved recently in implementations and experimentations of parallel algorithms, using bandwidth-limited general purpose models (see, e.g., [7], [15], [19], [21], [22], [25], [30], [32] and [44]). …”

Section: Modeling Parallel Bandwidthmentioning

confidence: 99%

Modeling Parallel Bandwidth: Local versus Global Restrictions

Adler

Gibbons²,

Matias

et al. 1999

Algorithmica

View full text Add to dashboard Cite

Abstract.Recently there has been an increasing interest in models of parallel computation that account for the bandwidth limitations in communication networks. Some models (e.g., BSP, LOGP, and QSM) account for bandwidth limitations using a per-processor parameter g > 1, such that each processor can send/receive at most h messages in g · h time. Other models (e.g., PRAM(m)) account for bandwidth limitations as an aggregate parameter m < p, such that the p processors can send at most m messages in total at each step.This paper provides the first detailed study of the algorithmic implications of modeling parallel bandwidth as a per-processor (local) limitation versus an aggregate (global) limitation. We consider a number of basic problems such as broadcasting, parity, summation, and sorting, and give several new upper and lower time bounds that demonstrate the advantage of globally limited models over locally limited models given the same aggregate bandwidth (i.e., p·1/g = m). In general, globally limited models have a possible advantage whenever there is an imbalance in the number of messages sent/received by the processors. To exploit this advantage, the processors must schedule the sending of messages so as to respect the aggregate bandwidth limit. We present a new parallel scheduling algorithm for globally limited models that enable an unknown, arbitrarily unbalanced set of messages to be sent through the limited bandwidth within a (1 + ε) factor of the optimal off-line schedule with high probability, even if the penalty for overloading the network is an exponential function of the overload. We also present a near-optimal algorithm for the case where long messages must be sent as flits in consecutive time steps, as well as for the case where new messages to be sent arrive dynamically over an infinite time line. These results consider both message passing (distributed memory) and shared memory scenarios, and improve upon the best results for the locally limited model by a factor of (g). Finally, we present results quantifying the power of concurrent reads in a globally limited bandwidth setting, including showing an ( p lg m/m lg p) time separation between the exclusive-read and the concurrent-read PRAM(m) models, which, when m p, greatly improves upon the 2 ( √ lg p) separation known previously.

show abstract

Bulk synchronous parallel computing-a paradigm for transportable software

Cited by 50 publications

References 15 publications

Accounting for memory bank contention and delay in high-bandwidth multiprocessors

Accounting for memory bank contention and delay in high-bandwidth multiprocessors

Parallelizing the Data Cube

Modeling Parallel Bandwidth: Local versus Global Restrictions

Contact Info

Product

Resources

About