A framework for partitioning parallel computations in heterogeneous environments

Abstract

In the paper we present a framework for partitioning data parallel computations across a heterogeneous metasystem at runtime. The framework is guided by program and resource information which is made available to the system. Three difficult problems are handled by the framework: processor selection, task placement and heterogeneous data domain decomposition. Solving each of these problems contributes to reduced elapsed time. In particular, processor selection determines the best gain size at which to run the computation, task placement reduces communication cost, and data domain decomposition achieves processor load balance. We present results which indicate that excellent performanceis achievableusing the framework. The paper extends our earlier work on partitioning data parallel computations across a singlelevel network of heterogeneous workstations. INTRODUCTIONA great deal of recent interest has been sparked within academic, industrial and government circles in the emerging technology of metmystem-based high-performance computing. A metasystern is a shared ensemble of workstations, vector and parallel machines connected by local-and wide-area networks (see Figure I). The promise of on-line gigabit networks coupled with the tremendous computing power of the metasystem makes it very attractive for parallel computations.The potentially large array of heterogeneous resources in the metasystem offers an opportunity for delivering high performance on a range of parallel computations. Choosing the best set of available resources is a difficult problem and is the subject of this paper. Consider the set of machines in Table 1 and observe that they have different computation and communication capacities. Loosely coupled parallel computations with infrequent communication would probably benefit by applying the fastest set of computational resources (perhaps the DEC-Alphacluster), and may benefit from distributionacross many machines. On the other hand, more tightly coupled parallel computations are best suited to machines that have a higher communication capacity (perhaps an Intel Paragon), but may also benefit from distribution across many machines if the computation granularity is sufficient. We address the latter problem in this paper.We present a framework that automates partitioning and placement of data parallel computations across metasystems such as in Figure 1. Partitioning is performed at runtime when the state of the metasystem resources are known. Three difficult problems are handled by the framework: processor selection, task placement and heterogeneous data domain decomposition. Solving each of these problems contributes to reduced completion time. Processor selection chooses the best number and type of processors to apply to the computation. This el = k = T = the i " ' network cluster the irh processor cluster application communication topology message size in bytes communication cost coefficients processor-dependent communication function router cost constants coercion cost constant number of messages th...

Gallop: The Benefits of Wide-Area Computing for Parallel Processing

“…However, in some systems, the SM and LS may be configured as a single process. In this system prototype, the LS is the Prophet scheduler [21]. The LS will explore and evaluate potential resource schedules within the site using application information provided by the callbacks (Fig.…”

“…Prophet explores a set of candidate processor configurations to apply to the application in order to minimize completion time. The details of this algorithm may be found in [21]. Prophet first orders the processor clusters based upon their computation and communication capabilities for the application in order to constrain the search process.…”

“…However, a library of classes for common application types such as stencil problems can be easily developed to simplify this task for the application programmer. The domain class is used by the Prophet scheduling system [21], a software scheduler that runs at every site described in Section 3.3.…”

“…Gallop schedules parallel applications including data parallel (SPMD) and task parallel pipelines (PP) to reduce application completion time. Gallop is built upon two established software technologies, Mentat [9] and Prophet [21]. Mentat provides the programming environment for Gallop applications and the necessary run-time support for remote execution.…”

Predicting Cpu Availability on the Computational Grid Using the Network Weather Service

“…We have used this notion of CPU availability to successfully schedule parallel programs in shared distributed environments [20,2]. Other scheduling systems such as Prophet [23], Winner [1], and MARS [12] use some notion of Unix load average to accomplish the same purpose.…”