Near-optimal adaptive control of a large grid application

Abstract

This paper develops a performance model that is used to control the adaptive execution the ATR code for solving large stochastic optimization problems on computational grids. A detailed analysis of the execution characteristics of ATR is used to construct the performance model that is then used to specify (a) near-optimal dynamic values of parameters that govern the distribution of work, and (b) a new task scheduling algorithm. Together, these new features minimize ATR execution time on any collection of compute nodes, including a varying collection of heterogeneous nodes. The new adaptive code runs up to eight-fold faster than the previously optimized code, and requires no input parameters from the user to guide the distribution of work. Furthermore, the modeling process led to several changes in the Condor runtime environment, including the new task scheduling algorithm, that produce significant performance improvements for master-worker computations as well as possibly other types of grid applications.

Modular, Fine-Grained Adaptation of Parallel Programs

“…In Grid and cluster computing, there is a large body of research work on runtime platforms for supporting program adaptation at the level of middleware or runtime platforms [4,5,6,7]. However, as their objective is to implement middleware support for adaptation between the application and the underlying execution layer, these efforts focus on resource management towards efficient utilization of the environment, such as load-balancing and scheduling of application tasks, where coarse-grained strategies based on resource constraints or external operating parameters are employed.…”

“…The approaches to support program adaptation include: languages and compilers for specifying adaptation strategies [1,2,3] and runtime platforms or middleware for adaptive execution [4,5,6,7]. These efforts are primarily centered around resource management to achieve efficient utilization of the environment, such as adaptive load-balancing and scheduling of application tasks, to match resource constraints or dynamic operating conditions of the environment.…”

A combinatorial model for self-organizing networks

“…These large scale systems are based on variants of the master/workers model [4,30,37,18,16,12,26,27,21,25,7,24]. The level of technical maturity reached by some of these projects is demonstrated by the fact that they have spawned commercial enterprises.…”

Self-Organizing Scheduling on the Organic Grid

“…A number of large-scale systems are based on variants of the master/workers model [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16]. The fact that some of these systems have resulted in commercial enterprises shows the level of technical maturity reached by the technology.…”