Toward loosely coupled programming on petascale systems

Abstract

Abstract-We have extended the Falkon execution framework to make loosely coupled petascale systems a practical and useful prog This work studies and measures the perf involved in applying this approach to enable th systems by a broader user community, and w Our work enables the execution of highly para composed of loosely coupled serial jobs with no the respective applications. This approach all potentially far larger-class of applications to l systems, such as the IBM Blue Gene/P sup present the challenges of I/O performance encou this model practical, and show resul microbenchmarks and real applications from economic energy modeling and molecular benchmarks show that we can scale up to 160K with high efficiency, and can achieve sustained thousands of tasks per second.

Task‐parallel message passing interface implementation of Autodock4 for docking of very large databases of compounds using high‐performance super‐computers

“…Although no overhead is detected up to 256 CPUs, there is a maximum overhead of 7% (VEGFR2) at 4096 CPUs, which increases up to 16.3% (PARP) at 8192 CPUs. At 16,384 CPUs, the overhead varies from 27.8% (ACE) up to 34.3% (PARP). The performance of Autodock4.lga.MPI is similar to that of the program Dovis2 using 256 CPUs, 31 as well as the program Dock6.MPI that shows overheads of about 8% at 4096 CPUs, 12% at 8192 CPUs, and 45% at 16,384 CPUs.…”

“…14- 16 The performance can be significantly affected by time spent by the system in computations that do not contribute to the advancement of any user tasks (i.e., ''overhead''), such as task management, allocation of resources, and input/output (I/O) operations (open/read/write/close). [14][15][16] A task-parallel docking procedure (or high-throughput computing 17 ), i.e., n simultaneous and independent docking jobs running on n CPUs, can be carried out by executing a serial docking program (i.e., one that cannot run on more than one CPU) on each of the n CPUs.…”

“…[14][15][16] A task-parallel docking procedure (or high-throughput computing 17 ), i.e., n simultaneous and independent docking jobs running on n CPUs, can be carried out by executing a serial docking program (i.e., one that cannot run on more than one CPU) on each of the n CPUs. Among the popular docking programs used in a task-parallel fashion are the academic software packages Autodock 18 and Dock, [19][20][21] which precalculate aspects of the protein-ligand interactions for a number of points equally spaced on a grid.…”

“…14,26 In other work, about 138K serial Dock6 jobs were run on 128K CPUs on Blue Gene using the recently developed task execution framework Falkon, 15 exhibiting 3-5% overhead. 16 As an alternative to starting serial jobs through an external task manager such as PBS, parallelization can be implemented within the docking software. The message passing interface (MPI) specification 27 is commonly used for parallelizing computing applications by providing functions that can be introduced in programs to allow inter-CPU communication.…”

Palacios and Kitten : high performance operating systems for scalable virtualized and native supercomputing.

“…We view this approach as overly restrictive and potentially harmful in several ways: system reliability is jeopardized by more reboot cycles, diagnosing and monitoring the health of individual nodes is difficult, and the system is less available for use. Management based on virtualization would also make it possible to backfill work on the machine using loosely-coupled programming jobs [26] or other low priority work. A batch-submission or grid computing system could be run on a collection of nodes where a new OS stack could be dynamically launched; this system could also be brought up and torn down as needed.…”