Workflow-Based Data Parallel Applications on the EGEE Production Grid Infrastructure

Abstract

Setting up and deploying complex applications on a Grid infrastructure is still challenging and the programming models are rapidly evolving. Efficiently exploiting Grid parallelism is often not straight forward. In this paper, we report on the techniques used for deploying applications on the EGEE production Grid through four experiments coming from completely different scientific areas: J. Montagnat I3S laboratory, CNRS, Sophia Antipolis, France J. Montagnat (B) EPU, RAINBOW, 930 route des Colles, nuclear fusion, astrophysics and medical imaging. These applications have in common the need for manipulating huge amounts of data and all are computationally intensive. All the cases studied show that the deployment of data intensive applications require the development of more or less elaborated application-level workload management systems on top of the gLite middleware

Strategies for Rescheduling Tightly-Coupled Parallel Applications in Multi-Cluster Grids

“…The strategies in this paper for rescheduling using rescheduling plans can also be applied to workflow applications consists of both serial and parallel jobs [43][44][45][46][47][48]. A workflow application is represented as a Directed Acyclic Graph (DAG) where each node of the graph represents a parallel or sequential job/application that forms a component of the overall application and an edge denotes the control and data dependency between two application components.…”

Evolution of grid-based services for Diffusion Tensor Image analysis

“…Similar to our earliest findings, queueing times significantly impact performance. Other more recently implemented neuroimaging applications also use workflow technology (e.g., [30][31][32][33]). A plugin was written for medical image processing workflows to run on the EGEE Grid using Taverna [34], and generic services with multiple layers of complexity were designed for neuroimaging applications [35].…”

Multi-objective Reinforcement Learning for Responsive Grids

“…The need for responsiveness arises in various situations, ranging from urgent computing applications [3], where the "limited time" might be quite high, to truly interactive applications involving computational steering, in which the individual task durations are extremely small [17]; another example is workflows where sequential supervision tasks are on the critical path [14]. Major industry players acknowledge interactivity as a critical requirement for enlarging the scope of high performance computing and invest in this direction.…”