XChange: coupling parallel applications in a dynamic environment

Abbasi, Hasan; Wolf, Matthew; Schwan, Karsten; Eisenhauer, Greg; Hilton, A.

doi:10.1109/clustr.2004.1392646

Cited by 12 publications

(15 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Several software libraries adopted this model like Intercomm [26], Meta-Chaos [27] or PAWS [28]. XChange [29] pushes the concept of CCA by adding the possibility to apply data transformation during transferts between simulation codes. Zhang et al [30] added a shared memory space to a Dart server to support both simulation code coupling and in situ/in transit scenarios.…”

Section: Related Workmentioning

confidence: 99%

A Flexible Framework for Asynchronous in Situ and in Transit Analytics for Scientific Simulations

Dreher

Raffin

2014

2014 14th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing

View full text Add to dashboard Cite

Abstract-High performance computing systems are today composed of tens of thousands of processors and deep memory hierarchies. The next generation of machines will further increase the unbalance between I/O capabilities and processing power. To reduce the pressure on I/Os, the in situ analytics paradigm proposes to process the data as closely as possible to where and when the data are produced. Processing can be embedded in the simulation code, executed asynchronously on helper cores on the same nodes, or performed in transit on staging nodes dedicated to analytics. Today, software environnements as well as usage scenarios still need to be investigated before in situ analytics become a standard practice.In this paper we introduce a framework for designing, deploying and executing in situ scenarios. Based on a component model, the scientist designs analytics workflows by first developing processing components that are next assembled in a dataflow graph through a Python script. At runtime the graph is instantiated according to the execution context, the framework taking care of deploying the application on the target architecture and coordinating the analytics workflows with the simulation execution. Component coordination, zerocopy intra-node communications or inter-nodes data transfers rely on per-node distributed daemons.We evaluate various scenarios performing in situ and in transit analytics on large molecular dynamics systems simulated with Gromacs using up to 2048 cores. We show in particular that analytics processing can be performed on the fraction of resources the simulation does not use well, resulting in a limited impact on the simulation performance (less than 9%). Our more advanced scenario combines in situ and in transit processing to compute a molecular surface based on the Quicksurf algorithm.

show abstract

Section: Related Workmentioning

confidence: 99%

A Flexible Framework for Asynchronous in Situ and in Transit Analytics for Scientific Simulations

Dreher

Raffin

2014

2014 14th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing

View full text Add to dashboard Cite

show abstract

“…First, proactivity can be used to prevent future performance problems rather than reacting to them. Second, service augmentation can be used to enhance standard communication infrastructures like the MxN framework presented in our own earlier research [36]. Third, the examples presented in this paper locate functionality at clients vs. servers.…”

Section: Discussionmentioning

confidence: 95%

Service Augmentation for High End Interactive Data Services

Wolf

Abbasi

Collins

et al. 2005

2005 IEEE International Conference on Cluster Computing

View full text Add to dashboard Cite

Advances in computational science, combined with the increasingly interdisciplinary and geographically distributed research teams, have led to a need to support multi-tiered, data-and meta-datarich collaboration infrastructures. Our research addresses the interactive, remote tasks undertaken in such collaborations, which require a flexible software infrastructure able to dynamically deploy services where and when needed, and to provide data to clients in the forms in which they require it with suitable levels of end-to-end performance. The concept of service augmentation advanced in this paper seeks to continuously adjust the differences or degrees of incompatibility between the data received and the data displayed or stored by clients. Diference adjustments occur anywhere on the paths between data providers and clients, and compatibility computations leverage all of the resources that may be brought to bear, including CPUs and GPUs on servers and additional data manipulations on server, overlay, and client nodes.A formal structure and experimental evaluations of this concept are performed with the SmartPointer scientific visualization and annotation framework, for which we show that data-driven SLAs provide improved client flexibility and the ability to maintain application-specific notions of quality of service.

show abstract

“…LIVE's dynamic flexibility in what data to extract and in how to prepare it for transmission to other processes (e.g., analysis or visualization) is shown to enable performance improvements beyond those attained by simpler methods for data extraction. Previously reported results measuring LIVE's ability to efficiently perform M-by-N data exchanges across multiple parallel machines demonstrate the fact that data can be efficiently re-formatted or re-organized as it is being transported across different machines [13].…”

Section: Introductionmentioning

confidence: 98%

“…Runtime data selection and transformation imply that application data formats can evolve without the need to change plugins [6] and that data flows can be customized according to available resources or current application needs [12]. Changes in data distribution can be used to cope with runtime changes in parallelism [13].…”

Section: Introductionmentioning

confidence: 99%

LIVE data workspace: A flexible, dynamic and extensible platform for petascale applications

Abbasi

Wolf

Schwan

2007

2007 IEEE International Conference on Cluster Computing

View full text Add to dashboard Cite

The data needs of current and future PetaScale applications have increased over the last half decade to the extent that appropriate data management has become a crucial requirement. This concerns not only the storage of data produced by the new class of PetaScale applications, but also the data exchanges needed for coupling applications with concurrent analysis, online data visualization for validation, and others. To address such dynamic code coupling, we introduce the concept of an extensible, dynamic, and flexible data workspace, termed LIVE. In contrast to the data exchanges programmed with MPI, MPI-IO, or grid software, LIVE focuses on data exchanges carried out without a priori knowledge of potential data requirements. Examples include exchanges required by adhoc or dynamically determined methods for data validation, for general data analysis tasks, or for data visualization. Run on an execution environment comprised of integrated dynamic discovery and on-line management services, LIVE is used to create a 'data workspace' for a working molecular dynamics code base utilized by mechanical and materials engineers at Georgia Tech, for multi-scale materials modeling. Measurements of both this application's data workspace and of the basic primitives in the LIVE framework demonstrate that the environment's substantial flexibility has minimal impact on overall performance, and in fact, that it improves performance in a number of usage scenarios. In particular, for a visualization pipeline example derived from our collaborators, we see a slight improvement over a solution based on MPI-IO, and a further improvement of up to 5% by utilizing LIVE's ability to overlap communication with user-specified computation.

show abstract

XChange: coupling parallel applications in a dynamic environment

Abstract: Modern computational science applications are becoming increasingly multi-disciplinaty involving widely distributed research teams and their underlying computa-

Cited by 12 publications

References 23 publications

A Flexible Framework for Asynchronous in Situ and in Transit Analytics for Scientific Simulations

A Flexible Framework for Asynchronous in Situ and in Transit Analytics for Scientific Simulations

Service Augmentation for High End Interactive Data Services

LIVE data workspace: A flexible, dynamic and extensible platform for petascale applications

Contact Info

Product

Resources

About