Armada: a parallel I/O framework for computational grids

Oldfield, Ron A.; Kotz, David

doi:10.1016/s0167-739x(01)00076-0

Cited by 15 publications

(7 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Instead, we place data queries into ongoing workflows in ways that improve both simulation and query performance. Additional input to our work can be derived from previous work on remote visualization in grid and distributed infrastructures [7][33][6] [22].…”

Section: Related Workmentioning

confidence: 99%

FlexQuery: An online query system for interactive remote visual data exploration at large scale

Zou

Schwan

Sławińska

et al. 2013

2013 IEEE International Conference on Cluster Computing (CLUSTER)

View full text Add to dashboard Cite

The remote visual exploration of live data generated by scientific simulations is useful for scientific discovery, performance monitoring, and online validation for the simulation results. Online visualization methods are challenged, however, by the continued growth in the volume of simulation output data that has to be transferred from its source -the simulation running on the high end machine -to where it is analyzed, visualized, and displayed. A specific challenge in this context is limits in the communication bandwidth between data source(s) and sinks. Previous work places queries 'near' data sources, exploiting their data reduction capabilities, but such work does not address the common scenario in which scientists make multiple different queries on the data being produced. This paper considers the general case in which science users are interested in different (sub)sets of the data produced by a high end simulation. We offer the FlexQuery online data query system that can deploy and execute data queries 'along' the I/O and analytics pipelines. FlexQuery carefully extends such analytics pipelines, using online performance monitoring and data location tracking, to realize data queries in ways that minimize additional data movement and offer low latency in data query execution. Using a real-world scientific application -the Maya astrophysics code and its analytics workflow -we demonstrate FlexQuery's ability to dynamically deploy queries for low-latency remote data visualization.

show abstract

Section: Related Workmentioning

confidence: 99%

FlexQuery: An online query system for interactive remote visual data exploration at large scale

Zou

Schwan

Sławińska

et al. 2013

2013 IEEE International Conference on Cluster Computing (CLUSTER)

View full text Add to dashboard Cite

show abstract

“…G2 [12] composes distributed parallel programs from Web services through Microsoft .Net. Armada [13] composes distributed parallel programs specialized to data movement and filtering.…”

Section: Related Researchmentioning

confidence: 99%

Productivity and performance through components: the ASCI Sweep3D application

Yoon

Browne

Crocker

et al. 2006

Concurrency and Computation

View full text Add to dashboard Cite

SUMMARYThis paper is a case study of the effectiveness of component-oriented development for enhancing both productivity and performance for parallel programs. A process for converting monolithic applications into semantically composable components is described. The supporting software, the P-COM 2 compositional compiler, is briefly described. The componentized version of Sweep3D is described. Productivity is illustrated by composing different instances of the Sweep3D code through automated composition of components using P-COM 2 . These instances, each of which targets improving performance for some execution environment or problem case, are examples of a family of instances which are composable from a modest set of components. It is found that customization of componentized codes by componentlevel adaptation may yield substantial performance improvement for specific execution environments. We identify and explain some of the benefits of component-oriented development for high-performance parallel systems.

show abstract

“…The Armada framework for parallel I/O [23] provides a solution for data-intensive applications in which the application programmer and the data set provider deploy a network of application-specific and data-set-specific functionality across the grid. Using the Armada framework, grid applications access remote data by sending data requests through a graph of distributed application objects called ships.…”

Section: Introductionmentioning

confidence: 99%

“…In previous papers, we describe the Armada framework [21,23]. In this paper, we present a formal description of the components that make up an Armada graph, and we describe and analyze an algorithm for restructuring a graph based on programmer-specified properties of the individual ships.…”

Section: Introductionmentioning

confidence: 99%

Improving Data Access for Computational Grid Applications

Oldfield

Kotz

2006

Cluster Comput

Self Cite

View full text Add to dashboard Cite

High-performance computing increasingly occurs on "computational grids" composed of heterogeneous and geographically distributed systems of computers, networks, and storage devices that collectively act as a single "virtual" computer. A key challenge in this environment is to provide efficient access to data distributed across remote data servers. Our parallel I/O framework, called Armada, allows application and data-set providers to flexibly compose graphs of processing modules that describe the distribution, application interfaces, and processing required of the dataset before computation. Although the framework provides a simple programming model for the application programmer and the data-set provider, the resulting graph may contain bottlenecks that prevent efficient data access. In this paper, we present an algorithm used to restructure Armada graphs that distributes computation and data flow to improve performance in the context of a wide-area computational grid.

show abstract

Armada: a parallel I/O framework for computational grids

Cited by 15 publications

References 35 publications

FlexQuery: An online query system for interactive remote visual data exploration at large scale

FlexQuery: An online query system for interactive remote visual data exploration at large scale

Productivity and performance through components: the ASCI Sweep3D application

Improving Data Access for Computational Grid Applications

Contact Info

Product

Resources

About