A New Flexible MPI Collective I/O Implementation

Coloma, Kenin; Ching, Avery; Choudhary, Alok; Liao, Wei-keng; Ross, Rob; Thakur, Rajeev; Ward, Lee

doi:10.1109/clustr.2006.311865

Cited by 22 publications

(15 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Through a C interface, one first declares variables, dimensions, and attributes in "define" mode; individual arrays are then written in "data access" mode. Both PnetCDF and HDF5 [23], another popular high-level scientific data library, derive their parallel performance from MPI-I/O [24].…”

Section: 2mentioning

confidence: 99%

High-Performance Computation of Distributed-Memory Parallel 3D Voronoi and Delaunay Tessellation

Peterka

Morozov

Phillips

2014

SC14: International Conference for High Performance Computing, Networking, Storage and Analysis

View full text Add to dashboard Cite

ABSTRACT. Computing a Voronoi or Delaunay tessellation from a set of points is a core part of the analysis of many simulated and measured datasets: N-body simulations, molecular dynamics codes, and LIDAR point clouds are just a few examples. Such computational geometry methods are common in data analysis and visualization; but as the scale of simulations and observations surpasses billions of particles, the existing serial and shared-memory algorithms no longer suffice. A distributed-memory scalable parallel algorithm is the only feasible approach. The primary contribution of this paper is a new parallel Delaunay and Voronoi tessellation algorithm that automatically determines which neighbor points need to be exchanged among the subdomains of a spatial decomposition. Other contributions include periodic and wall boundary conditions, comparison of our method using two popular serial libraries, and application to numerous science datasets.E-mail addresses: tpeterka@mcs.anl.gov, dmitriy@mrzv.org, cphillips@anl.gov. DISCLAIMER. This document was prepared as an account of work sponsored by the United States Government. While this document is believed to contain correct information, neither the United States Government nor any agency thereof, nor the Regents of the University of California, nor any of their employees, makes any warranty, express or implied, or assumes any legal responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. Reference herein to any specific commercial product, process, or service by its trade name, trademark, manufacturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States Government or any agency thereof, or the Regents of the University of California. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States Government or any agency thereof or the Regents of the University of California.

show abstract

Section: 2mentioning

confidence: 99%

High-Performance Computation of Distributed-Memory Parallel 3D Voronoi and Delaunay Tessellation

Peterka

Morozov

Phillips

2014

SC14: International Conference for High Performance Computing, Networking, Storage and Analysis

View full text Add to dashboard Cite

show abstract

“…In [4], the authors introduced the ability to easily modify parameters of collective I/O operations. However, the work focused entirely on collective I/O, leaving other aspects of parallel I/O unmodified.…”

Section: Related Workmentioning

confidence: 99%

OMPIO: A Modular Software Architecture for MPI I/O

Chaarawi

Gabriel

Keller³

et al. 2011

Recent Advances in the Message Passing Interface

View full text Add to dashboard Cite

Abstract. I/O is probably the most limiting factor on high-end machines for large scale parallel applications as of today. This paper introduces OMPIO, a new parallel I/O architecture for Open MPI. OMPIO provides a highly modular approach to parallel I/O by separating I/O functionality into smaller units (frameworks) and an arbitrary number of modules in each framework. Furthermore, each framework has a customized selection criteria that determines which module to use depending on the functionality of the framework as well as external parameters.

show abstract

“…Storage systems, especially those using hard disks for direct data access, usually cannot sustain high I/O throughput with small requests for random data from a large number of parallel processes. Many techniques have been proposed to improve requests' spatial locality by transforming them into large sequential requests, such as data sieving [11] in ROMIO. There are variants of collective I/O recently proposed to take account of on-disk data layout, including Resonant I/O [35] and reducing locking overhead [24] during request transformation.…”

Section: Related Workmentioning

confidence: 99%

Orthrus: A Framework for Implementing Efficient Collective I/O in Multi-core Clusters

Zhang

Davis

et al. 2014

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. Optimization of access patterns using collective I/O imposes the overhead of exchanging data between processes. In a multi-core-based cluster the costs of inter-node and intra-node data communication are vastly different, and heterogeneity in the efficiency of data exchange poses both a challenge and an opportunity for implementing efficient collective I/O. The opportunity is to effectively exploit fast intra-node communication. We propose to improve communication locality for greater data exchange efficiency. However, such an effort is at odds with improving access locality for I/O efficiency, which can also be critical to collective-I/O performance. To address this issue we propose a framework, Orthrus, that can accommodate multiple collective-I/O implementations, each optimized for some performance aspects, and dynamically select the best performing one accordingly to current workload and system patterns. We have implemented Orthrus in the ROMIO library. Our experimental results with representative MPI-IO benchmarks on both a small dedicated cluster and a large production HPC system show that Orthrus can significantly improve collective I/O performance under various workloads and system scenarios.

show abstract

A New Flexible MPI Collective I/O Implementation

Abstract: The

Cited by 22 publications

References 16 publications

High-Performance Computation of Distributed-Memory Parallel 3D Voronoi and Delaunay Tessellation

High-Performance Computation of Distributed-Memory Parallel 3D Voronoi and Delaunay Tessellation

OMPIO: A Modular Software Architecture for MPI I/O

Orthrus: A Framework for Implementing Efficient Collective I/O in Multi-core Clusters

Contact Info

Product

Resources

About