MPI for Python

Dalcín, Lisandro; Paz, Rodrigo R.; Storti, Mario A.

doi:10.1016/j.jpdc.2005.03.010

Cited by 349 publications

(231 citation statements)

References 8 publications

Supporting

Mentioning

231

Contrasting

Order By: Relevance

“…These delayed processes were observed on three different HPC systems and on different days, so they are unlikely to be infrastructure specific. In order to rule out the hypothesis that Dask is inherently limited in its applicability to our problem we re-implemented our mapreduce problem with MPI based on the Python mpi4py [DPS05], [DPKC11] module. The comparison was performed with the XTC600x trajectory on SDSC Comet.…”

Section: Performance Optimizationmentioning

confidence: 99%

“…MDAnalysis does not yet provide a standard interface for parallel analysis; instead, various existing parallel libraries such as Python multiprocessing, joblib, and mpi4py [DPS05], [DPKC11] are currently used to parallelize MDAnalysis-based code on a case-by-case basis. Here we evaluated performance for parallel map-reduce [DG08] type analysis with the Dask parallel computing library [Roc15] for task-graph based distributed computing on HPC and local computing resources.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Parallel Analysis in MDAnalysis using the Dask Parallel Computing Library

Khoshlessan¹,

Paraskevakos²,

Jha³

et al. 2017

Proceedings of the 16th Python in Science Conference

View full text Add to dashboard Cite

Abstract-The analysis of biomolecular computer simulations has become a challenge because the amount of output data is now routinely in the terabyte range. We evaluated if this challenge can be met by a parallel map-reduce approach with the Dask parallel computing library for task-graph based computing coupled with our MDAnalysis Python library for the analysis of molecular dynamics (MD) simulations. We performed a representative performance evaluation, taking into account the highly heterogeneous computing environment that researchers typically work in together with the diversity of existing file formats for MD trajectory data. We found that the underlying storage system (solid state drives, parallel file systems, or simple spinning platter disks) can be a deciding performance factor that leads to data ingestion becoming the primary bottleneck in the analysis work flow. However, the choice of the data file format can mitigate the effect of the storage system; in particular, the commonly used Gromacs XTC trajectory format, which is highly compressed, can exhibit strong scaling close to ideal due to trading a decrease in global storage access load against an increase in local per-core CPU-intensive decompression. Scaling was tested on a single node and multiple nodes on national and local supercomputing resources as well as typical workstations. Although very good strong scaling could be achieved for single nodes, good scaling across multiple nodes was hindered by the persistent occurrence of "stragglers", tasks that take much longer than all other tasks, and whose ultimate cause could not be completely ascertained. In summary, we show that, due to the focus on high interoperability in the scientific Python eco system, it is straightforward to implement map-reduce with Dask in MDAnalysis and provide an in-depth analysis of the considerations to obtain good parallel performance on HPC resources.

show abstract

Section: Performance Optimizationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Parallel Analysis in MDAnalysis using the Dask Parallel Computing Library

Khoshlessan¹,

Paraskevakos²,

Jha³

et al. 2017

Proceedings of the 16th Python in Science Conference

View full text Add to dashboard Cite

show abstract

“…On the other hand, the Parallel Python (PP) module [3] provides mechanisms for parallel execution of Python codes, with an API with specific functions to specify the number of workers to be used, submit the jobs for execution, get the results from the workers, etc. Finally, the mpi4py [20] library allows the user to open parallelism both inter-node and intra-node. In all cases, the management of the parallelism is the programmer's responsibility.…”

Section: Motivationmentioning

confidence: 99%

Enabling Python to execute efficiently in heterogeneous distributed infrastructures with PyCOMPSs

Amela

Ramon-Cortes

Ejarque

et al. 2017

Proceedings of the 7th Workshop on Python for High-Performance and Scientific Computing

View full text Add to dashboard Cite

Python has been adopted as programming language by a large number of scientific communities. Additionally to the easy programming interface, the large number of libraries and modules that have been made available by a large number of contributors, have taken this language to the top of the list of the most popular programming languages in scientific applications. However, one main drawback of Python is the lack of support for concurrency or parallelism. PyCOMPSs is a proved approach to support task-based parallelism in Python that enables applications to be executed in parallel in distributed computing platforms. This paper presents PyCOMPSs and how it has been tailored to execute tasks in heterogeneous and multi-threaded environments. We present an approach to combine the task-level parallelism provided by PyCOMPSs with the thread-level parallelism provided by MKL. Performance and behavioral results in distributed computing heterogeneous clusters show the benefits and capabilities of PyCOMPSs in both HPC and Big Data infrastructures.

show abstract

“…• Serial processing: SerialPool • Python standard-library multiprocessing: MultiPool • OpenMPI (Gabriel et al 2004) and mpich2 (Lusk, Doss, and Skjellum 1996) via the mpi4py package (Dalcín, Paz, and Storti 2005;Dalcín et al 2008): MPIPool • joblib: JoblibPool All pool classes provide a .map() method to distribute tasks to a specified worker function (or callable), and support specifying a callback function that is executed on the master process to enable post-processing or caching the results as they are delivered.…”

mentioning

confidence: 99%