Enabling Python to execute efficiently in heterogeneous distributed infrastructures with PyCOMPSs

Amela, Ramon; Ramon-Cortes, Cristian; Ejarque, Jorge; Conejero, Javier; Badía, Rosa M.

doi:10.1145/3149869.3149870

Cited by 9 publications

(11 citation statements)

References 17 publications

(17 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Thus, to parallelise a vector operation, the users must explicitly define the loop nests to apply an operation to all the vector elements. However, in our previous work (Amela et al, 2017(Amela et al, , 2018, we have demonstrated the benefits from combining inter-and intra-node parallelism using PyCOMPSs (Tejedor et al, 2017) and NumPy. Similarly, some NumPy extensions can be integrated with PyCOMPSs to boost the intra-node performance.…”

Section: State Of the Artmentioning

confidence: 99%

“…Therefore, the performance evaluation of PyCOMPSs and its Runtime is beyond the scope of this paper. For further details, in our previous work (Amela et al, 2017(Amela et al, , 2018, we analysed indepth the performance obtained when executing linear algebra applications when combining PyCOMPSs for inter-node parallelism and NumPy for intra-node parallelism. Also, in Conejero et al (2018), we compared the PyCOMPSs Runtime against Apache Spark.…”

Section: Performance Evaluationmentioning

confidence: 99%

See 1 more Smart Citation

AutoParallel: Automatic parallelisation and distributed execution of affine loop nests in Python

Ramon-Cortes

Amela

Ejarque

et al. 2020

The International Journal of High Performance Computing Applica

Self Cite

View full text Add to dashboard Cite

The last improvements in programming languages and models have focused on simplicity and abstraction; leading Python to the top of the list of the programming languages. However, there is still room for improvement when preventing users from dealing directly with distributed and parallel computing issues. This paper proposes and evaluates AutoParallel, a Python module to automatically find an appropriate task-based parallelisation of affine loop nests and execute them in parallel in a distributed computing infrastructure. It is based on sequential programming and contains one single annotation (in the form of a Python decorator) so that anyone with intermediate-level programming skills can scale up an application to hundreds of cores. The evaluation demonstrates that AutoParallel goes one step further in easing the development of distributed applications. On the one hand, the programmability evaluation highlights the benefits of using a single Python decorator instead of manually annotating each task and its parameters or, even worse, having to develop the parallel code explicitly (e.g., using OpenMP, MPI). On the other hand, the performance evaluation demonstrates that AutoParallel is capable of automatically generating task-based workflows from sequential Python code while achieving the same performances than manually taskified versions of established state-of-the-art algorithms (i.e., Cholesky, LU, and QR decompositions). Finally, AutoParallel is also capable of automatically building data blocks to increase the tasks’ granularity; freeing the user from creating the data chunks, and re-designing the algorithm. For advanced users, we believe that this feature can be useful as a baseline to design blocked algorithms.

show abstract

Section: State Of the Artmentioning

confidence: 99%

Section: Performance Evaluationmentioning

confidence: 99%

AutoParallel: Automatic parallelisation and distributed execution of affine loop nests in Python

Ramon-Cortes

Amela

Ejarque

et al. 2020

The International Journal of High Performance Computing Applica

Self Cite

View full text Add to dashboard Cite

show abstract

“…... COMPSs provides Java as native programming language and it also provides bindings for Python (PyCOMPSs [2]) and C/C++ [7]. Figure 1 shows an example of a task annotation and COMPSs main program.…”

Section: Compss Overviewmentioning

confidence: 99%

Managing Failures in Task-Based Parallel Workflows in Distributed Computing Environments

Ejarque

Trapero-Bertran

Cid-Fuentes

et al. 2020

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

Current scientific workflows are large and complex. They normally perform thousands of simulations whose results combined with searching and data analytics algorithms, in order to infer new knowledge, generate a very large amount of data. To this end, workflows comprise many tasks and some of them may fail. Most of the work done about failure management in workflow managers and runtimes focuses on recovering from failures caused by resources (retrying or resubmitting the failed computation in other resources, etc.) However, some of these failures can be caused by the application itself (corrupted data, algorithms which are not converging for certain conditions, etc.), and these fault tolerance mechanisms are not sufficient to perform a successful workflow execution. In these cases, developers have to add some code in their applications to prevent and manage the possible failures. In this paper, we propose a simple interface and a set of transparent runtime mechanisms to simplify how scientists deal with application-based failures in taskbased parallel workflows. We have validated our proposal with use-cases from e-science and machine learning to show the benefits of the proposed interface and mechanisms in terms of programming productivity and performance.

show abstract

“…Finally, other libraries and frameworks enable Python distributed and multi-threaded computations such as Dask [14], PySpark [15], and PyCOMPSs [16], [17]. Dask is a native Python library that allows both the creation of custom DAG's and the distributed execution of a set of operations on NumPy and pandas [18] objects.…”

Section: State Of the Artmentioning

confidence: 99%

AutoParallel: A Python module for automatic parallelization and distributed execution of affine loop nests

Ramon-Cortes¹,

Amela²,

Ejarque³

et al. 2018

Preprint

Self Cite

View full text Add to dashboard Cite

The last improvements in programming languages, programming models, and frameworks have focused on abstracting the users from many programming issues. Among others, recent programming frameworks include simpler syntax, automatic memory management and garbage collection, which simplifies code re-usage through library packages, and easily configurable tools for deployment. For instance, Python has risen to the top of the list of the programming languages [1] due to the simplicity of its syntax, while still achieving a good performance even being an interpreted language. Moreover, the community has helped to develop a large number of libraries and modules, tuning them to obtain great performance.However, there is still room for improvement when preventing users from dealing directly with distributed and parallel computing issues. This paper proposes and evaluates AutoParallel, a Python module to automatically find an appropriate task-based parallelization of affine loop nests to execute them in parallel in a distributed computing infrastructure. This parallelization can also include the building of data blocks to increase task granularity in order to achieve a good execution performance. Moreover, AutoParallel is based on sequential programming and only contains a small annotation in the form of a Python decorator so that anyone with little programming skills can scale up an application to hundreds of cores.

show abstract

Enabling Python to execute efficiently in heterogeneous distributed infrastructures with PyCOMPSs

Cited by 9 publications

References 17 publications

AutoParallel: Automatic parallelisation and distributed execution of affine loop nests in Python

AutoParallel: Automatic parallelisation and distributed execution of affine loop nests in Python

Managing Failures in Task-Based Parallel Workflows in Distributed Computing Environments

AutoParallel: A Python module for automatic parallelization and distributed execution of affine loop nests

Contact Info

Product

Resources

About