A Novel Data-Centric Programming Model for Large-Scale Parallel Systems

Talia, Domenico; Trunfio, Paolo; Marozzo, Fabrizio; Belcastro, Loris; Blas, Javier García; Rio, David del; Couvée, Philippe; Goret, Gaël; Vincent, Lionel; Fernández-Pena, Alberto; Blas, Daniel Martín de; Nardi, Mirko; Pizzuti, Teresa; Spataru, Adrian; Justyna, Marek

doi:10.1007/978-3-030-48340-1_35

Cited by 7 publications

(7 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, traditional MPI all-to-all communication does not scale well in Exascale systems (i.e., highly parallel computing systems capable of at least one exaFLOPS). Hence to solve this issue new MPI releases (like MPI+X) have been proposed to support neighbor collectives for providing sparse "all-to-some" communication patterns that reduce the data exchange on limited regions of processors [45,46]. Many alternatives to MPI have been proposed in the literature, aimed at including data locality, raising the level of abstraction, as well as leveraging modern language design features.…”

Section: Message Passingmentioning

confidence: 99%

Programming big data analysis: principles and solutions

et al. 2022

Self Cite

View full text Add to dashboard Cite

In the age of the Internet of Things and social media platforms, huge amounts of digital data are generated by and collected from many sources, including sensors, mobile devices, wearable trackers and security cameras. This data, commonly referred to as Big Data, is challenging current storage, processing, and analysis capabilities. New models, languages, systems and algorithms continue to be developed to effectively collect, store, analyze and learn from Big Data. Most of the recent surveys provide a global analysis of the tools that are used in the main phases of Big Data management (generation, acquisition, storage, querying and visualization of data). Differently, this work analyzes and reviews parallel and distributed paradigms, languages and systems used today to analyze and learn from Big Data on scalable computers. In particular, we provide an in-depth analysis of the properties of the main parallel programming paradigms (MapReduce, workflow, BSP, message passing, and SQL-like) and, through programming examples, we describe the most used systems for Big Data analysis (e.g., Hadoop, Spark, and Storm). Furthermore, we discuss and compare the different systems by highlighting the main features of each of them, their diffusion (community of developers and users) and the main advantages and disadvantages of using them to implement Big Data analysis applications. The final goal of this work is to help designers and developers in identifying and selecting the best/appropriate programming solution based on their skills, hardware availability, application domains and purposes, and also considering the support provided by the developer community.

show abstract

Section: Message Passingmentioning

confidence: 99%

Programming big data analysis: principles and solutions

et al. 2022

Self Cite

View full text Add to dashboard Cite

show abstract

“…Exascale supercomputers refer to computing systems capable of at least one exaflop or a quintillion calculations per second (10 18 ). Despite their future contribution to support very large and very complex applications, exascale systems are becoming harder and harder to use efficiently (Talia et al, 2020). In particular, in the area of Big Data analysis new solutions are needed to achieve scalable software systems running quickly on exascale platforms.…”

Section: Editorial On the Research Topic Towards Exascale Solutions F...mentioning

confidence: 99%

Editorial: Towards Exascale Solutions for Big Data Computing

Talia¹,

Trunfio²,

Carretero³

et al. 2022

Front. Big Data

Self Cite

View full text Add to dashboard Cite

“…With the increasing popularity of data-intensive workflows, several research projects have been carried out to define dataaware scheduling algorithms [6] [7] [8] aiming at improving scalability, energy efficiency and execution performance. In particular, due the imminent implementation of Exascale systems, task scheduling for massively parallel applications has become an important and strategic research area [3]. In particular, several algorithms and systems have been proposed to cope with the needs of large scale data-intensive applications, exploiting both static and dynamic scheduling [9] [10] [11].…”

Section: Related Workmentioning

confidence: 99%

“…Such applications will need to avoid or limit synchronization, use less communication and remote memory, and handle software and hardware faults that can occur. In order to achieve such computational speeds, more and more novel solutions are being proposed with the aim of harnessing the computational power of a large set of machines operating in parallel [3].…”

Section: Introductionmentioning

confidence: 99%

A Data-Aware Scheduling Strategy for Executing Large-Scale Distributed Workflows

et al. 2021

Self Cite

View full text Add to dashboard Cite

Task scheduling is a crucial key component for the efficient execution of data-intensive applications on distributed environments, by which many machines must be coordinated to reduce execution times and bandwidth consumption. This paper presents ADAGE, a data-aware scheduler designed to efficiently execute data-intensive workflows in large-scale computers. The proposed scheduler is based on three key features: i) critical path analysis, for discovering the critical tasks of a workflow and reducing data transferring between nodes; ii) work giving, a new dynamic planning strategy for migrating tasks from overloaded to unloaded nodes; and iii) task replication, which executes task replicas on different nodes for improving both execution time and fault tolerance. Experiments performed on a distributed computing environment composed of up to 1,024 processing nodes show that ADAGE achieves better performances than existing scheduling systems, obtaining an average reduction of up to 66% in execution time. INDEX TERMS Data-aware scheduler, workflow scheduling, distributed workflows, parallel programming, distributed computing, exascale computing

show abstract

A Novel Data-Centric Programming Model for Large-Scale Parallel Systems

Cited by 7 publications

References 8 publications

Programming big data analysis: principles and solutions

Programming big data analysis: principles and solutions

Editorial: Towards Exascale Solutions for Big Data Computing

A Data-Aware Scheduling Strategy for Executing Large-Scale Distributed Workflows

Contact Info

Product

Resources

About