Transparent Orchestration of Task-based Parallel Applications in Containers Platforms

Ramon-Cortes, Cristian; Serven, Albert; Ejarque, Jorge; Lezzi, Daniele; Badía, Rosa M.

doi:10.1007/s10723-017-9425-z

Cited by 25 publications

(16 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Also, it offers to the programmer the view that a single shared memory space is available, and takes care of all the necessary data-transfers between the nodes of the computing infrastructure. The COMPSs runtime is able to execute the applications in supercomputers and large clusters, public, private clouds and federated clouds, containerized clusters [19], mobile environments [20] and in fog computing environments [21]. PyCOMPSs/COMPSs has been used to implement scientific workflows from life sciences [22], [23], earth sciences [24] and other disciplines [25]- [28] .…”

Section: A Scientific Workflowsmentioning

confidence: 99%

Workflow Environments for Advanced Cyberinfrastructure Platforms

Badía

Ejarque

Lordan

et al. 2019

2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS)

Self Cite

View full text Add to dashboard Cite

Progress in science is deeply bound to the effective use of high-performance computing infrastructures and to the efficient extraction of knowledge from vast amounts of data. Such data comes from different sources that follow a cycle composed of pre-processing steps for data curation and preparation for subsequent computing steps, and later analysis and analytics steps applied to the results. However, scientific workflows are currently fragmented in multiple components, with different processes for computing and data management, and with gaps in the viewpoints of the user profiles involved. Our vision is that future workflow environments and tools for the development of scientific workflows should follow a holistic approach, where both data and computing are integrated in a single flow built on simple, high-level interfaces. The topics of research that we propose involve novel ways to express the workflows that integrate the different data and compute processes, dynamic runtimes to support the execution of the workflows in complex and heterogeneous computing infrastructures in an efficient way, both in terms of performance and energy. These infrastructures include highly distributed resources, from sensors and instruments, and devices in the edge, to High-Performance Computing and Cloud computing resources.This paper presents our vision to develop these workflow environments and also the steps we are currently following to achieve it.

show abstract

Section: A Scientific Workflowsmentioning

confidence: 99%

Workflow Environments for Advanced Cyberinfrastructure Platforms

Badía

Ejarque

Lordan

et al. 2019

2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS)

Self Cite

View full text Add to dashboard Cite

show abstract

“…COMPSs is a framework in constant development, it is receiving several new extensions and APIs to fit Big Data requirements, that vary from cloud connectors [9] to a resource manager integration [10]. In a recent work [3], COMPSs Storage API is presented, an official software interface that allows COMPSs applications and COMPSs runtime to work with persistent objects.…”

Section: Related Workmentioning

confidence: 99%

Upgrading a high performance computing environment for massive data processing

Ponce

Santos

Meira

et al. 2019

J Internet Serv Appl

Self Cite

View full text Add to dashboard Cite

High-performance computing (HPC) and massive data processing (Big Data) are two trends that are beginning to converge. In that process, aspects of hardware architectures, systems support and programming paradigms are being revisited from both perspectives. This paper presents our experience on this path of convergence with the proposal of a framework that addresses some of the programming issues derived from such integration. Our contribution is the development of an integrated environment that integretes (i) COMPSs, a programming framework for the development and execution of parallel applications for distributed infrastructures; (ii) Lemonade, a data mining and analysis tool; and (iii) HDFS, the most widely used distributed file system for Big Data systems. To validate our framework, we used Lemonade to create COMPSs applications that access data through HDFS, and compared them with equivalent applications built with Spark, a popular Big Data framework. The results show that the HDFS integration benefits COMPSs by simplifying data access and by rearranging data transfer, reducing execution time. The integration with Lemonade facilitates COMPSs's use and may help its popularization in the Data Science community, by providing efficient algorithm implementations for experts from the data domain that want to develop applications with a higher level abstraction.

show abstract

“…Moreover, even though Docker containers are used and managed using Amazon ECS, the underlying infrastructure still uses EC2, which is Xen-based and thus, in the end, container virtualization is placed on top of hypervisor-based virtualization. A recent related study has been conducted in [31], where parallel applications deployed on containers and using di↵erent managers including Mesos are compared to a cloud deployment with OpenStack and a bare-metal scenario. The results show that indeed Docker containers are closer to bare-metal when it comes to storage e ciency and have significantly lower overheads than the KVM hypervisor.…”

Section: Related Workmentioning

confidence: 99%

Big Data-Oriented PaaS Architecture with Disk-as-a-Resource Capability and Container-Based Virtualization

et al. 2018

View full text Add to dashboard Cite

With the increasing adoption of Big Data technologies as basic tools for the ongoing Digital Transformation, there is a high demand for data-intensive applications. In order to e ciently execute such applications, it is vital that cloud providers change the way hardware infrastructure resources are managed to improve their performance. However, the increasing use of virtualization technologies to achieve an e cient usage of infrastructure resources continuously widens the gap between applications and the underlying hardware, thus decreasing resource e ciency for the end user. Moreover, this scenario is especially troublesome for Big Data applications, as storage resources are one of the most heavily virtualized, thus imposing a significant overhead for large-scale data processing. This paper proposes a novel PaaS architecture specifically oriented for Big Data where the scheduler o↵ers disks as resources alongside the more common CPU and memory resources, looking forward to provide a better storage solution for the user. Furthermore, virtualization overheads are reduced to the bare minimum by replacing heavy hypervisor-based technologies with operating-system-level virtualization based on light software containers. This architecture has been deployed on a Big Data infrastructure at the CESGA supercomputing center, used as a testbed to compare its performance with OpenStack, a popular private cloud platform. Results have shown significant performance improvements, reducing the execution time of representative Big Data workloads by up to 4.5x.

show abstract

Transparent Orchestration of Task-based Parallel Applications in Containers Platforms

Cited by 25 publications

References 24 publications

Workflow Environments for Advanced Cyberinfrastructure Platforms

Workflow Environments for Advanced Cyberinfrastructure Platforms

Upgrading a high performance computing environment for massive data processing

Big Data-Oriented PaaS Architecture with Disk-as-a-Resource Capability and Container-Based Virtualization

Contact Info

Product

Resources

About