Towards DaaS 2.0: Enriching Data Models

Marti, Jonathan; Gasull, Daniel; Queralt, Anna; Cortes, Toni

doi:10.1109/services.2013.59

Cited by 4 publications

(2 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…By default, COMPSs works standalone with local storage, but it also provides support for interaction with existing storage technologies, such as dataClay (Marti et al, 2013) and Hecuba (Tejedor et al, 2015), through a simple API. In addition to the exploitation of Grids, Clusters, and Cloud environments, COMPSs contains specific connectors to enable its deployment in Dockers, and is currently working on the integration with Mesos.…”

Section: Compssmentioning

confidence: 99%

Task-based programming in COMPSs to converge from HPC to big data

Conejero

Corella

Badía

et al. 2017

The International Journal of High Performance Computing Applica

View full text Add to dashboard Cite

Abstract-Task-based programming has proven to be a suitable model for high-performance computing (HPC) applications. Different implementations have been good demonstrators of this fact, and have promoted the acceptance of task-based programming in the OpenMP standard.Furthermore, in recent years, Apache Spark has gained wide popularity in business and research environments as a programming model for addressing emerging Big-Data problems. COMP Superscalar (COMPSs) is a task-based environment that tackles distributed computing (including Clouds), and is a good alternative for a task-based programming model for Big Data applications.This paper describes why we consider that task-based programming models are a good approach for Big Data applications. The paper includes a comparison of Spark and COMPSs in terms of architecture, programming model and performance. It focuses on the differences that both frameworks have in structural terms, on their programmability interface, and in terms of their efficiency by means of three widely known benchmarking kernels: Wordcount, Kmeans and Terasort. These kernels enable the evaluation of the more important functionalities of both programming models and analyse different workflows and conditions.The main results achieved from this comparison are: (1) COMPSs is able to extract the inherent parallelism from the user code with minimal coding effort as opposed to Spark, which requires the existing algorithms to be adapted and rewritten by explicitly using their pre-defined functions; (2) it is an improvement in terms of performance when compared with Spark, and (3) COMPSs has shown to scale better than Spark in most cases.Finally, we discuss the advantages and disadvantages of both frameworks, highlighting the differences that make them unique, thereby helping to choose the right framework for each particular objective.

show abstract

Section: Compssmentioning

confidence: 99%

Task-based programming in COMPSs to converge from HPC to big data

Conejero

Corella

Badía

et al. 2017

The International Journal of High Performance Computing Applica

View full text Add to dashboard Cite

show abstract

“…A new type of consumerproducer applications can be designed, where the data can be stored in these persistent storage and be accessed during the execution of the producer application or after. While this data can be stored in files or databases, both are designed to use block devices, while this type of storage supports other alternatives, as direct object storage [36].In the environment described before, persistent storage can be used to store the results of simulations. The data can be consumed by the analytic services as soon as it has been produced and the results of the analytic steps can also be stored in persistent storage, in order to be used in visualization steps or in future queries.…”

mentioning

confidence: 99%

Workflows for science: a challenge when facing the convergence of HPC and Big Data

Badía

Ayguadé

Labarta

2017

JSFI

View full text Add to dashboard Cite

Workflows have been traditionally a mean to describe and implement the computing experiments, usually parametric studies and explorations searching for the best solution, that scientific researchers want to perform. A workflow is not only the computing application, but a way of documenting a process. Science workflows may be of very different nature depending on the area of research, matching the actual experiment that the scientist want to perform. Workflow Management Systems are environments that offer the researchers tools to define, publish, execute and document their workflows.In some cases, the science workflows are used to generate data; in other cases are used to analyse existing data; only in a few cases, workflows are used both to generate and analyse data. The design of experiments is in some cases generated blindly, without a clear idea of which points are relevant to be computed/simulated, ending up with huge amount of computation that is performed following a brute-force strategy.However, the evolution of systems and the large amount of data generated by the applications require an in-situ analysis of the data, thus requiring new solutions to develop workflows that includes both the simulation/computational part and the analytic part. What is more, the fact that both components, computation and analytics, can be run together will enable the possibility of defining more dynamic workflows, with new computations being decided by the analytics in a more efficient way.The first part of the paper will review current approaches that a set of scientific communities follow in the development of their workflows. This paper does not intent to be exhaustive in the compilation of different approaches available to develop and deploy workflows. We focus on the Workflow Management Systems used by a set of scientific communities and their representative use cases, with the objective of understanding their different needs and requirements. The second part of the paper proposes a new software architecture to develop a new family of end-to-end workflows that enables the management of dynamic workflows composed of simulations, analytics and visualization, including inputs/outputs from streams.Keywords: workflows, scientific applications, Big Data. IntroductionWorkflows appeared last century and have been used in the manufacturing industry as a mean to optimize their processes. Examples of traditional (non-IT) workflows can be found in the assembly lines, i.e., the Ford Model T assembly line standardized the production processes and was the first continuous delivery pipeline for the automotive industry. This process reduced the costs of manufacturing from $850 to $260 in 1924.The time and motion studies defined by Taylor [52] and Gilbreth [41] had significant impact in the manufacturing processes. These studies proposed to break manufacturing activities into small, simple steps, to determine with accuracy the amount of time required to perform each of the steps. Then, the sequence of movements taken by the employee has...

show abstract