A workflow-enabled big data analytics software stack for escience

Palazzo, Cosimo; Mariello, Andrea; Fiore, Sandro; D'Anca, Alessandro; Elia, Donatello; Williams, D. N.; Aloisio, Giovanni

doi:10.1109/hpcsim.2015.7237088

Cited by 12 publications

(10 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…First, a self join is applied on the row delta tables based on the id column of both "DDB1" and "DDB2" tables. Note that the id column should be provided by the user due to the lack of primary keys when (9) Total number of differences in data (10) Maximum difference in data (11) Minimum difference in data (12) First quartile (25th percentile) (13) Second quartile (50th percentile) − median (14) Third quartile (75th percentile) Fig. 9 The column validation step Fig.…”

Section: Column Validationmentioning

confidence: 99%

“…Column statistics (12), (13), and (14): Hive provides a function called percentile that can be used to calculate the lowest and highest quartile of a data set along with the median value "percentile (BIGINT col, array (p1 [, p2] ... ))" [34]. This function calculates the specified percentiles for a data set, which is the list of differences for each column in this case.…”

Section: Creatementioning

confidence: 99%

“…Companies develop workflows (data transformations) to process data continuously. Such workflows define and manage a series of steps that process and transform data [12]. They then store the new results in databases to be used for planning and decision-making, or as inputs to other workflows.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Diftong: a tool for validating big data workflows

et al. 2019

View full text Add to dashboard Cite

Data validation is about verifying the correctness of data. When organisations update and refine their data transformations to meet evolving requirements, it is imperative to ensure that the new version of a workflow still produces the correct output. We motivate the need for workflows and describe the implementation of a validation tool called Diftong. This tool compares two tabular databases resulting from different versions of a workflow to detect and prevent potential unwanted alterations. Row-based and column-based statistics are used to quantify the results of the database comparison. Diftong was shown to provide accurate results in test scenarios, bringing benefits to companies that need to validate the outputs of their workflows. By automating this process, the risk of human error is also eliminated. Compared to the more labourintensive manual alternative, it has the added benefit of improved turnaround time for the validation process. Together this allows for a more agile way of updating data transformation workflows.

show abstract

Section: Column Validationmentioning

confidence: 99%

Section: Creatementioning

confidence: 99%

See 1 more Smart Citation

Diftong: a tool for validating big data workflows

et al. 2019

View full text Add to dashboard Cite

show abstract

“…The Ophidia workflow management system [35] is a core component of the Ophidia platform. It allows coordinating and orchestrating the execution of scientific experiments composed of multiple data analytics, processing and visualization operators (e.g.…”

Section: Implementation Detailsmentioning

confidence: 99%

Towards an Open (Data) Science Analytics-Hub for Reproducible Multi-Model Climate Analysis at Scale

Fiore

Elia

Palazzo

et al. 2018

2018 IEEE International Conference on Big Data (Big Data)

View full text Add to dashboard Cite

Open Science is key to future scientific research and promotes a deep transformation in the whole scientific research process encouraging the adoption of transparent and collaborative scientific approaches aimed at knowledge sharing. Open Science is increasingly gaining attention in the current and future research agenda worldwide. To effectively address Open Science goals, besides Open Access to results and data, it is also paramount to provide tools or environments to support the whole research process, in particular the design, execution and sharing of transparent and reproducible experiments, including data provenance (or lineage) tracking. This work introduces the Climate Analytics-Hub, a new component on top of the Earth System Grid Federation (ESGF), which joins big data approaches and parallel computing paradigms to provide an Open Science environment for reproducible multi-model climate change data analytics experiments at scale. An operational implementation has been set up at the SuperComputing Centre of the Euro-Mediterranean Center on Climate Change, with the main goal of becoming a reference Open Science hub in the climate community regarding the multi-model analysis based on the Coupled Model Intercomparison Project (CMIP).

show abstract

“…Jobs () Apache Spark [12] Runtime RDD () Fireworks [13] Distributed -() Pegasus [14] (Dist.) Jobs -TaskFarmer [15] Commands Files/Shards Tigres [16] Runtime "Inputs" -Ophidea [17] Runtime Datasets Kepler [18] Runtime () -…”

Section: B Common Hpc Workflowsmentioning

confidence: 99%

Toward Understanding I/O Behavior in HPC Workflows

Lüttgau¹,

Snyder²,

Carns³

et al. 2018

2018 IEEE/ACM 3rd International Workshop on Parallel Data Storage &Amp; Data Intensive Scalable Computing Systems (PDSW-DISCS)

View full text Add to dashboard Cite

Scientific discovery increasingly depends on complex workflows consisting of multiple phases and sometimes millions of parallelizable tasks or pipelines. These workflows access storage resources for a variety of purposes, including preprocessing, simulation output, and postprocessing steps. Unfortunately, most workflow models focus on the scheduling and allocation of computational resources for tasks while the impact on storage systems remains a secondary objective and an open research question. I/O performance is not usually accounted for in workflow telemetry reported to users.In this paper, we present an approach to augment the I/O efficiency of the individual tasks of workflows by combining workflow description frameworks with system I/O telemetry data. A conceptual architecture and a prototype implementation for HPC data center deployments are introduced. We also identify and discuss challenges that will need to be addressed by workflow management and monitoring systems for HPC in the future. We demonstrate how real-world applications and workflows could benefit from the approach, and we show how the approach helps communicate performance-tuning guidance to users.

show abstract

A workflow-enabled big data analytics software stack for escience

Cited by 12 publications

References 10 publications

Diftong: a tool for validating big data workflows

Diftong: a tool for validating big data workflows

Towards an Open (Data) Science Analytics-Hub for Reproducible Multi-Model Climate Analysis at Scale

Toward Understanding I/O Behavior in HPC Workflows

Contact Info

Product

Resources

About