Towards optimising distributed data streaming graphs using parallel streams

Abstract

Modern scientific collaborations have opened up the opportunity of solving complex problems that involve multidisciplinary expertise and large-scale computational experiments. These experiments usually involve large amounts of data that are located in distributed data repositories running various software systems, and managed by different organisations. A common strategy to make the experiments more manageable is executing the processing steps as a workflow. In this paper, we look into the implementation of fine-grained data-flow between computational elements in a scientific workflow as streams. We model the distributed computation as a directed acyclic graph where the nodes represent the processing elements that incrementally implement specific subtasks. The processing elements are connected in a pipelined streaming manner, which allows task executions to overlap. We further optimise the execution by splitting pipelines across processes and by introducing extra parallel streams. We identify performance metrics and design a measurement tool to evaluate each enactment. We conducted experiments to evaluate our optimisation strategies with a real world problem in the Life Sciences-EURExpress-II. The paper presents our distributed data-handling model, the optimisation and instrumentation strategies and the evaluation experiments. We demonstrate linear speed up and argue that this use of data-streaming to enable both overlapped pipeline and parallelised enactment is a generally applicable optimisation strategy.

Performance database: capturing data for optimizing distributed streaming workflows

“…Thus, a good optimization choice is to avoid assigning these PEIs on separate DIVMs unless the competition for the CPU or RAM outweighs the transfer cost. We have achieved a linear speed-up in optimizing this workflow, as reported in Liew et al [2] and Han et al [9].…”

Introducing distributed dynamic data‐intensive (D3) science: Understanding applications and infrastructure

“…Doing so would bring immediate benefits in terms of reproducibility, but also in terms of standardization of components and interfaces. 38,60,61 There are many examples of stream [62][63][64] and event 41,65,66 processing frameworks, and recent works have combined streaming with more traditional file-based workflows. 67,68 Applications such as the marine sensor network (Section 2.6), distributed network intrusion detection (Section 2.13), power grids (Section 2.9), and industrial incident notification and response (Section 2.11) use stream and event processing abstractions.…”

Cost-Aware Streaming Workflow Allocation on Geo-Distributed Data Centers

“…Besid es, to achieve high energy efficiency and low response tim e in big d ata stream com pu ting environm ents, the au thors of [2] and [3] propose a real-tim e and energy-efficient resou rce sched u ling and optim ization fram ew ork, term ed the Re-Stream , w hich aid s in calcu lating the energy consu m ption of a resou rce allocation schem e for a d ata stream graph. What is m ore, a partitioning-based d ataintensive w orkflow optim ization algorithm [37], [39] has been proposed to provid e significantly red u ced latency w ith increase in the throu ghpu t. H ow ever, the issu e of VM allocation has not been properly ad d ressed in geod istribu ted DCs for stream ing w orkflow. Only a sm all nu m ber of w orks ad d ress VM allocation in clou d com p uting system s for stream ing big d ata processing.…”