Data-Intensive Workflow Optimization Based on Application Task Graph Partitioning in Heterogeneous Computing Systems

Abstract

Abstract-Stream based data processing model is proven to be an established method to optimize data-intensive applications. Data-intensive applications involve movement of huge amount of data between execution nodes that incurs large costs. Data-streaming model improves the execution performance of such applications. In the stream-based data processing model, performance is usually measured by throughput and latency. Optimization of these performance metrics in heterogeneous computing environment becomes more challenging due to the difference in the computing capacity of execution nodes and variations in the data transfer capability of communication links between these nodes. This paper presents a dual objective Partitioning based Data-intensive Workflow optimization Algorithm (PDWA) for heterogeneous computing systems. The proposed PDWA provides significantly reduced latency with increase in the throughput. In the proposed algorithm, the application task graph is partitioned such that the interpartition data movement is minimal. Such optimized partitioning enhances the throughput. Each partition is mapped to the execution node that gives minimum execution time for that particular partition. PDWA also exploits partial task duplication to reduce the latency. We evaluated the proposed algorithm with synthesized benchmarks and workflows from the real-world workloads, and the proposed algorithm shows 60% reduced latency with 47% improvement in the throughput as compared to the approach when workflows are not partitioned.

Cost-Aware Streaming Workflow Allocation on Geo-Distributed Data Centers

“…Besid es, to achieve high energy efficiency and low response tim e in big d ata stream com pu ting environm ents, the au thors of [2] and [3] propose a real-tim e and energy-efficient resou rce sched u ling and optim ization fram ew ork, term ed the Re-Stream , w hich aid s in calcu lating the energy consu m ption of a resou rce allocation schem e for a d ata stream graph. What is m ore, a partitioning-based d ataintensive w orkflow optim ization algorithm [37], [39] has been proposed to provid e significantly red u ced latency w ith increase in the throu ghpu t. H ow ever, the issu e of VM allocation has not been properly ad d ressed in geod istribu ted DCs for stream ing w orkflow. Only a sm all nu m ber of w orks ad d ress VM allocation in clou d com p uting system s for stream ing big d ata processing.…”

“…The stream ing w orkflow has also been stu d ied on distribu ted system s [35], [36], [37], [38] for years. For example, the au thors of [35] and [36] propose a sem anticsbased ap proach for the m anagem ent of fast d ata stream s, aim ing to provid e a d escription and m anagem ent layer to d efine and execu te stream processing pipelines.…”

Multiple Workflows Scheduling in Multi-tenant Distributed Systems

“…et al this information as being implemented in static work ow scheduling is not plausible. For example, a strategy of partitioning tasks before runtime in a static work ow scheduling to minimize the data transfer is proven to be e cient for data-intensive work ows (Ahmad et al 2014). is strategy actually can be done in multi-tenant platforms, but then, it becomes an inevitable bo leneck since the time required for partitioning a work ow may delay the next queue of arriving work ows for scheduling.…”

Optimization Strategies for Integration Pattern Compositions

“…Our main contribution is a graph-based representation of integration patterns, so that optimizations can be realized as graph rewriting rules. To show the feasibility of our framework, we analyzed its use on a catalog of over 900 real-world [44], optimization patterns [31,32] Workflow Optimization 396 6 data-aware processes instance scheduling [2,7,43], scheduling and partitioning for interaction [3], scheduling and placement [6], operator merge [22] Data Integration Optimization 61 2 data-aware processes optimization, (no schema-matching)…”