File-access patterns of data-intensive workflow applications and their implications to distributed filesystems

Shibata, Takeshi; Choi, SungJun; Taura, Kenjiro

doi:10.1145/1851476.1851585

Cited by 26 publications

(12 citation statements)

References 28 publications

(33 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To this end, this section focuses on pipeline, reduce, and broadcast patterns ( Figure 2). These are among the most used patterns uncovered by studying over 20 scientific workflow applications by Wozniak et al [20], Shibata et al [15], and Bharathi et al [8]). We note that, these benchmarks are designed to explore the limitations of the predictor and be a worst case in terms of accuracy , and are harder to predict accurately than real applications as they are composed exclusively of I/O operations, which leads to contention in the real storage system.…”

Section: Synthetic Benchmarks For Workflow Patternsmentioning

confidence: 89%

See 1 more Smart Citation

Predicting intermediate storage performance for workflow applications

Costa

Al-Kiswany

Barros

et al. 2013

Proceedings of the 8th Parallel Data Storage Workshop

View full text Add to dashboard Cite

System configuration decisions for I/O-intensive workflow applications can be complex even for expert users. Users face decisions to configure several parameters optimally (e.g., replication level, chunk size, number of storage node) -each having an impact on overall application performance. This paper presents our progress on addressing the problem of supporting storage system configuration decisions for workflow applications. Our approach accelerates the exploration of the configuration space based on a low-cost performance predictor that estimates turn-around time of a workflow application in a given setup. Our evaluation shows that the predictor is effective in identifying the desired system configuration, and it is lightweight using 2000-5000× less resources (machines × time) than running the actual benchmarks.

show abstract

Section: Synthetic Benchmarks For Workflow Patternsmentioning

confidence: 89%

“…Assembling workflow applications by putting together standalone binaries has become a popular approach to support large-scale science [8,15,20]. The processes spawned from these binaries communicate via temporary files stored on a shared storage system.…”

Section: Introductionmentioning

confidence: 99%

Predicting intermediate storage performance for workflow applications

Costa

Al-Kiswany

Barros

et al. 2013

Proceedings of the 8th Parallel Data Storage Workshop

View full text Add to dashboard Cite

show abstract

“…Not unlike many software applications, workflows involve several, often disparate systems or components which individually analyze input data, make specific calculations, and produce output data which is often then used by another component in the workflow [8]. Shibata presents a study of five workflow applications and their associated costs [12]. Deelman studied costs of scientific workflows running in the cloud [6], and Dun profiled data intensive workflows in a similar way to the detection approach for measurement covered in the current work [7].…”

Section: Related Workmentioning

confidence: 99%

Dynamic cost verification for cloud applications

Buell

Collofello

2012

Proceedings of the Ninth International Workshop on Dynamic Analysis

View full text Add to dashboard Cite

The pay-as-you-go economic model of cloud computing increases the visibility, traceability, and verifiability of software costs. Application developers must understand how their software uses resources when running in the cloud in order to stay within budgeted costs and/or produce expected profits. Scientific workflows often involve data intensive transactions which may be costly. Business and consumer application developers are likely to be particularly sensitive to costs in order to maximize profits. Verification of economic attributes of cloud applications has only been touched on lightly in the literature to date. Possibilities for cost verification of cloud applications include both static and dynamic analysis. We advocate for increased attention to economic attributes of cloud applications at every level of software development, and we discuss some measurement based approaches to cost verification of applications running in the cloud.

show abstract

“…(a) hardware design or architectural improvements [2], [3], (b) new storage and memory management techniques [4], [5], and (c) algorithms to optimize dataintensive applications [6], [7]. The science of workflows has emerged to simplify the complex scientific processes by step-wise representation in the form of workflows [8].…”

Section: Introductionmentioning

confidence: 99%

Data-Intensive Workflow Optimization Based on Application Task Graph Partitioning in Heterogeneous Computing Systems

Ahmad

Liew

Rafique

et al. 2014

2014 IEEE Fourth International Conference on Big Data and Cloud Computing

View full text Add to dashboard Cite

Abstract-Stream based data processing model is proven to be an established method to optimize data-intensive applications. Data-intensive applications involve movement of huge amount of data between execution nodes that incurs large costs. Data-streaming model improves the execution performance of such applications. In the stream-based data processing model, performance is usually measured by throughput and latency. Optimization of these performance metrics in heterogeneous computing environment becomes more challenging due to the difference in the computing capacity of execution nodes and variations in the data transfer capability of communication links between these nodes. This paper presents a dual objective Partitioning based Data-intensive Workflow optimization Algorithm (PDWA) for heterogeneous computing systems. The proposed PDWA provides significantly reduced latency with increase in the throughput. In the proposed algorithm, the application task graph is partitioned such that the interpartition data movement is minimal. Such optimized partitioning enhances the throughput. Each partition is mapped to the execution node that gives minimum execution time for that particular partition. PDWA also exploits partial task duplication to reduce the latency. We evaluated the proposed algorithm with synthesized benchmarks and workflows from the real-world workloads, and the proposed algorithm shows 60% reduced latency with 47% improvement in the throughput as compared to the approach when workflows are not partitioned.

show abstract

File-access patterns of data-intensive workflow applications and their implications to distributed filesystems

Cited by 26 publications

References 28 publications

Predicting intermediate storage performance for workflow applications

Predicting intermediate storage performance for workflow applications

Dynamic cost verification for cloud applications

Data-Intensive Workflow Optimization Based on Application Task Graph Partitioning in Heterogeneous Computing Systems

Contact Info

Product

Resources

About