Accelerating Distributed Workflows with Edge Resources

Abstract

Distributed data-intensive workflow applications are increasingly relying on and integrating remote resources including community data sources, services, and computational platforms. Increasingly, these are made available as data, SAAS, and IAAS clouds. The execution of distributed data-intensive workflow applications can expose network bottlenecks between clouds that compromise performance. In this paper, we focus on alleviating network bottlenecks by using a proxy network. In particular, we show how proxies can eliminate network bottlenecks by smart routing and perform in-network computations to boost workflow application performance. A novel aspect of our work is the inclusion of multiple proxies to accelerate different workflow stages optimizing different performance metrics. We show that the approach is effective for workflow applications and broadly applicable.Using Montage 1 as an exemplar workflow application, results obtained through experiments on PlanetLab showed how different proxies acting in a variety of roles can accelerate distinct stages of Montage. Our microbenchmarks also show that routing data through select proxies can accelerate network transfer for TCP/UDP bandwidth, delay, and jitter, in general.

Scientific Workflow Makespan Minimization In Edge Multiple Service Providers Environment

“…The highest priority task will be executed first. In [20] author tries to improve the network bottleneck using a proxy network. Through smart routing, the author proves that network bottleneck can be reduced, and workflow application performance increases significantly.…”

Multi-objective scheduling of extreme data scientific workflows in Fog

“…However, none of these works consider the context of scientific workflows. In [54], the benefits of accelerating execution of scientific workflows by employing Fog resources are discussed, motivating our work. Typical examples of scientifically workflows are described in [25,55].…”

Autonomic fault tolerant scheduling approach for scientific workflows in Cloud computing

“…Concurrent engineering is a workflow (Addo-Tenkorang, 2011) being used for parallel execution of independent tasks in scientific workflows which can comprise few tasks to millions of tasks (Bouikni et al, 2008). For large workflows, various tasks need to be distributed and parallelized among multiple resources in Cloud to complete the task in a reasonable time (Kara et al, 2001; Ramakrishnan et al, 2013). Cloud infrastructures have also been evaluated as an execution platform for scientific workflows and support all the techniques which had already been implemented in Grids and Clusters.…”