Streamline: scheduling streaming applications in a wide area environment

Abstract

Scheduling a streaming application on high-performance computing (HPC) resources has to be sensitive to the computation and communication needs of each stage of the application dataflow graph to ensure QoS criteria such as latency and throughput. Since the grid has evolved out of traditional high-performance computing, the tools available for scheduling are more appropriate for batch-oriented applications. Our scheduler, called Streamline, considers the dynamic nature of the grid and runs periodically to adapt scheduling decisions using application requirements (perstage computation and communication needs), application constraints (such as co-location of stages), and resource availability. The performance of Streamline is compared with an Optimal placement, Simulated Annealing (SA) approximations, and E-Condor, a streaming grid scheduler built using Condor. For kernels of streaming applications, we show that Streamline performs close to the Optimal and SA algorithms, and an order of magnitude better than E-Condor under nonuniform load conditions. We also conduct scalability studies An earlier version of this paper appeared in [1]. This paper includes Sect. 6 on experiments using wide area environment. We describe our experience implementing Streamline scheduler as a grid service on Planetlab. We also present our experimental results on Planetlab in Sect. 6. We update related work in Sect. 7.showing the advantage of Streamline over other approaches. Furthermore, we implement Streamline on Planetlab as a grid service and demonstrate that it performs close to SA algorithm under dynamic resource conditions.

Dynamic Control of Data Streaming and Processing in a Virtualized Environment

“…Tools including Eucalyptus [16], VMPlants [17], and Usher [18] can serve this management purpose. Some schedulers are developed to support data streaming applications, e.g., GATES [19] and Streamline [20], but they mainly concern on computing resource allocation without taking bandwidth into account. Several projects such as EnLIGHTened [21], G-lambda [22], and PHOSPHORUS [23] put emphases on networking resources.…”

dispel4py: A Python framework for data-intensive scientific computing

“…The optimisation of data-streaming workflows has been investigated recently (Agarwalla et al, 2007;Agrawal et al, 2012;Guirado et al, 2013;Ahmad et al, 2014). This has addressed the substantial challenges of task distribution, data dependencies, and inter-task data movement at large scale, since they can become a bottleneck, as Wozniak reports (Wozniak et al, 2013).…”

Agile Data Streaming for Grid Applications

“…By carefully optimizing data streaming, our environment makes required data available in an on-demand and just-in-time manner. Streamline [1,2]. Streamline schedules streaming applications on HPC resources using heuristic methods.…”