Rethinking elastic online scheduling of big data streaming applications over high-velocity continuous data streams

Abstract

Online scheduling plays a key role for big data streaming applications in a big data stream computing environment, as the arrival rate of high velocity continuous data stream might fluctuate over time. In this paper, an elastic online scheduling framework for big data streaming applications (E-Stream) is proposed, exhibiting the following features. (1) Profile mathematical relationships between system response time, multiple application fairness, and online features of high velocity continuous stream. (2) Scale out or scale in a data stream graph by quantifying computation and communication cost, and the vertex semantics for arrival rate of data stream, and adjust the degree of parallelism of vertices in the graph. Sub-graph is further constructed to minimize data dependencies among the sub-graphs. (3) Elastically schedule a graph by a priority based earliest finish time first online scheduling strategy, and schedule multiple graphs by a max-min fairness strategy. (4) Evaluate the low system response time and acceptable applications fairness objectives in a realworld big data stream computing environment. Experimental results conclusively demonstrate that the proposed E-Stream provides better system response time and applications fairness compared to the existing Storm framework.

Efficient deadline-aware scheduling for the analysis of Big Data streams in public Cloud

“…The performance of BCframework is compared with Storm default scheduler under simple and complex queries because it is one of the most popular Big Data stream computing platforms both in academia and industry [30]. Furthermore, the performance of the BCframework is also compared under real-world queries with a state-of-the-art scheduler [30] because of its similarity.…”

“…The performance of BCframework is compared with Storm default scheduler under simple and complex queries because it is one of the most popular Big Data stream computing platforms both in academia and industry [30]. Furthermore, the performance of the BCframework is also compared under real-world queries with a state-of-the-art scheduler [30] because of its similarity. Similar to the problem statement of BCframework, the state-of-the-art scheduler addresses the problem of provisioning and scheduling Big Data stream applications and its framework is based on a cost and deadline aware scheduler [34].…”

“…In [30] an elastic online scheduling framework is proposed which encompasses features such as online scheduling of Big Data stream applications, profiling the mathematical relationship between system response time, multiple application fairness and high-velocity continuous stream. In comparison with BCframework, its resource model does not take into account the analytical characteristics of the streaming Big Data queries.…”

Special section: towards high performance computing in the cloud

“…The concept of elastic online scheduling of big data streaming applications with respect to high-velocity continuous data streams is discussed by Sun et al on their manuscript Rethinking Elastic Online Scheduling of Big Data Streaming Applications over High Velocity Continuous Data Streams [5]. The authors proposed E-stream, an elastic online scheduling framework for big data streaming applications based on profiling mathematical relationships between system response time, multiple application fairness and online features of high velocity.…”

Tree stream mining algorithm with Chernoff-bound and standard deviation approach for big data stream

“…They proposed the E-Stream framework, some of the features of the framework include (1) creating a mathematical relationship between the streams that occur and the response time system, (2) enlarging and improving graph data streams by calculating the costs of communication and computing, (3) elastic performs graph scheduling based on priority-based earliest finish time, (4) evaluation of the response time of the system. E-stream has better measurement results compared to the pre-existing framework storm [21].…”