Dynamically Scaling Apache Storm for the Analysis of Streaming Data

Abstract

Stream processing platforms allow applications to analyse incoming data continuously. Several use cases exist that make use of these capabilities, ranging from monitoring of physical infrastructures to pre selecting video surveillance feeds for human inspection. It is difficult to predict how much computing resources are needed for these stream processing platforms, because the volume and velocity of input data may vary over time. The open source Apache Storm software provides a framework for developers to build processing applications that use the computing resources of all machines within an established cluster. Because of the varying processing needs of such applications, the platform should be able to automatically grow and shrink as needed. Unfortunately, the current Storm platform does not provide this capability. In this paper we describe the design and implementation of a tool that monitors several aspects of the Storm platform, the applications running on top of it, and external systems such as queues and databases. Based on this information, the tool decides whether extra servers are needed or machines may be decommissioned from the cluster, and then acts upon it by actually creating new virtual machines or shutting them down.

Runtime Adaptation of Data Stream Processing Systems: The State of the Art

“…Several solutions, especially those acting on data streams (e.g., load distribution, shedding), perform adaptation with iner granularity, at level of single tuples (e.g., [3,23,50,95,168,185]) or batches of tuples (e.g., [38,177]). Solutions acting at the infrastructure level usually work with the granularity of the computing node (e.g., [47,82,176]) or the network link [5].…”

“…Another example is given by Kumbhare et al [100], who use a utility function to combines resource cost and a so-called łapplication valuež that depends on current processing accuracy. Among system-oriented metrics, the most used one is resource utilization (e.g., [20,27,57,60,68,72,91,120,134,152,165,176]), which captures the utilization level of a computing resource, usually CPU. As in diferent application domains, utilization is often used in conjunction with threshold-based adaptation policies, where actions are triggered whenever the utilization level violates a pre-deined threshold value (e.g., [27,57,74]).…”

A Comprehensive Survey on Parallelization and Elasticity in Stream Processing

“…To this end, they leverage a resource estimator that predicts the resource consumption based on the current resource utilization. Similarly, Van der Veen et al [138] propose a controller to automatically adjust the number of virtual machines assigned to a deployment of the Storm ESP system.…”

Towards machine learning-based self-tuning of Hadoop-Spark system

“…The traditional computing system cannot offer the necessary efficiency and performance. Therefore, the big data industries have seen various platforms such ad Spark [4], Haddoo [5,6] and Strom [7] to entertain the demands of a large amount of big data processing. Apache spark is one of the most widespread frameworks among the prevailing distributes framework, due to its great capability to sustenance heavy applications and for complex data processing performance [2,4].…”