Distributed data stream processing and edge computing: A survey on resource elasticity and future directions

Abstract

Under several emerging application scenarios, such as in smart cities, operational monitoring of large infrastructure, wearable assistance, and Internet of Things, continuous data streams must be processed under very short delays. Several solutions, including multiple software engines, have been developed for processing unbounded data streams in a scalable and efficient manner. More recently, architecture has been proposed to use edge computing for data stream processing. This paper surveys state of the art on stream processing engines and mechanisms for exploiting resource elasticity features of cloud computing in stream processing. Resource elasticity allows for an application or service to scale out/in according to fluctuating demands. Although such features have been extensively investigated for enterprise applications, stream processing poses challenges on achieving elastic systems that can make efficient resource management decisions based on current load. Elasticity becomes even more challenging in highly distributed environments comprising edge and cloud computing resources. This work examines some of these challenges and discusses solutions proposed in the literature to address them.

Cloud broker proposal based on multicriteria decision‐making and virtual infrastructure migration

“…Stream process applications hosted atop VI based on QoS schema benefit from the internal service stability to offer a continuous and reliable service, fundamental requirements to improve end-user experience. 64 The configuration for web hosting represents the weighted distribution turned in security and load balancing nodes of the hierarchical organization attending to customers that work with confidential data and need network guarantee (eg, large-scale stores, financial banks, and insurer agencies). The network-intensive schema to attend distributed applications has priority on QoS, SDN, and VI nodes, representing tenants with advanced knowledge and management skills.…”

ECSNeT++ : A simulator for distributed stream processing on edge and cloud environments

“…This generates a time series of events that can be processed to extract vital information about the environment. Generally this event stream is unbounded and hence, Distributed Stream Processing (DSP) techniques can be utilised in order to process the events in real-time [7].…”

“…As mentioned earlier, with the involvement of IoT, the sources where events are generated are located outside the cloud. Therefore, if we are to process these event streams using DSP frameworks, the current status quo is to generate data streams at the IoT devices and transmit them to the cloud for processing [7]. However, this arrangement is not ideal due to the higher latency caused by the communication overhead between these IoT devices and the cloud, which often becomes the bottleneck [11] [12].…”

A Data Stream Processing Optimisation Framework for Edge Computing Applications

“…Many Distributed Stream Processing frameworks use a dataflow approach where incoming data traverses a directed graph of operators that execute transformations over the streaming data [1]. During application deployment, such operator tasks are placed onto cluster nodes to benefit from task and data parallelism.…”