Optimal operator placement for distributed stream processing applications

Abstract

Data Stream Processing (DSP) applications are widely used to timely extract information from distributed data sources, such as sensing devices, monitoring stations, and social networks. To successfully handle this ever increasing amount of data, recent trends investigate the possibility of exploiting decentralized computational resources (e.g., Fog computing) to define the applications placement. Several placement policies have been proposed in the literature, but they are based on different assumptions and optimization goals and,\ud as such, they are not completely comparable to each other. In this paper we study the placement problem for distributed DSP applications. Our contributions are twofold.\ud We provide a general formulation of the optimal DSP placement (for short, ODP) as an Integer Linear Programming\ud problem which takes explicitly into account the heterogeneity of computing and networking resources and which encompasses - as special cases - the different solutions proposed in the literature. We present an ODP-based scheduler for the Apache Storm DSP framework. This allows us to compare some well-known centralized and decentralized placement solutions. We also extensively analyze the ODP scalability with respect to various parameter settings

Optimized IoT service placement in the fog

“…Therefore, the application makespan duration m A k is a sum of communication link delays in each case of placement of a service on a particular virtualized resource multiplied by an according decision variable as in (7). The factors d(a i , f j ), d(a i , F), d(a i , R), and d(a i , N ) represent the makespan duration of a service a i when it is executed on the fog cell f j , the control node F, the cloud R, and the closest neighbor colony N , respectively.…”

Distributed data stream processing and edge computing: A survey on resource elasticity and future directions

“…For stream processing, Cardellini et al [119] introduce an integer programming formulation that takes into account resource heterogeneity for the Optimal Distributed Stream Processing Problem (ODP). They propose an extension to Apache Storm to incorporate an ODP-based scheduler, which estimates networks latency via a network coordination system built using the Vivaldi algorithm [99].…”

A Data Stream Processing Optimisation Framework for Edge Computing Applications

“…Due to the NP-Hard nature of this operator assignment problem [5] [6][15], a near optimal task allocation strategy would try to map tasks in the DSP application to the nodes in the host network to improve the quality of service requirements, and reduce the resource consumption. However, the optimality of the task allocation relies on the accuracy of the optimisation framework and the imposed constraints.…”

“…Their analysis focuses on the efficiency of the GA approach. Cardellini et al, [5][6] focus on modelling the operator placement problem by considering the availability of nodes, cost of operator execution on nodes, response time, and network QoS metrics. Their work does not focus on applications sharing both edge and cloud resources.…”