Operator fission for load balancing in distributed heterogeneous data stream processing systems

Abstract

Distributed data stream processing allows to optimize resource consumption. A query's operators can be executed by several systems. The placement of filter or aggregate operators near the data source omits unnecessary data transfer. The operator placement decision is a complex problem. In certain scenarios the goal is not only overall minimization of e.g. resource consumption but an evenly distributed load.We propose an operator fission algorithm, that works on the basis of an initial operator placement. The algorithm selects certain operators from the set of operators that allow fission for parallel execution by multiple systems. Load is thus divided between processors in a more fine-grained way, resulting in lower maximum load and lower load variance. We present and evaluate three different variants of the algorithm to allow tuning the trade-off between optimization time and result quality.

Self-adaptive processing graph with operator fission for elastic stream processing

“…The merging step is only performed if the operator is stateful, meaning that it stores a state built with past processing steps. The conditions that must be met to guarantee the same results are described in Pollner et al (2015) . A similar model of fission operators is presented in Zeitler and Risch (2010) ; Wu and Liu (2014) , and Schneider et al (2012) .…”