Economical and Fault-Tolerant Load Balancing in Distributed Stream Processing Systems

Abstract

SUMMARYWe present an economical and fault-tolerant load balancing strategy (EFTLBS) based on an operator replication mechanism and a load shedding method, that fully utilizes the network resources to realize continuous and highly-available data stream processing without dynamic operator migration over wide area networks. In this paper, we first design an economical operator distribution (EOD) plan based on a binpacking model under the constraints of each stream bandwidth as well as each server's CPU capacity. Next, we devise super-operator (SO) that load balances multi-degree operator replicas. Moreover, for improving the faulttolerance of the system, we color the SOs based on a coloring bin-packing (CBP) model that assigns peer operator replicas to different servers. To minimize the effects of input rate bursts upon the system, we take advantage of a load shedding method while keeping the QoS guarantees made by the system based on the SO scheme and the CBP model. Finally, we substantiate the utility of our work through experiments on ns-3. key words:distributed data stream processing systems (DDSPSs), load balancing strategy, fault-tolerance, economical operator distribution (EOD)

An Intelligent Complex Event Processing withDNumbers under Fuzzy Environment

“…The proposed system architecture of intelligent CEP is shown in Figure 2; it mainly involves two components: event collector engine and CEP engine under D number theory 6 Mathematical Problems in Engineering PRA 1 (5,7,9) (3, 5, 7) (7, 9, 10) (7, 9, 10) PRA 2 (3, 5, 7) (5, 7, 9) (3, 5, 7) (5, 7, 9) PRA 3 (7, 9, 10) (3, 5, 7) (7, 9, 10) (5,7,9) (b) Alternative Expert 2 1 2 3 4 PRA 1 (9, 10, 10) (3, 5, 7) (7, 9, 10) (7, 9, 10) PRA 2 (9, 10, 10) (5, 7, 9) (9, 10, 10) (3, 5, 7) PRA 3 (9, 10, 10) (5, 7, 9) (9, 10, 10) (7, 9, 10) PRA 1 (3, 5, 7) (7, 9, 10) (3, 5, 7) (7, 9, 10) PRA 2 (5,7,9) (5, 7, 9) (7, 9, 10) (3, 5, 7) PRA 3 (5, 7, 9) (7, 9, 10) (3, 5, 7) (9, 10, 10)…”

“…Nowadays, there has been increasing interest in distributed applications which require processing continuously flowing data from geographically distributed sources to obtain timely responses to complex queries, such as data stream processing (DSP) systems [1][2][3][4][5][6][7][8] and complex event processing (CEP) systems [9][10][11][12][13][14][15][16]. In principle, DSP systems differ from CEP systems as the DSP systems focus on transforming the incoming flow of information, while the CEP systems focus on detecting patterns of information that represent the higher-level events [17,18].…”

An Adaptive Parallel Processing Strategy for Complex Event Processing Systems over Data Streams in Wireless Sensor Networks

“…However, due to the differences between the DSP and CEP systems, most of the parallel methods that exclusively focus on aggregate queries or binary equi-joins in the DSP systems cannot be simply and directly used in the CEP systems that focus on multi-relational non-equi-joins on the time dimension, possibly with temporal ordering constraints, such as sequences (SEQ) operator and conjunctions (AND) operator [14,17,19,53]. Furthermore, the large volume and input rates of data streams are very common in the big data applications [54,55]. The increased time window sizes of operators and input rates of streams may cause bottlenecks of the CEP system.…”

New parallel processing strategies in complex event processing systems with data streams

“…11,13,15,39 The volume and input rates of the data, however, would become large, similar to event stream processing, especially for big data applications. 40,41 Increasing the size of the time window of an operator or the input rate of a stream may cause bottlenecks, which gives rise to query results of poor quality, losing the quality-of-service (QoS) guarantees of the system.…”