Mobile Distributed Complex Event Processing—Ubi Sumus? Quo Vadimus?

Abstract

One important class of applications for the Internet of Things is related to the need to gain timely and continuous situational awareness, like smart cities, automated traffic control, or emergency and rescue operations. Events happening in the real-world need to be detected in real-time based on sensor data and other data sources. Complex Event Processing (CEP) is a technology to detect complex (or composite) events in data streams and has been successfully applied in high volume and high velocity applications like stock market analysis. However, these application domains faced only the challenge of high performance, while the Internet of Things and Mobile Big Data introduce a new set of challenges caused by mobility. This chapter aims to explain these challenges and to give an overview on how they are solved respectively how far state-of-the-art research has advanced to be useful to solve Mobile Big Data problems. At the infrastructure level the main challenge is to trade performance against resource consumption and energy efficiency and operator placement is the most dominant mechanism to address these problems. At the application and consumer level, mobile queries pose a new set of challenges for CEP related to continuously changing positions of consumers and data sources, and the need to adapt the query processing to these changes. Finally, proper methods and tools for systematical testing and reproducible performance evaluation for mobile distributed CEP are needed but not yet available.

Network-wide complex event processing over geographically distributed data sources

“…Hence, they do not confront the greater challenge of the generic problem setup addressed by our work, as described above. Geographically Distributed CEP [64]: There is a pair of works that are more closely related to ours. The first is the work of Comet [36], which proposes to combine in-network operator placement with a push-pull mechanism for communication-efficient and latency-aware CE detection over mobile networks.…”

TCEP: Transitions in operator placement to adapt to dynamic network environments

“…The programming model provides a means for developers to implement novel OP mechanisms while utilizing IoT resources. Existing work [13,28,29] focus on proposing OP mechanisms for a diversity of QoS demands. However, none of them provides a common API for the development of novel OP mechanisms 5 .…”

“…As prominently discussed in the literature [29,28,30], our programming model characterizes the existing OP mechanisms based on the placement decision into two main categories: (i) centralized and (ii) decentralized. A centralized OP mechanism assumes global knowledge on the network and the nodes (specific QoS demands) to host an operator on a physical node.…”

“…Modern CEP programming models like Apache Flink [54], Heron [55], and Beam [56] provide extensive APIs to specify complex events for both batch and stream processing. Recently proposed benchmarking frameworks such as [57] and DCEP-Sim [28] unify CEP systems [58] and simulate CEP environment and operator placement, respectively. However, none of the above programming models has enabled the specification of distinct operator placement mechanisms in a heterogeneous environment of physical machines we do in this work.…”

“…However, none of the above programming models has enabled the specification of distinct operator placement mechanisms in a heterogeneous environment of physical machines we do in this work. DCEP-Sim [28] has enabled the development of operator placement but only in a simulation environment. While we study the effect of distinct operator placement mechanisms, perform adaptations between them, and analyze the cost of adaptations in real-world network infrastructure and dynamic environment.…”