Processing flows of information

Abstract

A large number of distributed applications requires continuous and timely processing of information as it flows from the periphery to the center of the system. Examples include intrusion detection systems which analyze network traffic in real-time to identify possible attacks; environmental monitoring applications which process raw data coming from sensor networks to identify critical situations; or applications performing online analysis of stock prices to identify trends and forecast future values.Traditional DBMSs, which need to store and index data before processing it, can hardly fulfill the requirements of timeliness coming from such domains. Accordingly, during the last decade, different research communities developed a number of tools, which we collectively call Information flow processing (IFP) systems, to support these scenarios. They differ in their system architecture, data model, rule model, and rule language. In this article, we survey these systems to help researchers, who often come from different backgrounds, in understanding how the various approaches they adopt may complement each other.In particular, we propose a general, unifying model to capture the different aspects of an IFP system and use it to provide a complete and precise classification of the systems and mechanisms proposed so far.

Predictive Analytics for Complex IoT Data Streams

“…CEP has been designed for processing and correlating high speed data on the fly without storing it [5]. Whereas ML methods are targeted for applications which are based on the historical data for extraction of knowledge [10].…”

“…It includes processing, analyzing, and correlating event streams from different data sources to infer more complex events in near real-time. The inherent distributed nature of CEP [5] makes it ideal candidate for many IoT applications as evident by examples found in literature A. Akbar, F. Carrez and K. Moessner are with the Institute for Communication Systems, University of Surrey, UK (email: adnan.akbar@surrey.ac.uk; f.carrez@surrey.ac.uk; k.moessner@surrey.ac.uk)…”

Towards Semantically Enabled Complex Event Processing

“…The last decade has seen an increased need for models that enable processing of streaming data [18]. The existing stream processing systems can be broadly divided into two classes: data stream management systems (DSMSs) and complex event processing (CEP) systems.…”

“…The existing stream processing systems can be broadly divided into two classes: data stream management systems (DSMSs) and complex event processing (CEP) systems. Both provide ways of dealing with high-velocity input data, while maintaining low processing latency; however, they differ with respect their approaches and have largely different origins [18]. DSMSs have been developed from traditional relational database management systems (DBMSs).…”

Big Data Acquisition

“…The bulk of big data acquisition is carried out within the message queuing paradigm, sometimes also called the streaming paradigm, publish/subscribe paradigm (Carzaniga et al 2000), or event processing paradigm (Cugola and Margara 2012;Luckham 2002). Here, the basic assumption is that manifold volatile data sources generate information that needs to be captured, stored, and analysed by a big data processing platform.…”