Configurable Distributed Data Management for the Internet of the Things

Abstract

One of the main challenges in modern Internet of Things (IoT) systems is the efficient collection, routing and management of data streams from heterogeneous sources, including sources with high ingestion rates. Despite the existence of various IoT data streaming frameworks, there is still no easy way for collecting and routing IoT streams in efficient and configurable ways that are easy to be implemented and deployed in realistic environments. In this paper, we introduce a programmable engine for Distributed Data Analytics (DDA), which eases the task of collecting IoT streams from different sources and accordingly, routing them to appropriate consumers. The engine provides also the means for preprocessing and analysis of data streams, which are two of the most important tasks in Big Data analytics applications. At the heart of the engine lies a Domain Specific Language (DSL) that enables the zero-programming definition of data routing and preprocessing tasks. This DSL is outlined in the paper, along with the middleware that supports its runtime execution. As part of the paper, we present the architecture of the engine, as well as the digital models that it uses for modelling data streams in the digital world. We also discuss the validation of the DDA in several data intensive IoT use cases in industrial environments, including use cases in pilot productions lines and in several real-life manufacturing environments. The latter manifest the configurability, programmability and flexibility of the DDA engine, as well as its ability to support practical applications.

An overview of big data analysis

“…As it is possible to perceive, the IoT opens up a world of limitless possibilities, more extensive than those offered a few years ago from the digital age. The creation of the IoT would have been much more complicated if a big data structure had not been followed since the latter allows analyzing vast amounts of data [35][36][37]. The multifaceted nature of big data is mainly due to the unstructured design of most of the data produced by modern technologies, such as web registers, radio frequency identification (RFID), sensors embedded in devices, machinery, vehicles, Internet searches, social networks, laptops, smartphones, GPS devices, and call center records [38][39][40].…”