A Flexible IoT Stream Processing Architecture Based on Microservices

Abstract

The Internet of Things (IoT) has created new and challenging opportunities for data analytics. The IoT represents an infinitive source of massive and heterogeneous data, whose real-time processing is an increasingly important issue. IoT applications usually consist of multiple technological layers connecting ‘things’ to a remote cloud core. These layers are generally grouped into two macro levels: the edge level (consisting of the devices at the boundary of the network near the devices that produce the data) and the core level (consisting of the remote cloud components of the application). The aim of this work is to propose an adaptive microservices architecture for IoT platforms which provides real-time stream processing functionalities that can seamlessly both at the edge-level and cloud-level. More in detail, we introduce the notion of μ-service, a stream processing unit that can be indifferently allocated on the edge and core level, and a Reference Architecture that provides all necessary services (namely Proxy, Adapter and Data Processing μ-services) for dealing with real-time stream processing in a very flexible way. Furthermore, in order to abstract away from the underlying stream processing engine and IoT layers (edge/cloud), we propose: (1) a service definition language consisting of a configuration language based on JSON objects (interoperability), (2) a rule-based query language with basic filter operations that can be compiled to most of the existing stream processing engines (portability), and (3) a combinator language to build pipelines of filter definitions (compositionality). Although our proposal has been designed to extend the Senseioty platform, a proprietary IoT platform developed by FlairBit, it could be adapted to every platform based on similar technologies. As a proof of concept, we provide details of a preliminary prototype based on the Java OSGi framework.

Peer Review #1 of "BMDD: a novel approach for IoT platform (broker-less and microservice architecture, decentralized identity, and dynamic transmission messages) (v0.1)"

“…many famous companies have deployed their service on Amazon EC2 15 , so we chose EC2 to deploy the cloud layer. However, in this paper, we only use Amazon EC2 for implementing the proof-ofconcept of the BMDD platform.…”

“…Indeed, in a short time, developers cannot fully understand the entire system, which leads to serious problems during construction and maintenance. Therefore, microservices architecture was introduced to fill this gap (15). This section focuses on summarizing prior work that used the microservices benefits in designing their architecture rather than detailing how it works and the direction of development.…”

BMDD: a novel approach for IoT platform (broker-less and microservice architecture, decentralized identity, and dynamic transmission messages)

“…Indeed, in a short time, developers cannot fully understand the entire system, which leads to serious problems during construction and maintenance. Therefore, microservices architecture was introduced to fill this gap ( Bixio et al, 2020 ). This section focuses on summarizing prior work that used the microservices benefits in designing their architecture rather than detailing how it works and the direction of development.…”

An Optimized Kappa Architecture for IoT Data Management in Smart Farming

“…Bixio et al presented an architecture based on proxy, adapter, and data processing microservices to manage stream data from IoT at edge and cloud level and able to manage dynamically and relocate microservices. This proposed architecture extends the IoT platform Senseioty 1 , use Java OSGi microservice framework to develop microservices, and Siddhi 2 and Apache Flink as stream processing engines [7]. Zschörnig et al suggested a personal analytics IoT platform based on a Kappa architecture where Kafka is the log data storage, Kafka stream is used for stream processing deployed as microservices 1 https://senseioty.com/ developed in Java, Druid is the database for the serving layer, API Services are written in Python, M etabase allows the visualization of data, and a data lake allows long term storage of data [8].…”