Online and Offline Analysis of Streaming Data

Abstract

Online and offline analytics have been traditionally treated separately in software architecture design, and there is no existing general architecture that can support both. Our objective is to go beyond and introduce a scalable and maintainable architecture for performing online as well as offline analysis of streaming data.In this paper, we propose a 7-layered architecture utilising microservices, publish-subscribe pattern, and persistent storage. The architecture ensures high cohesion, low coupling, and asynchronous communication between the layers, thus yielding a scalable and maintainable solution.This design can help practitioners to engage their online and offline use cases in one single architecture, and also is of interest to academics, as it is a building block for a general architecture supporting data analysis.

Anomaly Detection in a Large-Scale Cloud Platform

“…There are mainly two design options for building a learning model (based on the data usages in the modelling pipeline [11], [12]). The first one is batch learning, also known as offline learning, while the second one is online learning.…”

“…In online learning, data flows in streams into the learning algorithm and updates the model. This data flow can be seen as individual sample points in the dataset or mini-batches [11], [12]. This model is ideal in situations where we need to use real-time data samples to build a prediction model.…”

“…In the 7-layered architecture, the pub-sub (implemented in Layers 2, 4, and 6) is used to establish communication between the odd layers. Once our codebase stabilized and we were confident in the high availability of the underlying services, we made a tradeoff decision to enable direct communication 'A detailed description of the 7-layered architecture is given in [11], [20].…”

Anomaly Detection in a Large-scale Cloud Platform

“…There are mainly two design options for building a learning model (based on the data usages in the modelling pipeline [7,8]). The first one is batch learning, also known as offline learning, while the second one is online learning.…”

“…In online learning, data flows in streams into the learning algorithm and updates the model. This data flow can be as individual sample points in the dataset or mini-batches [7,8]. This model is ideal in situations where we need to use real-time data samples to build a prediction model.…”