A survey on data stream, big data and real-time

Gomes, Eliza; Plentz, Patricia Della Mea; Rolt, Carlos Roberto De; Dantas, Mário A. R.

doi:10.1504/ijnvo.2019.097631

Cited by 7 publications

(4 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, there needs to be a consensual definition of what real-time means. Gomes et al [ 3 ] focused their study on this concept in the context of data streams and big data. According to the authors, there are different intents when discussing real-time.…”

Section: Big Data Stream Processing Frameworkmentioning

confidence: 99%

“…Two years later, Apache Flink 3 and Apache Storm 4 were created. While Spark uses micro-batches for stream processing, Flink and Storm can perform stream processing natively.…”

Section: Data Processing Frameworkmentioning

confidence: 99%

“…However, we are more focused on data stream processing frameworks. In addition, the authors of [ 3 ] also discussed the concept of real-time associated with the processing of data streams, while the authors of [ 10 ] only perform a brief comparison of streaming processing frameworks. The authors of [ 10 ] conducted some practical evaluations of the streaming processing frameworks.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Time series big data: a survey on data stream frameworks, analysis and algorithms

et al. 2023

View full text Add to dashboard Cite

Big data has a substantial role nowadays, and its importance has significantly increased over the last decade. Big data’s biggest advantages are providing knowledge, supporting the decision-making process, and improving the use of resources, services, and infrastructures. The potential of big data increases when we apply it in real-time by providing real-time analysis, predictions, and forecasts, among many other applications. Our goal with this article is to provide a viewpoint on how to build a system capable of processing big data in real-time, performing analysis, and applying algorithms. A system should be designed to handle vast amounts of data and provide valuable knowledge through analysis and algorithms. This article explores the current approaches and how they can be used for the real-time operations and predictions.

show abstract

Section: Big Data Stream Processing Frameworkmentioning

confidence: 99%

“…Two years later, Apache Flink 3 and Apache Storm 4 were created. While Spark uses micro-batches for stream processing, Flink and Storm can perform stream processing natively.…”

Section: Data Processing Frameworkmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Time series big data: a survey on data stream frameworks, analysis and algorithms

et al. 2023

View full text Add to dashboard Cite

show abstract

“…The recent years have seen an enormous increase in the need for high-dimensional time-series data analysis as well as methods to serve those needs, from computing-cluster-based processing [21,7] on one side of the spectrum to edge computing [39,40] applications on the other. Traditional time-series analysis methods often times perform poorly on the high-dimensional data found in such applications [38], and thus the field of time-series data analysis has seen an explosion of deep-learning-based methods that intend to overcome these issues [62].…”

Section: Introductionmentioning

confidence: 99%

DeepHYDRA: A Hybrid Deep Learning and DBSCAN-Based Approach to Time-Series Anomaly Detection in Dynamically-Configured Systems

Stehle,

Vandelli,

Zahn

et al. 2024

Proceedings of the 38th ACM International Conference on Supercomputing

View full text Add to dashboard Cite

Anomaly detection in distributed systems such as High-Performance Computing (HPC) clusters is vital for early fault detection, performance optimisation, security monitoring, reliability in general but also operational insights. It enables proactive measures to address issues, ensuring system reliability, resource efficiency, and protection against potential threats. Deep Neural Networks have seen successful use in detecting long-term anomalies in multidimensional data, originating for instance from industrial or medical systems, or weather prediction. A downside of such methods is that they require a static input size, or lose data through cropping, sampling, or other dimensionality reduction methods, making deployment on systems with variability on monitored data channels, such as computing clusters difficult. To address these problems, we present DeepHYDRA (Deep Hybrid DBSCAN/Reduction-Based Anomaly Detection) which combines DBSCAN and learning-based anomaly detection. DBSCAN clustering is used to find point anomalies in time-series data, mitigating the risk of missing outliers through loss of information when reducing input data to a fixed number of channels. A deep learning-based time-series anomaly detection method is then applied to the reduced data in order to identify long-term outliers. This hybrid approach reduces the chances of missing anomalies that might be made indistinguishable from normal data by the reduction process, and likewise enables the algorithm to be scalable and tolerate partial system failures while retaining its detection capabilities. Using a subset of the well-known SMD dataset family, a modified variant of the Eclipse dataset, as well as an in-house dataset with a large variability in active data channels, made publicly available with this work, we furthermore analyse computational intensity, memory footprint, and activation counts. DeepHYDRA is shown to reliably detect different types of anomalies in both large and complex datasets. At the same time, the applied reduction approach is shown to enable real-time anomaly detection of a whole computing cluster while occupying proportionally minuscule compute resources, enabling its usage on existing systems without the need for hardware changes.

show abstract

A Fog-Cloud Approach to Enhance Communication in a Distance Learning Cloud Based System

Larcher

Ströele

Dantas

2019

Advances on P2P, Parallel, Grid, Cloud and Internet Computing

View full text Add to dashboard Cite

A survey on data stream, big data and real-time

Cited by 7 publications

References 0 publications

Time series big data: a survey on data stream frameworks, analysis and algorithms

Time series big data: a survey on data stream frameworks, analysis and algorithms

DeepHYDRA: A Hybrid Deep Learning and DBSCAN-Based Approach to Time-Series Anomaly Detection in Dynamically-Configured Systems

A Fog-Cloud Approach to Enhance Communication in a Distance Learning Cloud Based System

Contact Info

Product

Resources

About