Real-time big data processing for anomaly detection: A Survey

Habeeb, Riyaz Ahamed Ariyaluran; Nasaruddin, Fariza Hanum; Gani, Abdullah; Hashem, Mohamed; Ahmed, Ejaz; Imran, Muhammad

doi:10.1016/j.ijinfomgt.2018.08.006

Cited by 275 publications

(52 citation statements)

References 88 publications

(129 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Authors in [21] presented the status of empirical research and application areas in big data by employing a systematic mapping method. In the same vein, authors in [22] also conducted a survey on big data technologies and machine learning algorithms with a particular focus on anomaly detection. A systematic review of literature which aims to determine the scope, application, and challenges of big data analytics in healthcare was presented by [23].…”

Section: Related Workmentioning

confidence: 99%

Big data stream analysis: a systematic literature review

2019

View full text Add to dashboard Cite

Advances in information technology have facilitated large volume, high-velocity of data, and the ability to store data continuously leading to several computational challenges. Due to the nature of big data in terms of volume, velocity, variety, variability, veracity, volatility, and value [1] that are being generated recently, big data computing is a new trend for future computing. Big data computing can be generally categorized into two types based on the processing requirements, which are big data batch computing and big data stream computing Abstract Recently, big data streams have become ubiquitous due to the fact that a number of applications generate a huge amount of data at a great velocity. This made it difficult for existing data mining tools, technologies, methods, and techniques to be applied directly on big data streams due to the inherent dynamic characteristics of big data. In this paper, a systematic review of big data streams analysis which employed a rigorous and methodical approach to look at the trends of big data stream tools and technologies as well as methods and techniques employed in analysing big data streams. It provides a global view of big data stream tools and technologies and its comparisons. Three major databases, Scopus, ScienceDirect and EBSCO, which indexes journals and conferences that are promoted by entities such as IEEE, ACM, SpringerLink, and Elsevier were explored as data sources. Out of the initial 2295 papers that resulted from the first search string, 47 papers were found to be relevant to our research questions after implementing the inclusion and exclusion criteria. The study found that scalability, privacy and load balancing issues as well as empirical analysis of big data streams and technologies are still open for further research efforts. We also found that although, significant research efforts have been directed to real-time analysis of big data stream not much attention has been given to the preprocessing stage of big data streams. Only a few big data streaming tools and technologies can do all of the batch, streaming, and iterative jobs; there seems to be no big data tool and technology that offers all the key features required for now and standard benchmark dataset for big data streaming analytics has not been widely adopted. In conclusion, it was recommended that research efforts should be geared towards developing scalable frameworks and algorithms that will accommodate data stream computing mode, effective resource allocation strategy and parallelization issues to cope with the ever-growing size and complexity of data.

show abstract

Section: Related Workmentioning

confidence: 99%

Big data stream analysis: a systematic literature review

2019

View full text Add to dashboard Cite

show abstract

“…Several applications provide insights for data security, e.g. the recent papers on botnet traffic [14], data stream of network infections [15], physical intrusions in building [16]. Safety applications of anomaly detection are not common: some researches forecast specific hazards, e.g.…”

Section: Literature Reviewmentioning

confidence: 99%

Machine learning for anomaly detection and process phase classification to improve safety and maintenance activities

Quatrini

Costantino

Gravio

et al. 2020

Journal of Manufacturing Systems

View full text Add to dashboard Cite

Anomaly detection is a crucial aspect for both safety and efficiency of modern process industries. This paper proposes a two-steps methodology for anomaly detection in industrial processes, adopting machine learning classification algorithms. Starting from a real-time collection of process data, the first step identifies the ongoing process phase, the second step classifies the input data as "Expected", "Warning", or "Critical". The proposed methodology is extremely relevant where machines carry out several operations without the evidence of production phases. In this context, the difficulty of attributing the real-time measurements to a specific production phase affects the success of the condition monitoring. The paper proposes the comparison of the anomaly detection step with and without the process phase identification step, validating its absolute necessity. The methodology applies the decision forests algorithm, as a well-known anomaly detector from industrial data, and decision jungle algorithm, never tested before in industrial applications. A real case study in the pharmaceutical industry validates the proposed anomaly detection methodology, using a 10 months database of 16 process parameters from a granulation process.

show abstract

“…Several methods for anomaly detection have been presented in literature, some of most popular anomaly detection are SVM, Replicator neural networks, correlation-based detection, density-based techniques (k-nearest neighbor, local outlier factor), deviations from association rules and ensemble techniques (using feature bagging or score normalization). Output of anomaly detection system could be score or label [93]. Score (commonly used for unsupervised or semi-supervised) is a ranked list of anomalies that are assigned to each instance depending on the degree to which instance is considered anomaly whereas labeling (used for supervised) are binary output (anomaly or not) ( Fig.…”

Section: Anomaly Detectionmentioning

confidence: 99%

Big data analytics for preventive medicine

Razzak

Imran

2019

Neural Comput & Applic

Self Cite

141

View full text Add to dashboard Cite

Medical data is one of the most rewarding and yet most complicated data to analyze. How can healthcare providers use modern data analytics tools and technologies to analyze and create value from complex data? Data analytics, with its promise to efficiently discover valuable pattern by analyzing large amount of unstructured, heterogeneous, non-standard and incomplete healthcare data. It does not only forecast but also helps in decision making and is increasingly noticed as breakthrough in ongoing advancement with the goal is to improve the quality of patient care and reduces the healthcare cost. The aim of this study is to provide a comprehensive and structured overview of extensive research on the advancement of data analytics methods for disease prevention. This review first introduces disease prevention and its challenges followed by traditional prevention methodologies. We summarize state-of-the-art data analytics algorithms used for classification of disease, clustering (unusually high incidence of a particular disease), anomalies detection (detection of disease) and association as well as their respective advantages, drawbacks and guidelines for selection of specific model followed by discussion on recent development and successful application of disease prevention methods. The article concludes with open research challenges and recommendations.

show abstract

Real-time big data processing for anomaly detection: A Survey

Cited by 275 publications

References 88 publications

Big data stream analysis: a systematic literature review

Big data stream analysis: a systematic literature review

Machine learning for anomaly detection and process phase classification to improve safety and maintenance activities

Big data analytics for preventive medicine

Contact Info

Product

Resources

About