Automated Log Analysis and Anomaly Detection Using Machine Learning

Abstract

Reducing the number of alerts and anomalies has been the focus of several studies, but an automated anomaly detection using log files is still an ongoing challenge. One of the pertinent challenges in the detection of anomalies using log files is dealing with ‘unlabelled’ data. In the existing approaches, there is a lack of anomalous examples and that log anomalies can have many different patterns. One solution is to label the data manually, but this can be a tedious task as the data size could be very large and log files are not easily understandable. In this paper, we have presented an automated anomaly detection model that combines supervised and unsupervised machine learning with domain knowledge. Our method reduces the number of alerts by accurately predicting anomalous log events based on domain expertise, which is used to create automated rules that allow generating a labelled dataset from unlabelled log records, which are unstructured and present in many different formats. This labelled dataset is then used to train a classification model that will help predict anomalous log events. Our results show that we can accurately predict anomalous and non-anomalous events with an average accuracy of 98%. Our approach offers a practical solution for systems where logs are collected without any labelling, making it difficult to create an accurate model to identify anomalous log records. The methodology presented is very fast and efficient, which can provide real-time anomaly detection for time critical environments.

Machine Learning-Based Anomaly Detection in NFV: A Comprehensive Survey

“…For instance, Pang, Guansong, et al [20] (2021), Nassif, Ali Bou, et al [21] (2021), Wang, Song, et al [22](2021), Gebremariam, A. A., Usman, M. and Qaraqe, M. [23] (2019), Alam, Iqbal, et al [24] (2020), Ghaffar, Zeba, et al [25] (2021), Lohrasbinasab, Iraj, et al [26] (2022), Shah, Ali Hussain, et al [27] (2022), Gallego-Madrid, Jorge, et al [28] (2022), Ahmed, Md, et al [29] (2021), Nunez-Agurto, Daniel, et al [30] (2022), and Di Mauro, Mario, et al [31] (2021) have surveyed different aspects of anomaly detection in NFV network but they lack in considering all aspects of anomaly detection in the NFV network using machine learning techniques.A brief comparison of our survey paper with all these existing survey papers is shown in Table 1.…”

“…Additionally, the paper does not provide a performance comparison between different techniques, which could be useful for readers who want to compare them. Shah, Ali Hussain, et al [27] propose an approach for automated log analysis and anomaly detection using machine learning. Their paper follows a structured approach and presents the advantages and limitations of the proposed methodology.…”

Runtime security monitoring by an interplay between rule matching and deep learning-based anomaly detection on logs

“…The related work includes the anomaly detection analysis of large-scale system log data using a BERT-based pre-training method for the self-supervised learning-based anomaly detection based on a highly efficient natural language processing performance [3] [4], and using the masked language modeling loss function per log keyword during the inference process [14]. Recent research includes also the combination of supervised and unsupervised machine learning with domain knowledge, reducing the number of alerts by predicting anomalous log events based on that domain expertise [12]. It also includes the refinement of the algorithms from the perspective of anomaly scoring and anomaly decision using self-attention neural networks and data augmentation [13], and the improvement of the model within the training data selection, data grouping, class distribution, data noise, and early detection ability [11].…”