Latent Variable Based Anomaly Detection in Network System Logs

Otomo, Kazuki; Kobayashi, Satoru; Fukuda, Kensuke; Esaki, Hiroshi

doi:10.1587/transinf.2018ofp0007

Cited by 16 publications

(12 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Various classification techniques have been applied to identify the abnormalities in application behavior for fault detection in the study by researchers Jain et al (2009). In the research by Otomo et al (2018Otomo et al ( , 2019 Natural Language Processing algorithms were used for log analysis considering log data as a normal text file.…”

Section: Literature Reviewmentioning

confidence: 99%

Bibliometric survey of IT Infrastructure Management to Avoid Failure Conditions

Bhanage

Pawar

2020

IDD

View full text Add to dashboard Cite

Purpose The purpose of this paper is to present the bibliometric study of articles IT Infrastructure Management to Avoid Failure Conditions. As in today’s era of IT Industries, IT infrastructure management plays a crucial role. As a result, substantial research is going on to improve the reliability and availability of assets in IT infrastructure. Design/methodology/approach The paper analyzes and focuses the results acquired from articles accessed from Scopus for the past 15 years by examining in terms of frequently used keywords, the amount of work done in different countries and year-wise progression of the research, prolific authors, article citation frequencies, etc. Tools such as Gephi, Word Cloud, BiblioShiny, GPS visualizer, etc. are used for bibliometric analysis. Findings The study comes out with maximum publications of IT infrastructure management from conferences and journals. Anomaly detection, log analysis and learning system are the most frequently used keywords in the publications. Significant research has been done in the USA, followed by China under the area of Computer Science with an increase in publication since 2018. Originality/value This paper provides an accurate idea about the amount of work done in different countries and year-wise progression of the research. This bibliometric analysis will be useful for beginners to conduct a literature survey using appropriate literature available on the Scopus database.

show abstract

Section: Literature Reviewmentioning

confidence: 99%

Bibliometric survey of IT Infrastructure Management to Avoid Failure Conditions

Bhanage

Pawar

2020

IDD

View full text Add to dashboard Cite

show abstract

“…Farzad et al [21] proposed to use a generative adversarial network (GAN) to carry out synthetic sampling, with the goal of enhancing data modeling. Several kinds of research of this approach are [20] and [24]. However, the remaining challenge is to integrate the imbalanced datasets solution (as presented in Section II-A) and the learning model as a hybrid process.…”

Section: B Ml/dl-based Intrusion Detection Systemsmentioning

confidence: 99%

Machine Learning With Variational AutoEncoder for Imbalanced Datasets in Intrusion Detection

et al. 2022

View full text Add to dashboard Cite

As a result of the explosion of security attacks and the complexity of modern networks, machine learning (ML) has recently become the favored approach for intrusion detection systems (IDS). However, the ML approach usually faces three challenges: massive attack variants, imbalanced data issues, and appropriate data segmentation. Improper handling of the issues will significantly degrade ML performance, e.g., resulting in high false-negative and low recall rates. Despite many efforts have done in the literature, detecting security attacks in a complicated network environment with imperfect data collection is still an open issue. This work proposes a machine learning framework with a combination of variational autoencoder and multilayer perceptron to deal with imbalanced datasets and detecting the explosion of attack variants on the Internet. The detection engine also includes an efficient range-based sequential search algorithm to address the segmentation challenge in data pre-processing from multiple sources (network packets, system/statistic logs) effectively. Our work is the first attempt to demonstrate the effect of using an appropriate combination of ML models for boosting IDS detection capability in a heterogeneous environment, where data collection imperfection is common. Experimental results on a public system log dataset (e.g., HDFS) show that our method gains approximately as much as 97% on F1 score and 98% on recall rate, a promising result compared to the same measurement of other solutions. Even better, we found that the proposed treatment of imbalanced datasets can improve up to 35% on the F1 score and 27% on recall rate. The testing results also indicate that our model can detect new attack variants. Code is available at https://github.com/tuonglinhhm/Hybrid-Learning-AutoEncoder-IDS.

show abstract

“…Any input data that is fed into an already trained network and yields a high reconstruction error is then considered as anomalous. Besides the standard model for Autoencoders, there are also several related types, such as Variational Autoencoders (VAE) that operate on statistical distributions [1], [52], [58], Conditional Variational Autoencoders (CVAE) that add conditional information such as event types to the training [49], and Convolutional Autoencoder (CAE) that leverage the advantages of CNNs regarding learning of location-independent features [58].…”

Section: B Deep Learning Techniquesmentioning

confidence: 99%

Deep Learning for Anomaly Detection in Log Data: A Survey

Landauer¹,

Onder²,

Skopik³

et al. 2022

Preprint

View full text Add to dashboard Cite

Automatic log file analysis enables early detection of relevant incidents such as system failures. In particular, selflearning anomaly detection techniques capture patterns in log data and subsequently report unexpected log event occurrences to system operators without the need to provide or manually model anomalous scenarios in advance. Recently, an increasing number of approaches leveraging deep learning neural networks for this purpose have been presented. These approaches have demonstrated superior detection performance in comparison to conventional machine learning techniques and simultaneously resolve issues with unstable data formats. However, there exist many different architectures for deep learning and it is nontrivial to encode raw and unstructured log data to be analyzed by neural networks. We therefore carry out a systematic literature review that provides an overview of deployed models, data pre-processing mechanisms, anomaly detection techniques, and evaluations. The survey does not quantitatively compare existing approaches but instead aims to help readers understand relevant aspects of different model architectures and emphasizes open issues for future work.

show abstract

Latent Variable Based Anomaly Detection in Network System Logs

Cited by 16 publications

References 20 publications

Bibliometric survey of IT Infrastructure Management to Avoid Failure Conditions

Bibliometric survey of IT Infrastructure Management to Avoid Failure Conditions

Machine Learning With Variational AutoEncoder for Imbalanced Datasets in Intrusion Detection

Deep Learning for Anomaly Detection in Log Data: A Survey

Contact Info

Product

Resources

About