ConAnomaly: Content-Based Anomaly Detection for System Logs

Abstract

Enterprise systems typically produce a large number of logs to record runtime states and important events. Log anomaly detection is efficient for business management and system maintenance. Most existing log-based anomaly detection methods use log parser to get log event indexes or event templates and then utilize machine learning methods to detect anomalies. However, these methods cannot handle unknown log types and do not take advantage of the log semantic information. In this article, we propose ConAnomaly, a log-based anomaly detection model composed of a log sequence encoder (log2vec) and multi-layer Long Short Term Memory Network (LSTM). We designed log2vec based on the Word2vec model, which first vectorized the words in the log content, then deleted the invalid words through part of speech tagging, and finally obtained the sequence vector by the weighted average method. In this way, ConAnomaly not only captures semantic information in the log but also leverages log sequential relationships. We evaluate our proposed approach on two log datasets. Our experimental results show that ConAnomaly has good stability and can deal with unseen log types to a certain extent, and it provides better performance than most log-based anomaly detection methods.

Deep Learning for Anomaly Detection in Log Data: A Survey

“…It turns out that data instability, i.e., the appearance of previously unknown events, is one of the main issues addressed by the reviewed approaches. The key idea to resolving this problem is currently to represent logs as semantic vectors so that new or changed events can still be compared to known events by measuring their similarities [17], [28], [41], [46], [48], [50], [57], [65], [69], [73]. There are many techniques for generating numeric vectors to represent log events (cf.…”

“…comp. Failures [1], [17], [20], [22], [23], [25], [27], [29]- [31], [33], [37]- [40], [42], [44], [46]- [48], [51], [52], [55], [56], [58], [59], [61], [62], [65]- [74], [76]- [78] BlueGene/L (BGL) [89] 2007 High-perf. comp.…”

“…-Failures [1], [20], [22], [24], [25], [28], [30], [33]- [37], [39]- [44], [46]- [48], [54], [55], [59], [61], [62], [64]- [66], [69], [73], [74] Thunderbird [89] 2007 High-perf. comp.…”

DRL-GAN: A Hybrid Approach for Binary and Multiclass Network Intrusion Detection

“…DL techniques commonly employed for IDSs include convolutional neural networks for spatial pattern recognition in network traffic [8], recurrent neural networks such as LSTMs for analyzing sequential data such as system logs [9], and autoencoders for anomaly detection by learning compressed representations of normal behavior [10]. While these techniques offer solutions for detecting various cyberthreats and anomalies in diverse network environments, they often require fixed training datasets and may lack the ability to adapt dynamically to new threats.…”