LogUAD: Log Unsupervised Anomaly Detection Based on Word2Vec

Wang, Jin; Zhao, Changqing; He, Shiming; Gu, Yu; Alfarraj, Osama; Abugabah, Ahed

doi:10.32604/csse.2022.022365

Cited by 33 publications

(18 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We consider this aspect an intrinsic limitation of the model rather than a weakness of the study. Other promising scenarios are worthy of mention since they may prove more accurate in the near future, as suggested by some groups [ 37 , 38 , 39 , 40 ].…”

Section: Discussionmentioning

confidence: 99%

Word2vec Word Embedding-Based Artificial Intelligence Model in the Triage of Patients with Suspected Diagnosis of Major Ischemic Stroke: A Feasibility Study

Desai

Zumbo

Giordano

et al. 2022

IJERPH

View full text Add to dashboard Cite

Background: The possible benefits of using semantic language models in the early diagnosis of major ischemic stroke (MIS) based on artificial intelligence (AI) are still underestimated. The present study strives to assay the feasibility of the word2vec word embedding-based model in decreasing the risk of false negatives during the triage of patients with suspected MIS in the emergency department (ED). Methods: The main ICD-9 codes related to MIS were used for the 7-year retrospective data collection of patients managed at the ED with a suspected diagnosis of stroke. The data underwent “tokenization” and “lemmatization”. The word2vec word-embedding algorithm was used for text data vectorization. Results: Out of 648 MIS, the word2vec algorithm successfully identified 83.9% of them, with an area under the curve of 93.1%. Conclusions: Natural language processing (NLP)-based models in triage have the potential to improve the early detection of MIS and to actively support the clinical staff.

show abstract

Section: Discussionmentioning

confidence: 99%

Word2vec Word Embedding-Based Artificial Intelligence Model in the Triage of Patients with Suspected Diagnosis of Major Ischemic Stroke: A Feasibility Study

Desai

Zumbo

Giordano

et al. 2022

IJERPH

View full text Add to dashboard Cite

show abstract

“…The most widely used for word embedding models are Word2Vec (Mikolov et al, 2013b,a) and GloVe (Pennington et al, 2014) both of which are based on unsupervised learning. LogEvent2vec (Wang et al, 2020) and LogUAD (Wang et al, 2022) use Word2Vec to generate word vectors and generates weighted log sequence feature vectors. Doc2Vec (Le and Mikolov, 2014) is similar to the Word2vec algorithm, but instead of vectorizing words, it creates a vector embedding of text snippets.…”

Section: Semantics Modelsmentioning

confidence: 99%

“…• Clustering of similar events to reduce the volume of processed information and classify security events to event detection (ED) (Liu et al, 2019b;Deng and Hooi, 2021), event grouping (EG) (Hostiadi et al, 2019;Sun et al, 2020), and event pattern extraction (EPE) (Dhaou et al, 2021;Zeng et al, 2021). • Intrusion detection (ID), which deals with multi-stage and targeted attacks (Joloudari et al, 2020;Sen et al, 2022), or anomaly detection (AD) (Han et al, 2020;Wang et al, 2022) to notify the security administrator about misuses and deviations from normal behavior, respectively. • Intrusion prediction (IP) (Holgado et al, 2017;Oki et al, 2018) based on incoming events, which allows early detection of intruder targets.…”

Section: Summary Of Ai-based Security Event Correlation Modelsmentioning

confidence: 99%

A survey on artificial intelligence techniques for security event correlation: models, challenges, and opportunities

Gaifulina

Kotenko

2023

Artif Intell Rev

View full text Add to dashboard Cite

Information systems need to process a large amount of event monitoring data. The process of finding the relationships between events is called correlation, which creates a context between independent events and previously collected information in real time and normalizes it for subsequent processing. In cybersecurity, events can determine the steps of attackers and can be analyzed as part of a specific attack strategy. In this survey, we present the systematization of security event correlation models in terms of their representation in AI-based monitoring systems as: rule-based, semantic, graphical and machine learning based-models. We define the main directions of current research in the field of AI-based security event correlation and the methods used for the correlation of both single events and their sequences in attack scenarios. We also describe the prospects for the development of hybrid correlation models. In conclusion, we identify the existing problems in the field and possible ways to overcome them.

show abstract

“…[7] proposes a deep neural network model utilising Long Short-Term Memory (LSTM) to learn log patterns different from the normal to allow detection of anomalies. [8] uses semantic relationships between logs to generate word vectors and weighted log sequence feature vectors with Term Frequency-Inverse Document Frequency (TF-IDF), which is then used with K-Means clustering to detect anomalous logs. [9] uses isolation forest and deep autoencoder neural networks to detect anomalous logs.…”

Section: Related Workmentioning

confidence: 99%

“…There have been several studies that focus on reducing the number of alerts and anomalies [1][2][3][4][5][6][7][8][9][10], but an automated anomaly detection using log data is still an ongoing challenge. Anomaly detection is, in fact, a binary classification, deciding between normal and anomalous classes; however, there are a number of challenges, which can be summarised as follows:…”

Section: Introductionmentioning

confidence: 99%

Automated Log Analysis and Anomaly Detection Using Machine Learning

Shah

Pasha

Zadeh

et al. 2022

Frontiers in Artificial Intelligence and Applications

View full text Add to dashboard Cite

Reducing the number of alerts and anomalies has been the focus of several studies, but an automated anomaly detection using log files is still an ongoing challenge. One of the pertinent challenges in the detection of anomalies using log files is dealing with ‘unlabelled’ data. In the existing approaches, there is a lack of anomalous examples and that log anomalies can have many different patterns. One solution is to label the data manually, but this can be a tedious task as the data size could be very large and log files are not easily understandable. In this paper, we have presented an automated anomaly detection model that combines supervised and unsupervised machine learning with domain knowledge. Our method reduces the number of alerts by accurately predicting anomalous log events based on domain expertise, which is used to create automated rules that allow generating a labelled dataset from unlabelled log records, which are unstructured and present in many different formats. This labelled dataset is then used to train a classification model that will help predict anomalous log events. Our results show that we can accurately predict anomalous and non-anomalous events with an average accuracy of 98%. Our approach offers a practical solution for systems where logs are collected without any labelling, making it difficult to create an accurate model to identify anomalous log records. The methodology presented is very fast and efficient, which can provide real-time anomaly detection for time critical environments.

show abstract

LogUAD: Log Unsupervised Anomaly Detection Based on Word2Vec

Cited by 33 publications

References 18 publications

Word2vec Word Embedding-Based Artificial Intelligence Model in the Triage of Patients with Suspected Diagnosis of Major Ischemic Stroke: A Feasibility Study

Word2vec Word Embedding-Based Artificial Intelligence Model in the Triage of Patients with Suspected Diagnosis of Major Ischemic Stroke: A Feasibility Study

A survey on artificial intelligence techniques for security event correlation: models, challenges, and opportunities

Automated Log Analysis and Anomaly Detection Using Machine Learning

Contact Info

Product

Resources

About