Anomalies Detection Using Isolation in Concept-Drifting Data Streams

Togbe, Maurras Ulbricht; Chabchoub, Yousra; Boly, Aliou; Barry, Mariam; Chiky, Raja; Bahri, Maroua

doi:10.3390/computers10010013

Cited by 40 publications

(22 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The proposed framework is evaluated by comparing results with its standalone base learners and other some predictive machine learning techniques such as Logistic Regression (LR), SVM, GaussianNB, K-NN, Artificial Neural Network (ANN), Gradient Boosting, and Local Outlier Factor (LOF). In addition, ESOD is compared with many state-ofthe-art methods found in [14], [15], [21], [22], [26], [27], and [28]. We individually show the performance of the comparisons on the aforementioned 11 datasets using the accuracy, precision, recall, and F1-score metrics.…”

Section: E Results and Discussionmentioning

confidence: 99%

“…The next set of experiments is performed to compare the proposed framework with some state-of-the-art methods for streaming outlier detection presented in [14], [15], [21], [22], [26], [27] and [28] and the results are shown in Fig 6 . In more detail, from Fig. 6(a)-(k), the performance of the proposed model (ESOD) is better than its competitive methods on all benchmark datasets, where it achieves higher rates than others.…”

Section: E Results and Discussionmentioning

confidence: 99%

“…They built the iForestASD on the scikit-multiflow framework; it is a machine learning framework for streaming data that is free and open source. In 2021 [27], they modified their implementation of iForestASD to address the idea of data stream drift, proposing three algorithms for drift detection: ADWIN, KSWIN, and finally extending KSWIN to deal with n-dimensional data streams. Another recent study was presented in [28], where the authors proposed an ensemble-based outlier detection framework for high-dimensional data named Average Selection and Ensemble of Candidates for Outlier Detection (ASEC-OD).…”

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

An Effective Ensemble-based Framework for Outlier Detection in Evolving Data Streams

Hassan¹,

Barakat²,

Rezk³

2022

IJACSA

View full text Add to dashboard Cite

In the last few years, data streams have drawn lots of researchers' attention due to their various applications, such as healthcare monitoring systems, fraud and intrusion detection, the internet of things (IoT), and financial market applications. A data stream is an unbounded sequence of data continually generated over time and is prone to evolution. Outliers in streaming data are the elements that significantly deviate from the majority of elements and then have to be detected as they may be error values or events of interest. Detection of outliers is a challenging issue in streaming data and is one of the most crucial tasks in data stream mining. Existing outlier detection methods for static data are unsuitable for use in data stream settings due to the unique characteristics of streaming data such as unpredictability, uncertainty, high-dimensionality, and changes in data distribution. Thus, in this paper, a novel ensemble learning framework called Ensemble-based Streaming Outlier Detection (ESOD) is presented to perfectly detect outliers over streaming data using a sliding window technique that is updated in response to the incoming events from the data streaming environment to overcome the concept evolution nature of streaming data. The proposed framework has three phases, namely the training phase, testing/offline phase, and outlier detection/online phase. A detection weighted vote technique is used to determine the final decisions for potential outliers. In the extensive experimental study, which was conducted on 11 realworld benchmark datasets, the proposed framework was assessed using many accuracy metrics. The experiment results showed that the proposed framework beats many other state-ofthe-art methods.

show abstract

Section: E Results and Discussionmentioning

confidence: 99%

Section: E Results and Discussionmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

An Effective Ensemble-based Framework for Outlier Detection in Evolving Data Streams

Hassan¹,

Barakat²,

Rezk³

2022

IJACSA

View full text Add to dashboard Cite

show abstract

“…Over the past couple of years, methods have been proposed that satisfy the unsupervised and online requirement, such as [17][18][19], but just a few, Isolation Forest (iForest) [20], HS-Trees [21], RS-Hash [22] and Loda [23], have been shown to outperform numerous competitors and are therefore regarded as state of the art [24,25]. Even if iForest was originally intended as an offline algorithm, a handful of variants, such as [16,[26][27][28][29], have been proposed that are adapting it or are taking advantage of its concept to operate on SD.…”

Section: Related Work 21 Aspects On Unsupervised Online Outlier Detectionmentioning

confidence: 99%

Exploiting the Outcome of Outlier Detection for Novel Attack Pattern Recognition on Streaming Data

et al. 2021

View full text Add to dashboard Cite

Future-oriented networking infrastructures are characterized by highly dynamic Streaming Data (SD) whose volume, speed and number of dimensions increased significantly over the past couple of years, energized by trends such as Software-Defined Networking or Artificial Intelligence. As an essential core component of network security, Intrusion Detection Systems (IDS) help to uncover malicious activity. In particular, consecutively applied alert correlation methods can aid in mining attack patterns based on the alerts generated by IDS. However, most of the existing methods lack the functionality to deal with SD data affected by the phenomenon called concept drift and are mainly designed to operate on the output from signature-based IDS. Although unsupervised Outlier Detection (OD) methods have the ability to detect yet unknown attacks, most of the alert correlation methods cannot handle the outcome of such anomaly-based IDS. In this paper, we introduce a novel framework called Streaming Outlier Analysis and Attack Pattern Recognition, denoted as SOAAPR, which is able to process the output of various online unsupervised OD methods in a streaming fashion to extract information about novel attack patterns. Three different privacy-preserving, fingerprint-like signatures are computed from the clustered set of correlated alerts by SOAAPR, which characterizes and represents the potential attack scenarios with respect to their communication relations, their manifestation in the data's features and their temporal behavior. Beyond the recognition of known attacks, comparing derived signatures, they can be leveraged to find similarities between yet unknown and novel attack patterns. The evaluation, which is split into two parts, takes advantage of attack scenarios from the widely-used and popular CICIDS2017 and CSE‐CIC‐IDS2018 datasets. Firstly, the streaming alert correlation capability is evaluated on CICIDS2017 and compared to a state-of-the-art offline algorithm, called Graph-based Alert Correlation (GAC), which has the potential to deal with the outcome of anomaly-based IDS. Secondly, the three types of signatures are computed from attack scenarios in the datasets and compared to each other. The discussion of results, on the one hand, shows that SOAAPR can compete with GAC in terms of alert correlation capability leveraging four different metrics and outperforms it significantly in terms of processing time by an average factor of 70 in 11 attack scenarios. On the other hand, in most cases, all three types of signatures seem to reliably characterize attack scenarios such that similar ones are grouped together, with up to 99.05\% similarity between the FTP and SSH Patator attack.intrusion detection; alert analysis; alert correlation; outlier detection; attack scenario; streaming data; network security

show abstract

“…IBFS is the only approach that might be able to satisfy those requirements since it works unsupervised and is tailored for the purpose of OD by exploiting the nature of iForest. Numerous recent advancements of iForest in the streaming setting [48][49][50], in particular, let IBFS constitute a promising candidate for FS on SD.…”

Section: Feature Selection For Outlier Detectionmentioning

confidence: 99%

Unsupervised Feature Selection for Outlier Detection on Streaming Data to Enhance Network Security

et al. 2021

View full text Add to dashboard Cite

Over the past couple of years, machine learning methods—especially the outlier detection ones—have anchored in the cybersecurity field to detect network-based anomalies rooted in novel attack patterns. However, the ubiquity of massive continuously generated data streams poses an enormous challenge to efficient detection schemes and demands fast, memory-constrained online algorithms that are capable to deal with concept drifts. Feature selection plays an important role when it comes to improve outlier detection in terms of identifying noisy data that contain irrelevant or redundant features. State-of-the-art work either focuses on unsupervised feature selection for data streams or (offline) outlier detection. Substantial requirements to combine both fields are derived and compared with existing approaches. The comprehensive review reveals a research gap in unsupervised feature selection for the improvement of outlier detection methods in data streams. Thus, a novel algorithm for Unsupervised Feature Selection for Streaming Outlier Detection, denoted as UFSSOD, will be proposed, which is able to perform unsupervised feature selection for the purpose of outlier detection on streaming data. Furthermore, it is able to determine the amount of top-performing features by clustering their score values. A generic concept that shows two application scenarios of UFSSOD in conjunction with off-the-shell online outlier detection algorithms has been derived. Extensive experiments have shown that a promising feature selection mechanism for streaming data is not applicable in the field of outlier detection. Moreover, UFSSOD, as an online capable algorithm, yields comparable results to a state-of-the-art offline method trimmed for outlier detection.

show abstract

Anomalies Detection Using Isolation in Concept-Drifting Data Streams

Cited by 40 publications

References 48 publications

An Effective Ensemble-based Framework for Outlier Detection in Evolving Data Streams

An Effective Ensemble-based Framework for Outlier Detection in Evolving Data Streams

Exploiting the Outcome of Outlier Detection for Novel Attack Pattern Recognition on Streaming Data

Unsupervised Feature Selection for Outlier Detection on Streaming Data to Enhance Network Security

Contact Info

Product

Resources

About