Exploiting the Outcome of Outlier Detection for Novel Attack Pattern Recognition on Streaming Data

Abstract

Future-oriented networking infrastructures are characterized by highly dynamic Streaming Data (SD) whose volume, speed and number of dimensions increased significantly over the past couple of years, energized by trends such as Software-Defined Networking or Artificial Intelligence. As an essential core component of network security, Intrusion Detection Systems (IDS) help to uncover malicious activity. In particular, consecutively applied alert correlation methods can aid in mining attack patterns based on the alerts generated by IDS. However, most of the existing methods lack the functionality to deal with SD data affected by the phenomenon called concept drift and are mainly designed to operate on the output from signature-based IDS. Although unsupervised Outlier Detection (OD) methods have the ability to detect yet unknown attacks, most of the alert correlation methods cannot handle the outcome of such anomaly-based IDS. In this paper, we introduce a novel framework called Streaming Outlier Analysis and Attack Pattern Recognition, denoted as SOAAPR, which is able to process the output of various online unsupervised OD methods in a streaming fashion to extract information about novel attack patterns. Three different privacy-preserving, fingerprint-like signatures are computed from the clustered set of correlated alerts by SOAAPR, which characterizes and represents the potential attack scenarios with respect to their communication relations, their manifestation in the data's features and their temporal behavior. Beyond the recognition of known attacks, comparing derived signatures, they can be leveraged to find similarities between yet unknown and novel attack patterns. The evaluation, which is split into two parts, takes advantage of attack scenarios from the widely-used and popular CICIDS2017 and CSE‐CIC‐IDS2018 datasets. Firstly, the streaming alert correlation capability is evaluated on CICIDS2017 and compared to a state-of-the-art offline algorithm, called Graph-based Alert Correlation (GAC), which has the potential to deal with the outcome of anomaly-based IDS. Secondly, the three types of signatures are computed from attack scenarios in the datasets and compared to each other. The discussion of results, on the one hand, shows that SOAAPR can compete with GAC in terms of alert correlation capability leveraging four different metrics and outperforms it significantly in terms of processing time by an average factor of 70 in 11 attack scenarios. On the other hand, in most cases, all three types of signatures seem to reliably characterize attack scenarios such that similar ones are grouped together, with up to 99.05\% similarity between the FTP and SSH Patator attack.intrusion detection; alert analysis; alert correlation; outlier detection; attack scenario; streaming data; network security

Unsupervised Feature Selection for Outlier Detection on Streaming Data to Enhance Network Security

“…Instead of using existing datasets, we will apply UFSSOD in a real world computer network (no labeled data available) in parallel to classifiers without feature selection to examine the differences in classification results. A recently proposed method [69] might be applied to exploit and evaluate the outcome of outlier detection for novel attack pattern recognition on streaming data with and without the application of UFSSOD's feature selection.…”

A survey on artificial intelligence techniques for security event correlation: models, challenges, and opportunities

“…Correlation can be based on the similarity of events by parameters (for example, source and destination IP addresses and ports), and the scenario can be represented as a graph (Haas and Fischer, 2019;Bajtoš et al, 2020). So SOAAPR (Heigl et al, 2021) (Streaming Outlier Analysis and Attack Pattern Recognition) matches and groups alerts in streaming mode, and the resulting clusters are converted into a graphical representation. The result is an attack signature that represents the attack scenario in terms of communication behavior, cause in data features, and time sequence of associated alerts.…”

“…• Similarity-based (SB) methods are based on the idea that similar events can have the same root cause or the same type, and the found links depend on the inherent similarity between attributes of each event (Kotenko et al, 2018a;Heigl et al, 2021). • Causal-based (CB) methods focus on the causal structure of a event sequence, when previous steps determine the ones that follow (Zegeye et al, 2018;Hossain and Xie, 2020).…”

“…Causal-based correlation methods make it easy to interpret the results of correlation by the operator. Therefore, this category of methods is well suited for visualizing the sequence of events (Heigl et al, 2021;Wang et al, 2021a). The simplicity of implementing such models is reduced, and also requires more computing resources to process a large amount of data.…”