Parallel Continuous Outlier Mining in Streaming Data

Abstract

In this work, we focus on distance-based outliers in a metric space, where the status of an entity as to whether it is an outlier is based on the number of other entities in its neighborhood. In recent years, several solutions have tackled the problem of distance-based outliers in data streams, where outliers must be mined continuously as new elements become available. An interesting research problem is to combine the streaming environment with massively parallel systems to provide scalable streambased algorithms. However, none of the previously proposed techniques refer to a massively parallel setting. Our proposal fills this gap and investigates the challenges in transferring state-of-the-art techniques to Apache Flink, a modern platform for intensive streaming analytics. We thoroughly present the technical challenges encountered and the alternatives that may be applied. We show speed-ups of up to 117 (resp. 2076) times over a naive parallel (resp. non-parallel) solution in Flink, by using just an ordinary four-core machine and a real-world dataset. When moving to a three-machine cluster, due to less contention, we manage to achieve both better scalability in terms of the window slide size and the data dimensionality, and even higher speed-ups, e.g., by a factor of 510. Overall, our results demonstrate that oulier mining can be achieved in an efficient and scalable manner. The resulting techniques have been made publicly available as open-source software.

A Methodology for Runtime Detection and Extraction of Threat Patterns

“…As such, abnormal behavior is contextualized with respect to the majority of the other behavioral logs in the same time period. A full description of MCOD is described in [21], while in [30], there exist efficient parallel implementations to handle intensive streams. Additionally, [31] provides impartial information about MCOD efficiency and superiority over competitors.…”

“…Regarding the OD component, MCOD is publicly available by authors of this work, as explained in [21,30]. Here we provide an example execution of the OD sub-component for a given pcap file.…”

An Effective Minimal Probing Approach With Micro-Cluster for Distance-Based Outlier Detection in Data Streams

“…Several studies [9]- [15] have focused on solving various issues in data streams, such as the issue of high and unbounded data volumes [7], [16], [18], single discovery of distance-based outliers in data streams [10], [19], multiple outlier discoveries [20] and in the detection of outliers in continuous uncontrollable arrival rate of data [21], [22]. However, there is still a need for better techniques to address these issues.…”