MStream: Fast Anomaly Detection in Multi-Aspect Streams

Abstract

Given a stream of entries in a multi-aspect data setting i.e., entries having multiple dimensions, how can we detect anomalous activities in an unsupervised manner? For example, in the intrusion detection setting, existing work seeks to detect anomalous events or edges in dynamic graph streams, but this does not allow us to take into account additional attributes of each entry. Our work aims to define a streaming multi-aspect data anomaly detection framework, termed MStream which can detect unusual group anomalies as they occur, in a dynamic manner. MStream has the following properties: (a) it detects anomalies in multi-aspect data including both categorical and numeric attributes; (b) it is online, thus processing each record in constant time and constant memory; (c) it can capture the correlation between multiple aspects of the data. MStream is evaluated over the KDDCUP99, CICIDS-DoS, UNSW-NB 15 and CICIDS-DDoS datasets, and outperforms state-of-the-art baselines.

Fast and Multi-aspect Mining of Complex Time-stamped Event Streams

“…Typical anomaly detection methods [15,25,31,61,81], such as local outlier factor (LOF) [24] and tree-based approaches [34,56], can be used in event tensors by converting multiple attributes to numerical ones. [20,45,62,79,80] use a stream of multi-aspect records as input. MemStream [21] can learn dynamically changing trends to handle time-varying data distribution known as concept drift [27,35,57].…”

“…Given a large, online stream of time-stamped events, how can we statistically summarize all the event streams and find important patterns, rules, and anomalies? Time-stamped event data are generated and collected by many real applications [10,17,29,84], including online marketing analytics [52,75], social network/location-based services [28,71], and cybersecurity systems [19,80], with increasingly larger sizes and faster rates of transactions. For example, an online shopping service could generate millions of logging entries every second, with rich information about items and users.…”

Subset Node Anomaly Tracking over Large Dynamic Graphs

“…For DARPA dataset, we track 151 nodes 7 which have anomalous edges after initial snapshot. Similarly we track 190,170 anomalous nodes over EU-CORE-S and EU-Core-L. We exclude edge-level method SedanSpot for node-level task because it can not calculate node-level anomaly score.We present the precision and running time in Table 4 and Table 5.…”

“…For example, the raw graph data used in Fig. 1 Despite the extensive literature [2,7,38] on graph anomaly detection, previous work focuses on different problem definitions of anomaly. On the other hand, works [36] on graph-level anomaly detection cannot identify individual node changes but only uncover the global changes in the overall graph structure.…”

“…We use Procrustes analysis to find optimal scale, translation and rotation to align6 In experiment, we assume that the number of anomalies is known for each node to calculate the prediction precision 7. We track totally 200 nodes, but 49 nodes have all anomalous edges before the initial snapshot, so we exclude them in precision calculation 8.…”