Deep Anomaly Detection with Deviation Networks

Abstract

Although deep learning has been applied to successfully address many data mining problems, relatively limited work has been done on deep learning for anomaly detection. Existing deep anomaly detection methods, which focus on learning new feature representations to enable downstream anomaly detection methods, perform indirect optimization of anomaly scores, leading to data-inefficient learning and suboptimal anomaly scoring. Also, they are typically designed as unsupervised learning due to the lack of large-scale labeled anomaly data. As a result, they are difficult to leverage prior knowledge (e.g., a few labeled anomalies) when such information is available as in many real-world anomaly detection applications.This paper introduces a novel anomaly detection framework and its instantiation to address these problems. Instead of representation learning, our method fulfills an end-to-end learning of anomaly scores by a neural deviation learning, in which we leverage a few (e.g., multiple to dozens) labeled anomalies and a prior probability to enforce statistically significant deviations of the anomaly scores of anomalies from that of normal data objects in the upper tail. Extensive results show that our method can be trained substantially more data-efficiently and achieves significantly better anomaly scoring than state-of-the-art competing methods.

Mobility Irregularity Detection with Smart Transit Card Data

“…Each passenger data (x N +1 , X) is fit with an iForest model and a score of x N +1 is given. -DevNet [10]. A new deviation loss based on z-score is proposed for anomaly detection.…”

“…Conventional methods include isolation Forest [9], and One-Class Support Vector Machine. Present deep learning-based models try to learn a distinguished density space between the normal and fraud data [10,11,15]. DevNet [10] defines a deviation loss based on z-score of a prior Gaussian distribution and squeezes the outliers to the tail of the distribution.…”

“…Present deep learning-based models try to learn a distinguished density space between the normal and fraud data [10,11,15]. DevNet [10] defines a deviation loss based on z-score of a prior Gaussian distribution and squeezes the outliers to the tail of the distribution. Perera et al [11] present a compactness and a descriptiveness loss for one-class learning.…”

“…(5) u N +1 ∼ u N refers to normal data and u N +1 ∼ u N refers to fraud data. The reason why the problem is designed in a supervised manner is that a limited number of labeled anomalies can always provide critical prior knowledge for an unsupervised model [10]. However, in many real-world applications it is unable to collect massive labels.…”

Deep Learning for Anomaly Detection

“…(39) presents a simplified τ for brevity. The idea of enforcing a prior on the anomaly scores is explored in [116]. Motivated by the extensive empirical results in [75] that show the anomaly scores in a variety of real-world data sets fits Gaussian distribution very well, the work uses a Gaussian prior to encode the anomaly scores and enable the direct optimization of the scores.…”