Annotated Minimum Volume Sets for Nonparametric Anomaly Discovery

Abstract

We consider an anomaly detection problem, wherein a combination of typical and anomalous data are observed and it is necessary to identify the anomalies in this particular dataset without recourse to labeled exemplars. We take as our goal to produce an annotated ranking of the observations, indicating the relative priority for each to be examined further as a possible anomaly, while making no assumptions on the distribution of typical data. We propose a framework in which each observation is linked to a corresponding minimum volume set and, implicitly adopting a hypothesis testing perspective, each set is associated with a test. An inherent ordering of these sets yields a natural ranking, while the association of each test with a false discovery rate yields an appropriate annotation. The combination of minimum volume set methods with false discovery rate principles, in the context of data contaminated by anomalies, is new and estimation of the key underlying quantities requires that a number of issues be addressed. We offer some solutions to the relevant estimation problems, and illustrate the proposed methodology on synthetic and computer network traffic data.Index Terms-minimum volume sets, false discovery rate, nonparametric outlier detection, multiple level set estimation, monotone density estimation

Semi-supervised and unsupervised novelty detection using nested support vector machines

“…As any Gaussian kernel method, the standard deviation σ has to be properly tuned. An unsupervised way of selecting it is using the Minimum Integrated Volume (MIV) criterion [7]. The optimal σ is resulting in the minimum area under the curve, represented by the volume of the different level sets as a function of the percentage of enclosed samples.…”

“…image difference). In this case, the detection of changes is a density estimation problem where the novelties are present in the tails of the distribution [7].…”

Distributed Detection in Sensor Networks With Limited Range Multimodal Sensors

“…Note that transformations that map null distributions to uniform distribution is not unique. For example, in [7], the authors propose to obtain test statistics in a dimension-by-dimension manner, and in [25] a minimum volume set approach is taken. Some of these transformations are compared in Section 7.…”

Nested Support Vector Machines

“…Scott and Kolaczyk [4] and Scott and Blanchard [7] present approaches to classifying the contaminated, unlabeled data by solving multiple level set estimation and multiple cost-sensitive classification problems, respectively.…”

“…For example, estimating density level sets at multiple levels is an important task G. Lee for many problems including clustering [1], outlier ranking [2], minimum volume set estimation [3], and anomaly detection [4]. Estimating cost-sensitive classifiers at a range of different cost asymmetries is important for ranking [5], Neyman-Pearson classification [6], transductive anomaly detection [7], and ROC studies [8].…”