Toward Supervised Anomaly Detection

Abstract

Anomaly detection is being regarded as an unsupervised learning task as anomalies stem from adversarial or unlikely events with unknown distributions. However, the predictive performance of purely unsupervised anomaly detection often fails to match the required detection rates in many tasks and there exists a need for labeled data to guide the model generation. Our first contribution shows that classical semi-supervised approaches, originating from a supervised classifier, are inappropriate and hardly detect new and unknown anomalies. We argue that semi-supervised anomaly detection needs to ground on the unsupervised learning paradigm and devise a novel algorithm that meets this requirement. Although being intrinsically non-convex, we further show that the optimization problem has a convex equivalent under relatively mild assumptions. Additionally, we propose an active learning strategy to automatically filter candidates for labeling. In an empirical study on network intrusion detection data, we observe that the proposed learning methodology requires much less labeled data than the state-of-the-art, while achieving higher detection accuracies

Learning to control a structured-prediction decoder for detection of HTTP-layer DDoS attackers

“…For Fraction of all connections of all clients that specified HTTP header field Content-Type as any text variant the detection of SQL-injection, cross-site-scripting (XSS), and PHP file-inclusion (L/RFI), traffic can be modeled based on HTTP header and query string information using HMMs (Ariu et al 2011), n-gram models (Wressnegger et al 2013), general kernels (Düssel et al 2008), or other models (Robertson and Maggi 2010). Anomaly-detection mechanisms were investigated, from centroid anomaly-detection models (Kloft and Laskov 2012) to setting hard thresholds on the likelihood of new HTTP requests given the model, to unsupervised learning of support-vector data description (SVDD) models (Düssel et al 2008, Görnitz et al 2013.…”

“…A binary SVM trained on labeled data has been observed to consistently outperform a one-class SVM using n-gram features (Wressnegger et al 2013). Similarly, augmenting SVDDs with labeled data has been observed to greatly improve detection accuracy (Görnitz et al 2013). Other work has studied SVMs (Khan et al 2007;Li et al 2012) and other classification methods (Koc et al 2012;Peddabachigari et al 2007;Gharibian and Ghorbani 2007).…”

“…A great variety of heuristic and principled approaches is used. In our study, we represent this family of approaches by SVDD which has been used successfully for several related computer-security problems (Düssel et al 2008;Görnitz et al 2013). Prior work generally uses smaller feature sets.…”

Probabilistic Anomaly Detection Method for Authorship Verification

“…We use a probabilistic anomaly detection method that can benefit from anomalous examples for the authorship verification process based on a multivariate Gaussian modelling. Given the fact that unsupervised anomaly detection approaches often fail to match the required detection rates in many tasks and there exists a need for labelled data to guide the model generation [7], our proposed methods is weakly supervised in the sense that it takes into consideration a small amount of representative anomalous data for the model generation.…”

ZERO++: Harnessing the Power of Zero Appearances to Detect Anomalies in Large-Scale Data Sets

“…Some examples of these methods are Support Vector Machines (SVMs) and Neural Networks (Erfani, Baktashmotlagh, Rajasegarar, Karunasekera, & Leckie, 2015). Semi-supervised methods require labelled instances of the normal class only, in order to train their detection models, e.g., one-class classifiers (Görnitz, Kloft, Rieck, & Brefeld, 2013). Compared to supervised methods and semi-supervised methods, unsupervised methods, which do not require labelled instances, are more widely used in industry, because obtaining accurate labelled data for anomaly detection often has a very high cost (Chandola et al, 2009).…”