Large scale anomaly detection in mixed numerical and categorical input spaces

Abstract

This work presents the ADMNC method, designed to tackle anomaly detection for large-scale problems with a mixture of categorical and numerical input variables. A flexible parametric probability measure is adjusted to input data, allowing to track low likelihood values as anomalies. The main contribution of this method is that, to cope with the different nature of variables, we factorize the joint probability measure into two parts: the marginal density of the continuous variables and the conditional probability of the categorical variables given the continuous part of the feature vector, resulting in a model trained through a maximum likelihood objective function optimized with Stochastic Gradient Descent. This results in an effective and scalable algorithm. The comparison with other wellknown anomaly detection algorithms over several datasets shows that ADMNC offers top level accuracy even in datasets that are out of reach for the most effective existing methods, and scales well to process very large datasets, which transforms it in a powerful tool for a problem growing in popularity that currently lacks suitable algorithms.

Regression Tree Based Explanation for Anomaly Detection Algorithm

“…To assess the validity of our approach, we considered two large datasets focusing on the network intrusion detection domain, KDDCup99 [5] and ISCXIDS 2012. For each resulting clustering, we measured its quality Q and weighted variance.…”

“…The original ADMNC algorithm [5] is a method for large-scale offline learning to obtain a model of normal data that is then used to detect anomalies. The model used to obtain the pre-hoc explanation will consist of a grouping of the input patterns attending to their numerical variables.…”

“…Clusters will be defined as the leaf nodes of a shallow decision tree [6]. Each pattern will be assigned its ADMNC estimator [5]. This estimator will then be approximated with a simple regression model, learned using the Apache Spark MLLib implementation of CART.…”

“…This work extends the ADMNC algorithm [5], an anomaly detection algorithm developed by our research group, with a new layer that opens the ADMNC black box by offering pre-hoc explainability. Regression decision trees are used to segment input data into homogeneous groups that can be described with a few variables.…”

DSVD‐autoencoder: A scalable distributed privacy‐preserving method for one‐class classification

“…It has also been possible thanks to the support received by the National Plan for Scientific and Technical Research and Innovation of the Spanish Government (Grant no. TIN2015-65069-C2-1-R and PID2019-109238GB-C2), and by *The values were obtained from Reference [71].…”