Online Multivariate Anomaly Detection and Localization for High-dimensional Settings

Abstract

This paper considers the real-time detection of anomalies in high-dimensional systems. The goal is to detect anomalies quickly and accurately so that the appropriate countermeasures could be taken in time, before the system possibly gets harmed. We propose a sequential and multivariate anomaly detection method that scales well to high-dimensional datasets. The proposed method follows a nonparametric, i.e., data-driven, and semi-supervised approach, i.e., trains only on nominal data. Thus, it is applicable to a wide range of applications and data types. Thanks to its multivariate nature, it can quickly and accurately detect challenging anomalies, such as changes in the correlation structure and stealth low-rate cyberattacks. Its asymptotic optimality and computational complexity are comprehensively analyzed. In conjunction with the detection method, an effective technique for localizing the anomalous data dimensions is also proposed. We further extend the proposed detection and localization methods to a supervised setup where an additional anomaly dataset is available, and combine the proposed semi-supervised and supervised algorithms to obtain an online learning algorithm under the semi-supervised framework. The practical use of proposed algorithms are demonstrated in DDoS attack mitigation, and their performances are evaluated using a real IoT-botnet dataset and simulations.

Sequential Multivariate Change Detection with Calibrated and Memoryless False Detection Rates

“…Whilst some authors advocate leveraging an ensemble of univariate detectors to deal with multivariate data streams [Faithfull et al, 2019], such approaches can only detect changes to the marginal distributions and require a multiple testing correction which only allows false positive rates to be loosely bounded rather than targeted. Designing detectors that are flexible enough to detect any change in the distribution governing a multivariate data stream has been an area of recent focus [Bu et al, 2017, Li et al, 2019, Mozaffari and Yilmaz, 2019, Hinder et al, 2020, Chen et al, 2019, Kurt et al, 2020, Dasu et al, 2006.…”

Distributed Injection-Locking in Analog Ising Machines to Solve Combinatorial Optimizations

“…However, the oscillators are coupled to injectionlocking oscillator by a shared routing Iinj = N j=1 I2,j, driving a different phase dynamics, θi(t) = −κs j=1 sin(θi(t)−θj(t))−κssin(θi(t) − θinj(t)). (6) Accordingly, the phase dynamics of the oscillator becomes,…”

“…With the advent of world-wide communications and the availability of massive data, increasing application of neural networks and artificial intelligence and end of Moore's law, computing becomes a significant obstacle in recent decades [2,3,4,5]. Particularly computational power is crucial in real-time and high-dimensional machine learning settings, such as detecting anomalous processes [6,7]. The Ising model has attracted considerable prominence due to its network-based structure and the ability to map many NP-complete problems to Ising model, and various accelerators and machines have been proposed [8,9].…”

Classification of Schizophrenia from Functional MRI Using Large-scale Extended Granger Causality

“…Several methods have been proposed to capture directional relations in multivariate time-series data, e.g., transfer entropy [5] and mutual information [6]. However, as the multivariate problem's dimensions increase, the density function's computation becomes computationally expensive [7,8]. Under the Gaussian assumption, transfer entropy is equivalent to Granger causality [9].…”