Using labeled data to evaluate change detectors in a multivariate streaming environment

Abstract

We consider the problem of detecting changes in a multivariate data stream. A change detector is defined by a detection algorithm and an alarm threshold. A detection algorithm maps the stream of input vectors into a univariate detection stream. The detector signals a change when the detection stream exceeds the chosen alarm threshold. We consider two aspects of the problem: (1) setting the alarm threshold and (2) measuring/comparing the performance of detection algorithms. We assume we are given a segment of the stream where changes of interest are marked. We present evidence that, without such marked training data, it might not be possible to accurately estimate the false alarm rate for a given alarm threshold. Commonly used approaches assume the data stream consists of independent observations, an implausible assumption given the time series nature of the data. Lack of independence can lead to estimates that are badly biased. Marked training data can also be used for realistic comparison of detection algorithms. We define a version of the receiver operating characteristic curve adapted to the change detection problem and propose a block bootstrap for comparing such curves. We illustrate the proposed methodology using multivariate data derived from an image stream.

Ordinal Patterns, Entropy, and EEG

“…A time point where some characteristic of a time series changes is called a change-point. The detection of change-points is a typical problem of time series analysis [59,[61][62][63][64]; in particular it provides a segmentation of time series into stationary segments. In this section we suggest an ordinal-patterns-based method for change-point detection.…”

“…This is a classical problem of change-point detection (see Section 1.1.2.2 in [59]); to solve it, we compare the value of CEofOP( t * ; d) with a certain threshold h: if CEofOP( t * ; d) ≥ h then one decides that there is a change-point in t * , otherwise it is concluded that no change has occurred. We compute the threshold h using block bootstrapping from the sequence of ordinal patterns (see [63,64] for a comprehensive description of the block bootstrapping; since this approach is rather often used we do not go into details). The solution of Problem 1 using the CEofOP statistic is described in Algorithm 1 (Appendix A.1).…”

Change-Point Detection using the Conditional Entropy of Ordinal Patterns

“…On the contrary, the higher h, the higher the possibility of false rejection of the H A is. As it is usually done, we consider the threshold h as a function of the desired probability α of false alarm; for computing the threshold h(α) we use block bootstrapping from the sequence π d,L of ordinal patterns (bootstrapping is often used in change-point detection for computing a threshold, see [44,45] for a theoretical discussion and [46,47] for applications of bootstrapping with detailed and clear explanations). Namely we shuffle blocks of ordinal patterns from the original sequence, in order to create a new artificial sequence.…”

Nonparametric Bayesian approach to the detection of change point in statistical process control

“…Change point analysis is also used in the detection of credit card fraud (Bolton and Hand [2]) and other anomalies. In practical applications, the applications of change point also be found in signal processing: where change point analysis can be used to detect significant changes within a stream of images (Kim et al, [9]). …”