Detecting Volatility Shift in Data Streams

Abstract

Current drift detection techniques detect a change in distribution within a stream. However, there are no current techniques that analyze the change in the rate of these detected changes. We coin the term stream volatility, to describe the rate of changes in a stream. A stream has a high volatility if changes are detected frequently and has a low volatility if changes are detected infrequently. We are particularly interested in a volatility shift which is a change in the rate of change (e.g. from high volatility to low volatility). We introduce and define the concept of stream volatility, and propose a novel technique to detect volatility on data streams in the presence of concept drifts. In the experiments we show our algorithm to be both fast and efficient. We also propose a new algorithm for drift detection called SEED that is faster and more memory efficient than the existing state-of-the-art drift detection approach. A faster drift detection algorithm has a flow-on benefit to the subsequent volatility detection stage because both algorithms run concurrently on the data stream.

Fast Reaction to Sudden Concept Drift in the Absence of Class Labels

“…SEED: This method, proposed in [17], within a window W, compares two sub-windows. The older part of this window is dropped when there is a distinct average exhibited by the two sub-windows.…”

Combining univariate approaches for ensemble change detection in multivariate data

“…References Category SEED [40] Monitoring Distributions ADWIN [41,8] Monitoring Distributions SEQ1 [42] Monitoring Distributions Page-Hinkley [1,8] Sequential Analysis CUSUM1 [1] Sequential Analysis CUSUM2 [8] Sequential Analysis GEOMA [43,44] Control Chart HDDM A [36] Control Chart EDDM [38,8] Control Chart DDM [37,8] Control Chart EWMA [43,8,44] Control Chart HDDM W [36] Control Chart All the univariate detectors are provided by MOA except CUSUM1, which is a CUSUM chart with upper and lower limits which was implemented in Java, and integrated into the experiment to serve as a baseline. We arrive at a final figure of 88 detectors, 3 of which are the multivariate approaches listed in Table 2, and the remaining 85 are ensembles of the univariate approaches with varying thresholds.…”

Online detection of continuous changes in stochastic processes

“…Changes that occur gradually over time are called incremental changes in the context of concept drifts [7,22], but there are no studies on online detection algorithms tailored for incremental changes to the best of our knowledge. Recently, changes in the rate of change have been studied in the scenario of volatility shift change detection [12]. This implicitly assumes that changes can be continuous.…”

“…This implicitly assumes that changes can be continuous. Our work differs in that ours deals with the rate of change with continuously changing smooth models, while [12] deals with that with a piecewise stationary model.…”