Drift detection using uncertainty distribution divergence

Abstract

Data generated from naturally occurring processes tends to be nonstationary. For example, seasonal and gradual changes in climate data and sudden changes in financial data. In machine learning the degradation in classifier performance due to such changes in the data is known as concept drift and there are many approaches to detecting and handling it. Most approaches to detecting concept drift, however, make the assumption that true classes for test examples will be available at no cost shortly after classification and base the detection of concept drift on measures relying on these labels. The high labelling cost in many domains provides a strong motivation to reduce the number of labelled instances required to detect and handle concept drift. Triggered detection approaches that do not require labelled instances to detect concept drift show great promise for achieving this. In this paper we present Confidence Distribution Batch Detection (CDBD), an approach that provides a signal correlated to changes in concept without using labelled data. This signal combined with a trigger and a rebuild policy can maintain classifier accuracy which, in most cases, matches the accuracy achieved using classification error based detection techniques but using only a limited amount of labelled data.

On the reliable detection of concept drift from streaming unlabeled data

“…This is because, unlike the feature based change detectors like HDDDM, the margin density (MD) approach implicitly includes the model in the drift detection process. Other unlabeled drift detection techniques developed in literature (Dries and Rückert, 2009;Lindstrom et al, 2013;Dredze et al, 2010;Zliobaite, 2010) and described in Section 2.2.3, also incorporate the notion of a margin. However, these techniques differ from the MD3 approach in the signal being tracked.…”

“…False alarms lead to wasted human intervention effort and as such are undesirable. The model dependent approaches of (Dries and Rückert, 2009;Lindstrom et al, 2013;Dredze et al, 2010;Zliobaite, 2010), directly consider the classifica-tion process by tracking the posterior probability estimates of classifiers, to detect drift. They can be used with probabilistic classifiers, which output the class probabilities P(Y|X) before thresholding them to generate the final class label.…”

“…Novelty detection/ clustering methods OLINDDA (Spinosa et al, 2007), MINAS (Faria et al, 2013), Woo (Ryu et al, 2012), DETECTNOD (Hayat and Hashemi, 2010), ECSMiner (Masud et al, 2011), GC3 (Sethi et al, 2016b) Multivariate distribution monitoring CoC (Lee and Magoules, 2012), HDDDM (Ditzler and Polikar, 2011), PCA-detect (Kuncheva and Faithfull, 2014;Qahtan et al, 2015) Model dependent monitoring A-distance (Dredze et al, 2010), CDBD (Lindstrom et al, 2013), Margin (Dries and Rückert, 2009) when it matters. Implicit drift detectors rely on properties of the unlabeled data's feature values, to signal deviations.…”

An Efficient Concept Drift Detection Method for Streaming Data under Limited Labeling

“…(2) uMD (using Margin Density) [7], CDBD (Confidence Distribution Batch Detection) [6]: Using SVM as a classifier, it performs drift detection on unlabeled data streams. When drift is detected, a new classifier is built with labeled data within a drift warning interval.…”

“…The method in [5] analyzes a sequence of the posterior estimates derived from the classifier by using the univariate statistical tests such as Two sample t-test and Wilcoxon Rank sum test. CDBD (Confidence Distribution Batch Detection) [6] uses the confidence estimated by the classifier. It uses KullbackLeibler Divergence to compare the distribution of the confidence values.…”