Prequential AUC for Classifier Evaluation and Drift Detection in Evolving Data Streams

Brzeziński, Dariusz; Stefanowski, Jerzy

doi:10.1007/978-3-319-17876-9_6

Cited by 31 publications

(33 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the following section, we present a simple and efficient algorithm for calculating AUC incrementally with forgetting, which we previously introduced in [9]. Later, we investigate the properties of the resulting evaluation measure with respect to classifiers for evolving imbalanced data streams.…”

Section: Area Under the Roc Curvementioning

confidence: 99%

“…Algorithm 1 lists the pseudo-code for calculating prequential AUC. Contrary to [9], here we present an extended version of the algorithm that deals with score ties.…”

Section: Prequential Aucmentioning

confidence: 99%

“…To address these limitations, in an earlier paper [9] we introduced a new approach for calculating AUC. We presented an efficient algorithm, which incorporates a sorted tree structure with a sliding window as a forgetting mechanism, making it both computationally feasible and appropriate for concept-drifting streams.…”

Section: Introductionmentioning

confidence: 99%

“…Nonetheless, the properties of this new evaluation measure have not been thoroughly investigated in our previous work [9]. Thus, the aim of this paper is to carry out a detailed study of basic properties of prequential AUC, which in our view include the following issues.…”

Section: Introductionmentioning

confidence: 99%

“…To make the paper self-contained, in Sect. 3 we recall our algorithm [9] for computing AUC online with forgetting. Sections 5.1 and 5.2 summarize the experimental setup and used datasets.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Prequential AUC: properties of the area under the ROC curve for data streams with concept drift

2017

Self Cite

View full text Add to dashboard Cite

Modern data-driven systems often require classifiers capable of dealing with streaming imbalanced data and concept changes. The assessment of learning algorithms in such scenarios is still a challenge, as existing online evaluation measures focus on efficiency, but are susceptible to class ratio changes over time. In case of static data, the area under the receiver operating characteristics curve, or simply AUC, is a popular measure for evaluating classifiers both on balanced and imbalanced class distributions. However, the characteristics of AUC calculated on time-changing data streams have not been studied. This paper analyzes the properties of our recent proposal, an incremental algorithm that uses a sorted tree structure with a sliding window to compute AUC with forgetting. The resulting evaluation measure, called prequential AUC, is studied in terms of: visualization over time, processing speed, differences compared to AUC calculated on blocks of examples, and consistency with AUC calculated traditionally. Simulation results show that the proposed measure is statistically consistent with AUC computed traditionally on streams without drift and comparably fast to existing evaluation procedures. Finally, experiments on real-world and synthetic data showcase characteristic properties of prequential AUC compared to classification accuracy, G-mean, Kappa, Kappa M, and recall when used to evaluate classifiers on imbalanced streams with various difficulty factors.

show abstract

Section: Area Under the Roc Curvementioning

confidence: 99%

“…Algorithm 1 lists the pseudo-code for calculating prequential AUC. Contrary to [9], here we present an extended version of the algorithm that deals with score ties.…”

Section: Prequential Aucmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

“…To make the paper self-contained, in Sect. 3 we recall our algorithm [9] for computing AUC online with forgetting. Sections 5.1 and 5.2 summarize the experimental setup and used datasets.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Prequential AUC: properties of the area under the ROC curve for data streams with concept drift

2017

Self Cite

View full text Add to dashboard Cite

show abstract

A Dynamic Ensemble Learning Framework for Data Stream Analysis and Real-Time Threat Detection

Demertzis

Iliadis

Anezakis

2018

Artificial Neural Networks and Machine Learning – ICANN 2018

View full text Add to dashboard Cite

Business Intelligence

Zimnyi¹,

Kutsche²

2015

Lecture Notes in Business Information Processing

View full text Add to dashboard Cite

Abstract. The last ten years were prolific in the statistical learning and data mining field and it is now easy to find learning algorithms which are fast and automatic. Historically a strong hypothesis was that all examples were available or can be loaded into memory so that learning algorithms can use them straight away. But recently new use cases generating lots of data came up as for example: monitoring of telecommunication network, user modeling in dynamic social network, web mining, etc. The volume of data increases rapidly and it is now necessary to use incremental learning algorithms on data streams. This article presents the main approaches of incremental supervised classification available in the literature. It aims to give basic knowledge to a reader novice in this subject.

show abstract

Prequential AUC for Classifier Evaluation and Drift Detection in Evolving Data Streams

Cited by 31 publications

References 24 publications

Prequential AUC: properties of the area under the ROC curve for data streams with concept drift

Prequential AUC: properties of the area under the ROC curve for data streams with concept drift

A Dynamic Ensemble Learning Framework for Data Stream Analysis and Real-Time Threat Detection

Business Intelligence

Contact Info

Product

Resources

About