Evaluation methods and decision theory for classification of streaming data with temporal dependence

źLiobaităź, Indrăź; Bifet, Albert; Read, Jesse; Pfahringer, Bernhard; Holmes, Geoff

doi:10.1007/s10994-014-5441-4

Cited by 104 publications

(83 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The Kappa statistic (κ) is an alternative to accuracy, that may be seen as the probability of outperforming a base classifier. Furthermore, this metric has been recently extended to take into account class skew (κ m ) [4] and temporal dependence (κ per ) [49].…”

Section: Introductionmentioning

confidence: 99%

Prequential AUC: properties of the area under the ROC curve for data streams with concept drift

2017

View full text Add to dashboard Cite

Modern data-driven systems often require classifiers capable of dealing with streaming imbalanced data and concept changes. The assessment of learning algorithms in such scenarios is still a challenge, as existing online evaluation measures focus on efficiency, but are susceptible to class ratio changes over time. In case of static data, the area under the receiver operating characteristics curve, or simply AUC, is a popular measure for evaluating classifiers both on balanced and imbalanced class distributions. However, the characteristics of AUC calculated on time-changing data streams have not been studied. This paper analyzes the properties of our recent proposal, an incremental algorithm that uses a sorted tree structure with a sliding window to compute AUC with forgetting. The resulting evaluation measure, called prequential AUC, is studied in terms of: visualization over time, processing speed, differences compared to AUC calculated on blocks of examples, and consistency with AUC calculated traditionally. Simulation results show that the proposed measure is statistically consistent with AUC computed traditionally on streams without drift and comparably fast to existing evaluation procedures. Finally, experiments on real-world and synthetic data showcase characteristic properties of prequential AUC compared to classification accuracy, G-mean, Kappa, Kappa M, and recall when used to evaluate classifiers on imbalanced streams with various difficulty factors.

show abstract

Section: Introductionmentioning

confidence: 99%

Prequential AUC: properties of the area under the ROC curve for data streams with concept drift

2017

View full text Add to dashboard Cite

show abstract

“…Twitter sentiment analysis is another example, where we want to predict if the sentiment of a new incoming tweet is positive or negative. The probability distribution generating the data may be changing, and this is why streaming methods need to be adaptive to the changes on the streams [23,24].…”

Section: Introductionmentioning

confidence: 99%

Extremely Fast Decision Tree Mining for Evolving Data Streams

Bifet

Zhang

Wang³

et al. 2017

Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

Self Cite

View full text Add to dashboard Cite

Nowadays real-time industrial applications are generating a huge amount of data continuously every day. To process these large data streams, we need fast and efficient methodologies and systems. A useful feature desired for data scientists and analysts is to have easy to visualize and understand machine learning models. Decision trees are preferred in many real-time applications for this reason, and also, because combined in an ensemble, they are one of the most powerful methods in machine learning.In this paper, we present a new system called streamDM-C++, that implements decision trees for data streams in C++, and that has been used extensively at Huawei. Streaming decision trees adapt to changes on streams, a huge advantage since standard decision trees are built using a snapshot of data, and can not evolve over time. streamDM-C++ is easy to extend, and contains more powerful ensemble methods, and a more efficient and easy to use adaptive decision trees. We compare our new implementation with VFML, the current state of the art implementation in C, and show how our new system outperforms VFML in speed using less resources.

show abstract

“…To further bridge the connections between our detection results and clustering results in [11], a recently developed measurement -Kappa Plus Statistic (KPS) [2,23] -have been proposed. KPS, defined as κ + = p0−p e 1−p e , aims to evaluate data stream classifier performance taken into account temporal dependence as well as the effectiveness (or rationality) of classifier adaptation, where p 0 is the classifier's prequential accuracy and p e is the accuracy of No-Change classifier.…”

confidence: 99%

Concept Drift Detection with Hierarchical Hypothesis Testing

Yu¹,

Abraham²

2017

Proceedings of the 2017 SIAM International Conference on Data Mining

View full text Add to dashboard Cite

When using statistical models (such as a classifier) in a streaming environment, there is often a need to detect and adapt to concept drifts to mitigate any deterioration in the model's predictive performance over time. Unfortunately, the ability of popular concept drift approaches in detecting these drifts in the relationship of the response and predictor variable is often dependent on the distribution characteristics of the data streams, as well as its sensitivity on parameter tuning. This paper presents Hierarchical Linear Four Rates (HLFR), a framework that detects concept drifts for different data stream distributions (including imbalanced data) by leveraging a hierarchical set of hypothesis tests in an online setting. The performance of HLFR is compared to benchmark approaches using both simulated and real-world datasets spanning the breadth of concept drift types. HLFR significantly outperforms benchmark approaches in terms of accuracy, G-mean, recall, delay in detection and adaptability across the various datasets.

show abstract

Evaluation methods and decision theory for classification of streaming data with temporal dependence

Cited by 104 publications

References 33 publications

Prequential AUC: properties of the area under the ROC curve for data streams with concept drift

Prequential AUC: properties of the area under the ROC curve for data streams with concept drift

Extremely Fast Decision Tree Mining for Evolving Data Streams

Concept Drift Detection with Hierarchical Hypothesis Testing

Contact Info

Product

Resources

About