Ensemble of distributed learners for online classification of dynamic data streams

Abstract

Abstract-We present an efficient distributed online learning scheme to classify data captured from distributed, heterogeneous, and dynamic data sources. Our scheme consists of multiple distributed local learners, that analyze different streams of data that are correlated to a common event that needs to be classified. Each learner uses a local classifier to make a local prediction. The local predictions are then collected by each learner and combined using a weighted majority rule to output the final prediction. We propose a novel online ensemble learning algorithm to update the aggregation rule in order to adapt to the underlying data dynamics. We rigorously determine a bound for the worst-case mis-classification probability of our algorithm which depends on the mis-classification probabilities of the best static aggregation rule, and of the best local classifier. Importantly, the worst-case mis-classification probability of our algorithm tends asymptotically to 0 if the mis-classification probability of the best static aggregation rule or the mis-classification probability of the best local classifier tend to 0. Then we extend our algorithm to address challenges specific to the distributed implementation and we prove new bounds that apply to these settings. Finally, we test our scheme by performing an evaluation study on several data sets. When applied to data sets widely used by the literature dealing with dynamic data streams and concept drift, our scheme exhibits performance gains ranging from 34% to 71% with respect to state-of-the-art solutions.

Context-based unsupervised ensemble learning and feature ranking

“…The majority rule is the most widely used fusion rule for ensemble learning since the earliest studies of the subject (Blum 1995;Breiman 1996;Canzian et al 2013;Fan et al 1999;Freund and Schapire 1997;Hadavandi et al 2015;Herbster and Warmuth 1998;Littlestone and Warmuth 1994;Schapire 1990;Stahl et al 2015;Wang et al 2003Wang et al , 2015. In majority rule, the combiner's final decision is made by taking a vote among all the experts at each instant.…”

“…In this experiment, we set c = 0.05. We should point out that the experts use all the 9 features all the time; however, at each experiment only one of the features is available as a context for the combiner and the rest are unknown unsupervised versions of the method of tracking the best expert (MTBE) (Herbster and Warmuth 1998), adaptive Perceptron weighted majority rule (APMR) (Canzian et al 2013), and the supervised optimal fusion rule (SOFR) (Chair and Varshney 1986), in term of probability of error, p e . In MTBE, at each instance, the decision of each expert is compared against the actual label in the supervised version (or the pool of the decisions in the unsupervised version).…”

Real-time stream mining: online knowledge extraction using classifier networks

“…Online learning schemes to learn the optimal classifier operating points are proposed and analyzed in [16][17][18]. These works consider a network of processing nodes that observe distributed, heterogeneous, and dynamic data sources.…”

“…Reference [16] focuses on regression problems and proposes an online learning scheme in which the amount of information exchanged depends on the estimated correlation among processing nodes; this allows trade-off of system performance and amount of exchanged information. Instead, [17] focuses on classification problems and proposes a learning scheme with the following features: • It requires a minimal exchange of information.…”

ToPs: Ensemble Learning With Trees of Predictors

“…Index Terms-Ensemble learning, Model tree, Personalized predictive models I. INTRODUCTION E NSEMBLE methods [1], [2], [3], [4] are general techniques in machine learning that combine several learners; these techniques include bagging [5], [6], [7], boosting [8], [9] and stacking [10]. Ensemble methods frequently improve predictive performance.…”