Boosting decision stumps for dynamic feature selection on data streams

Barddal, Jean Paul; Enembreck, Fabrício; Gomes, Heitor Murilo; Bifet, Albert; Pfahringer, Bernhard

doi:10.1016/j.is.2019.02.003

Cited by 30 publications

(18 citation statements)

References 47 publications

(67 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…-Feature drift This is a type of change in data streams that happens when a subset of features becomes, or stops to be, relevant to the learning task (Barddal et al 2017). Additionally, new features may emerge (thus extending the feature space), while the old ones may cease to arrive (Barddal et al 2019a). Therefore, classifiers need to adapt to these changes in feature space (Barddal et al 2016) by performing a dynamic feature selection (Yuan et al 2018;Barddal et al 2019b), using randomness in selected features (Abdulsalam et al 2011), or employing a sliding window and feature space transformation (Nguyen et al 2012).…”

Section: Overviewmentioning

confidence: 99%

Kappa Updated Ensemble for drifting data stream mining

2019

View full text Add to dashboard Cite

Learning from data streams in the presence of concept drift is among the biggest challenges of contemporary machine learning. Algorithms designed for such scenarios must take into an account the potentially unbounded size of data, its constantly changing nature, and the requirement for real-time processing. Ensemble approaches for data stream mining have gained significant popularity, due to their high predictive capabilities and effective mechanisms for alleviating concept drift. In this paper, we propose a new ensemble method named Kappa Updated Ensemble (KUE). It is a combination of online and block-based ensemble approaches that uses Kappa statistic for dynamic weighting and selection of base classifiers. In order to achieve a higher diversity among base learners, each of them is trained using a different subset of features and updated with new instances with given probability following a Poisson distribution. Furthermore, we update the ensemble with new classifiers only when they contribute positively to the improvement of the quality of the ensemble. Finally, each base classifier in KUE is capable of abstaining itself for taking a part in voting, thus increasing the overall robustness of KUE. An extensive experimental study shows that KUE is capable of outperforming state-of-the-art ensembles on standard and imbalanced drifting data streams while having a low computational complexity. Moreover, we analyze the use of Kappa versus accuracy to drive the criterion to select and update the classifiers, the contribution of the abstaining mechanism, the contribution of the diversification of classifiers, and the contribution of the hybrid architecture to update the classifiers in an online manner.

show abstract

Section: Overviewmentioning

confidence: 99%

Kappa Updated Ensemble for drifting data stream mining

2019

View full text Add to dashboard Cite

show abstract

“…This selection method was shown to improve the performance of two different types of classifiers. Adaptive Boosting for FS (ABFS) was introduced by Barddal et al [6] and uses a combination of boosting [23] and decision stumps (a decision tree whereby the root node is connected to the terminal nodes) to select features. Boosting gives higher weights to training instances which are harder to classify, then decision stumps are used to select features from these difficult-to-classify samples.…”

Section: Related Workmentioning

confidence: 99%

Dynamic Feature Selection for Clustering High Dimensional Data Streams

Fahy

Yang

2019

IEEE Access

View full text Add to dashboard Cite

Change in a data stream can occur at the concept level and at the feature level. Change at the feature level can occur if new, additional features appear in the stream or if the importance and relevance of a feature changes as the stream progresses. This type of change has not received as much attention as concept-level change. Furthermore, a lot of the methods proposed for clustering streams (density-based, graph-based, and grid-based) rely on some form of distance as a similarity metric and this is problematic in high-dimensional data where the curse of dimensionality renders distance measurements and any concept of ''density'' difficult. To address these two challenges we propose combining them and framing the problem as a feature selection problem, specifically a dynamic feature selection problem. We propose a dynamic feature mask for clustering high dimensional data streams. Redundant features are masked and clustering is performed along unmasked, relevant features. If a feature's perceived importance changes, the mask is updated accordingly; previously unimportant features are unmasked and features which lose relevance become masked. The proposed method is algorithm-independent and can be used with any of the existing density-based clustering algorithms which typically do not have a mechanism for dealing with feature drift and struggle with high-dimensional data. We evaluate the proposed method on four density-based clustering algorithms across four high-dimensional streams; two text streams and two image streams. In each case, the proposed dynamic feature mask improves clustering performance and reduces the processing time required by the underlying algorithm. Furthermore, change at the feature level can be observed and tracked.INDEX TERMS Data stream clustering, dynamic feature selection, feature drift, feature evolution, unsupervised feature selection.

show abstract

“…if |AA|==m (13) \*Selecting features in the storage window*\ (14) for ii = 1 to |AA| do (15) Calculate the feature repulsion loss for each feature by Equation (12); (16) end for (17) Select the feature f ii with the largest FRL; (18) S=S∪{f ii } (19) AA=AA-{f ii }; (20) end if (21) end if (22) end for (23) end for (24) end while; (25) Until no feature groups are available; (26) Return S;…”

Section: Streaming Feature Selectionmentioning

confidence: 99%

“…Nevertheless, the aforementioned algorithms assume full knowledge of the entire feature space, while features emerge incrementally or dynamically in numerous modern applications [11,21,22]. For example, hot news is constantly updated with different keywords in each piece of news, indicating that features of every data sample cannot be necessarily available in advance.…”

Section: Introductionmentioning

confidence: 99%

Streaming Feature Selection for Multi-Label Data with Dynamic Sliding Windows and Feature Repulsion Loss

Cheng

2019

Entropy

View full text Add to dashboard Cite

In recent years, there has been a growing interest in the problem of multi-label streaming feature selection with no prior knowledge of the feature space. However, the algorithms proposed to handle this problem seldom consider the group structure of streaming features. Another shortcoming arises from the fact that few studies have addressed atomic feature models, and particularly, few have measured the attraction and repulsion between features. To remedy these shortcomings, we develop the streaming feature selection algorithm with dynamic sliding windows and feature repulsion loss (SF-DSW-FRL). This algorithm is essentially carried out in three consecutive steps. Firstly, within dynamic sliding windows, candidate streaming features that are strongly related to the labels in different feature groups are selected and stored in a fixed sliding window. Then, the interaction between features is measured by a loss function inspired by the mutual repulsion and attraction between atoms in physics. Specifically, one feature attraction term and two feature repulsion terms are constructed and combined to create the feature repulsion loss function. Finally, for the fixed sliding window, the best feature subset is selected according to this loss function. The effectiveness of the proposed algorithm is demonstrated through experiments on several multi-label datasets, statistical hypothesis testing, and stability analysis.

show abstract

Boosting decision stumps for dynamic feature selection on data streams

Cited by 30 publications

References 47 publications

Kappa Updated Ensemble for drifting data stream mining

Kappa Updated Ensemble for drifting data stream mining

Dynamic Feature Selection for Clustering High Dimensional Data Streams

Streaming Feature Selection for Multi-Label Data with Dynamic Sliding Windows and Feature Repulsion Loss

Contact Info

Product

Resources

About