LCDDF: An Adaptive and Learning based Framework with Feature Selection for Efficient Detection of Concept Drift in Data Streams

Abstract

In the contemporary era, there has been increased collaboration among machines and things due to innovative technologies like Internet of Things (IoT). With use cases of IoT pertaining to industries, there is unprecedented increase in data generation and dissemination resulting in large data streams. In this context, data stream analytics is given paramount importance but it suffers from concept drift issues leading to performance deterioration in many automation applications. There are many existing methods for automatic detection of concept drifts in data streams. However, there is need for an adaptive approach that learns dynamically through machine learning (ML) techniques. Another drawback of existing method is lack of efficient feature selection method that leverages drift detection performance. In this paper, we address these shortcomings by proposing a framework known as Learning based Concept Drift Detection Framework (LCDDF). We also proposed two algorithms, namely Concept Drift-aware Feature Engineering (CDFE) and Learning based Model Selection (LbMS), to realize the framework and improve the state of the art in detection accuracy. The former is used to perform feature engineering which concept drift-aware leading to improving quality of learning process. The latter detects best models for making an ensemble towards improving prediction performance. Our empirical study has revealed that the proposed framework with underlying algorithms outperform many state of the art methods.