Adaptive infinite dropout for noisy and sparse data streams

Abstract

The ability to analyze data streams, which arrive sequentially and possibly infinitely, is increasingly vital in various online applications. However, data streams pose various challenges, including sparse and noisy data as well as concept drifts, which easily mislead a learning method. This paper proposes a simple yet robust framework, called Adaptive Infinite Dropout (aiDropout), to effectively tackle these problems. Our framework uses a dropout technique in a recursive Bayesian approach in order to create a flexible mechanism for balancing between old and new information. In detail, the recursive Bayesian approach imposes a constraint on the model parameters to make a regularization term between the current and previous mini-batches. Then, dropout whose drop rate is autonomously learned can adjust the constraint to new data. Thanks to the ability to reduce overfitting and the ensemble property of Dropout, our framework obtains better generalization, thus it effectively handles undesirable effects of noise and sparsity. In particular, theoretical analyses show that aiDropout imposes a data-dependent regularization, therefore, it can adapt quickly to sudden changes from data streams. Extensive experiments show that aiDropout significantly outperforms the state-of-the-art baselines on a variety of tasks such as supervised and unsupervised learning.

A Time-Series-Based Sample Amplification Model for Data Stream with Sparse Samples

“…The Adaptive Infinite Dropout (AIDropout) [23] and Approximate Linear Dependence (ALD) [24] method increased the processing time, which has a volatility issue and is complex while training the data as well as containing noisy data that misleads the samples and causes sudden variation in the input data stream. However, the state-of-the-art methods do not apply to large datasets, and thus the complicated data stream can cause errors [25]. Data augmentation is not efficient in time series data, as well as increasing the time complexity.…”

“…which do not share any classes with CIFAR-10. LeNet [25] is used as the feature extractor, and batch normalization is added. The data set has the time series length of 1639.…”