From Big to Smart Data: Iterative ensemble filter for noise filtering in Big Data classification

Abstract

The quality of the data is directly related to the quality of the models drawn from that data. For that reason, many research is devoted to improve the quality of the data and to amend errors that it may contain. One of the most common problems is the presence of noise in classification tasks, where noise refers to the incorrect labeling of training instances. This problem is very disruptive, as it changes the decision boundaries of the problem. Big Data problems pose a new challenge in terms of quality data due to the massive and unsupervised accumulation of data. This Big Data scenario also brings new problems to classic data preprocessing algorithms, as they are not prepared for working with such amounts of data, and these algorithms are key to move from Big to Smart Data. In this paper, an iterative ensemble filter for removing noisy instances in Big Data scenarios is proposed. Experiments carried out in six Big Data datasets have shown that our noise filter outperforms the current state‐of‐the‐art noise filter in Big Data domains. It has also proved to be an effective solution for transforming raw Big Data into Smart Data.

LR-BCA: Label Ranking for Bridge Condition Assessment

“…Consequently, they proposed an ensemble filter based on C4.5, 1-NN, and linear machine. Similarly, some researchers [11], [12] proposed iterative partition filters that iteratively removed detected noise. In these methods, N filters were learned based on N group of N − 1 partitions and voted to detect noise on the entire dataset.…”

Redundancy and Complexity Metrics for Big Data Classification: Towards Smart Data

“…values [11] and reducing redundant [12] and noisy data [13] to obtain quality data from big datasets. In addition, there are contributions as proposed by Liu et al [14], where the results are improved and the runtime reduced in classification problems by selecting the appropriate classification rule according to a given neighborhood, instead of using the complete dataset.…”

Trust‐aware generative adversarial network with recurrent neural network for recommender systems

“…With the rapid development of Internet of Things (IoTs) technologies, 2–5 the current explosive growth of data 6 makes the information overload more and more serious, which can be solved by recommender systems 7–10 . Generally, the recommender systems can be classified into three categories: collaborative filtering methods, content‐based methods, and hybrid methods 11 .…”