Enabling Smart Data: Noise filtering in Big Data classification

Abstract

In any knowledge discovery process the value of extracted knowledge is directly related to the quality of the data used. Big Data problems, generated by massive growth in the scale of data observed in recent years, also follow the same dictate. A common problem affecting data quality is the presence of noise, particularly in classification problems, where label noise refers to the incorrect labeling of training instances, and is known to be a very disruptive feature of data. However, in this Big Data era, the massive growth in the scale of the data poses a challenge to traditional proposals created to tackle noise, as they have difficulties coping with such a large amount of data. New algorithms need to be proposed to treat the noise in Big Data problems, providing high quality and clean data, also known as Smart Data. In this paper, two Big Data preprocessing approaches to remove noisy examples are proposed: an homogeneous ensemble and an heterogeneous ensemble filter, with special emphasis in their scalability and performance traits. The obtained results show that these proposals enable the practitioner to efficiently obtain a Smart Dataset from any Big Data classification problem.accepted that we have entered the Big Data era [31]. Big Data is the set of technologies that make processing such large amounts of data possible [7], while most of the classic knowledge extraction methods cannot work in a Big Data environment because they were not conceived for it.Big Data as concept is defined around five aspects: data volume, data velocity, data variety, data veracity and data value [24]. While the volume, variety and velocity aspects refer to the data generation process and how to capture and store the data, veracity and value aspects deal with the quality and the usefulness of the data. These two last aspects become crucial in any Big Data process, where the extraction of useful and valuable knowledge is strongly influenced by the quality of the used data.In Big Data, the usage of traditional preprocessing techniques [16,34,18] to enhance the data is even more time consuming and resource demanding, being unfeasible in most cases. The lack of efficient and affordable preprocessing techniques implies that the problems in the data will affect the models extracted. Among all the problems that may appear in the data, the presence of noise in the dataset is one of the most frequent. Noise can be defined as the partial or complete alteration of the information gathered for a data item, caused by an exogenous factor not related to the distribution that generates the data. Learning from noisy data is an important topic in machine learning, data mining and pattern recognition, as real world data sets may suffer from imperfections in data acquisition, transmission, storage, integration and categorization. Noise will lead to excessively complex models with deteriorated performance [49], resulting in even larger computing times for less value.The impact of noise in Big Data, among other pernicious traits, has not been disrega...

Transforming big data into smart data: An insight on the use of the k‐nearest neighbors algorithm to obtain quality data

“…As we have indicated in Section 4.2, noise filtering is a popular option in these cases, which becomes even more helpful in Big Data environments as noise filters reduce the size of the datasets. However, designing Big Data noise filters is a challenge and only some prior designs and methods can be found in the literature Zerhari (); García‐Gil, Luengo, García, and Herrera (). On the other hand, k‐NN has been the seminal method to remove redundant and noisy instances in learning problems.…”

“…Other relevant examples of this family of methods are: All‐kNN Tomek (), NCN‐Edit Sánchez et al () or RNG Sánchez, Pla, and Ferri (). A distributed version of the ENN algorithm based on Apache Spark is proposed in García‐Gil et al () for very large datasets. This distributed version of ENN performs a global filtering of the instances, considering the whole dataset at once.…”

From Big to Smart Data: Iterative ensemble filter for noise filtering in Big Data classification

“…As a classifier, we use Spark's MLlib implementation of a decision tree . We compare our method with the most recent and best‐performing proposal in the literature for noise‐cleaning in Big Data, homogeneous ensemble for Big Data (HME‐BD) . We show that, for some problems, the classifier benefits from the noise filtering even when no noise is added.…”

“…In other words, Smart Data refers to the challenge of transforming information into knowledge. Smart Data aims to separate the raw (or Big) part of the data (volume/velocity), from the Smart part of it (veracity/value) . Therefore, Smart Data is focused on extracting valuable knowledge from data, in the form of a subset, that contains enough quality for a successful data mining process …”

“…That is why there is a special need for noise filters in Big Data. Although we can find many proposals for dealing with noise for normal‐sized data in the literature, in Big Data scenarios we can find only a handful of proposals devoted to this problem …”