Low Level Big Data Processing

Salvador-Meneses, Jaime; Ruiz-Chavez, Zoila; García-Rodríguez, José

doi:10.5220/0007227103470352

Cited by 1 publication

(1 citation statement)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This implies that to store a numerical value it is necessary to use 32 bits (or its equivalent in 2 or 1 byte). As an example, we will consider the case in which the information is represented as a set of four-bytes integer values (some frameworks use this representation [ 24 ]).…”

Section: Introductionmentioning

confidence: 99%

Compressed kNN: K-Nearest Neighbors with Data Compression

Salvador-Meneses

Ruiz-Chavez

García-Rodríguez

2019

Entropy

View full text Add to dashboard Cite

The kNN (k-nearest neighbors) classification algorithm is one of the most widely used non-parametric classification methods, however it is limited due to memory consumption related to the size of the dataset, which makes them impractical to apply to large volumes of data. Variations of this method have been proposed, such as condensed KNN which divides the training dataset into clusters to be classified, other variations reduce the input dataset in order to apply the algorithm. This paper presents a variation of the kNN algorithm, of the type structure less NN, to work with categorical data. Categorical data, due to their nature, can be compressed in order to decrease the memory requirements at the time of executing the classification. The method proposes a previous phase of compression of the data to then apply the algorithm on the compressed data. This allows us to maintain the whole dataset in memory which leads to a considerable reduction of the amount of memory required. Experiments and tests carried out on known datasets show the reduction in the volume of information stored in memory and maintain the accuracy of the classification. They also show a slight decrease in processing time because the information is decompressed in real time (on-the-fly) while the algorithm is running.

show abstract

Section: Introductionmentioning

confidence: 99%