Multithreaded and Spark parallelization of feature selection filters

Eiras-Franco, Carlos; Bolón-Canedo, Verónica; Ramos, Sabela; González-Domínguez, Jorge; Alonso‐Betanzos, Amparo; Touriño, Juan

doi:10.1016/j.jocs.2016.07.002

Cited by 35 publications

(20 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The method seeks to identify and delete as much as possible irreparable and redundant information. It simplifies the data set both in size and in complexity of understanding, which leads to simpler and faster classification algorithms, better problem comprehension and reduced storage requirements (Eiras‐Franco et al, ; Phyu & Oo, ).…”

Section: Methodsmentioning

confidence: 99%

Prediction for global African swine fever outbreaks based on a combination of random forest algorithms and meteorological data

Liang

et al. 2019

Transbound Emerg Dis

View full text Add to dashboard Cite

African swine fever (ASF) is a virulent infectious disease of pigs. As there is no effective vaccine and treatment method at present, it poses a great threat to the pig industry once it breaks out. In this paper, we used ASF outbreak data and the WorldClim database meteorological data and selected the CfsSubset Evaluator‐Best First feature selection method combined with the random forest algorithms to construct an African swine fever outbreak prediction model. Subsequently, we also established a test set for data other than modelling, and the accuracy accuracy value range of the model on the independent test set was 76.02%–84.64%, which indicated that the modelling effect was better and the prediction accuracy was higher than previous estimates. In addition, logistic regression analysis was conducted on 12 features used for modelling and the ROC curves were drawn. The results showed that the bio14 features (precipitation of driest month) had the largest contribution to the outbreak of ASF, and it was speculated that the outbreak of the epidemic was significantly related to precipitation. Finally, we used this qualitative prediction model to build a global online prediction system for ASF outbreaks, in the hope that this study will help to decision‐makers who can then take the relevant prevention and control measures in order to prevent the further spread of future epidemics of the disease.

show abstract

Section: Methodsmentioning

confidence: 99%

Prediction for global African swine fever outbreaks based on a combination of random forest algorithms and meteorological data

Liang

et al. 2019

Transbound Emerg Dis

View full text Add to dashboard Cite

show abstract

“…The experiments tested and compared time-efficiency and scalability for the horizontal and vertical DiCFS approaches so as to check whether they improved on the original non-distributed version of the CFS. We also tested and compared execution times with that reported in the recently published research by Eiras-Franco et al [10] into a distributed version of CFS for regression problems.…”

Section: Methodsmentioning

confidence: 99%

“…We compare the two versions -DiCFS-hp and DiCFSvp, respectively -and also compare them with a baseline, represented by the classical non-distributed implementation of CFS in WEKA [17]. Finally, their benefits in terms of reduced execution time are compared with those of the CFS version developed by Eiras-Fanco et al [10] for regression problems. The results show that the time-efficiency and scalability of our two versions are an improvement on those of the original version of the CFS; furthermore, similar or improved execution times are obtained with respect to the Eiras-Franco et al [10] regression version.…”

Section: Introductionmentioning

confidence: 99%

“…Finally, their benefits in terms of reduced execution time are compared with those of the CFS version developed by Eiras-Fanco et al [10] for regression problems. The results show that the time-efficiency and scalability of our two versions are an improvement on those of the original version of the CFS; furthermore, similar or improved execution times are obtained with respect to the Eiras-Franco et al [10] regression version. In the interest of reproducibility, our software and sources are available as a Spark package 1 called DiCFS, with a corresponding mirror in Github.…”

Section: Introductionmentioning

confidence: 99%

“…Section 5 describes and discusses our DiCFS-hp and DiCFS-vp versions of the CFS algorithm. Section 6 describes our experiments to compare results for DiCFS-hp and DiCFS-vp, the WEKA approach and the Eiras-Fanco et al [10] approach. Finally, conclusions and future work are outlined in Section 7.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Distributed correlation-based feature selection in spark

Palma-Mendoza

de‐Marcos

Rodríguez

et al. 2019

Information Sciences

Self Cite

View full text Add to dashboard Cite

Feature selection (FS) is a key preprocessing step in data mining. CFS (Correlation-Based Feature Selection) is an FS algorithm that has been successfully applied to classification problems in many domains. We describe Distributed CFS (DiCFS) as a completely redesigned, scalable, parallel and distributed version of the CFS algorithm, capable of dealing with the large volumes of data typical of big data applications. Two versions of the algorithm were implemented and compared using the Apache Spark cluster computing model, currently gaining popularity due to its much faster processing times than Hadoop's MapReduce model. We tested our algorithms on four publicly available datasets, each consisting of a large number of instances and two also consisting of a large number of features. The results show that our algorithms were superior in terms of both time-efficiency and scalability. In leveraging a computer cluster, they were able to handle larger datasets than the non-distributed WEKA version while maintaining the quality of the results, i.e., exactly the same features were returned by our algorithms when compared to the original algorithm available in WEKA.

show abstract

Scalable feature selection using ReliefF aided by locality‐sensitive hashing

Eiras-Franco

Guijarro-Berdiñas

Alonso‐Betanzos

et al. 2021

Int J of Intelligent Sys

View full text Add to dashboard Cite

Feature selection algorithms, such as ReliefF, are very important for processing high‐dimensionality data sets. However, widespread use of popular and effective such algorithms is limited by their computational cost. We describe an adaptation of the ReliefF algorithm that simplifies the costliest of its step by approximating the nearest neighbor graph using locality‐sensitive hashing (LSH). The resulting ReliefF‐LSH algorithm can process data sets that are too large for the original ReliefF, a capability further enhanced by distributed implementation in Apache Spark. Furthermore, ReliefF‐LSH obtains better results and is more generally applicable than currently available alternatives to the original ReliefF, as it can handle regression and multiclass data sets. The fact that it does not require any additional hyperparameters with respect to ReliefF also avoids costly tuning. A set of experiments demonstrates the validity of this new approach and confirms its good scalability.

show abstract

Multithreaded and Spark parallelization of feature selection filters

Cited by 35 publications

References 6 publications

Prediction for global African swine fever outbreaks based on a combination of random forest algorithms and meteorological data

Prediction for global African swine fever outbreaks based on a combination of random forest algorithms and meteorological data

Distributed correlation-based feature selection in spark

Scalable feature selection using ReliefF aided by locality‐sensitive hashing

Contact Info

Product

Resources

About