Normalization and outlier removal in class center-based firefly algorithm for missing value imputation

Abstract

A missing value is one of the factors that often cause incomplete data in almost all studies, even those that are well-designed and controlled. It can also decrease a study’s statistical power or result in inaccurate estimations and conclusions. Hence, data normalization and missing value handling are considered the major problems in the data pre-processing stage, while classification algorithms are adopted to handle numerical features. In cases where the observed data contained outliers, the missing value estimated results are sometimes unreliable or even differ greatly from the true values. Therefore, this study aims to propose the combination of normalization and outlier removals before imputing missing values on the class center-based firefly algorithm method (ON + C3FA). Moreover, some standard imputation techniques like mean, a random value, regression, as well as multiple imputation, KNN imputation, and decision tree (DT)-based missing value imputation were utilized as a comparison of the proposed method. Experimental results on the sonar dataset showed normalization and outlier removals effect in the methods. According to the proposed method (ON + C3FA), AUC, accuracy, F1-Score, Precision, Recall, and AUC-PR had 0.972, 0.906, 0.906, 0.908, 0.906, 0.61 respectively. The result showed combining normalization and outlier removals in C3-FA (ON + C3FA) was an efficient technique for obtaining actual data in handling missing values, and it also outperformed the previous studies methods with r and RMSE values of 0.935 and 0.02. Meanwhile, the Dks value obtained from this technique was 0.04, which indicated that it could maintain the values or distribution accuracy.

Smoothing Target Encoding and Class Center-Based Firefly Algorithm for Handling Missing Values in Categorical Variable

“…It also showed that C3FA + Dist had better results at a low missing data < 20%. The C3FA method had a good performance on the missing data of 40% through the MCAR mechanism which is in line with previous research [28,29]. Other experimental results showed that the C3FA-STD method produced the best performance evaluation when the dataset had a reasonably high amount of missing data (60%).…”

“…Further research conducted by the author where at the beginning the imputation method, the author employed a standardization and outlier identification strategy. The result showed combining normalization and outlier removals in C3FA was an efficient technique for obtaining actual data in handling missing values [29]. However, the imputation method with the class center-based firefly algorithm was not analyzed on the categorical datasets in other previous studies.…”

Performance evaluation of IRI-Plas 2017 model with ionosonde data measurements of ionospheric parameters

“…A reliable indicator of a set of data's degree of dispersion is the median absolute deviation (MAD). MAD is a more widely used outlier removal technique and a known statistical methodology that uses the standard deviation from the mean, which was recommended by different researchers [ [23] , [24] , [25] , [26] , [27] , [28] , [29] , [30] , [31] , [32] ]. The ionosonde measures the electron densities for each layer as well as their estimated altitudes as functions of time [ 33 , 34 ].…”

Harnessing the Power of Artificial Intelligence to Forecast Startup Success: An Empirical Evaluation of the SECURE AI Model

“…The EAS was normalized to be able to predict the target value of Predictive Entrepreneurship Opportunity (PEO). This data was pre-processed to suit ML algorithms, involving missing value imputation, outlier removal, and normalization of numerical values (Nugroho et al, 2021). Key features encapsulating the SECURE pillars were selected for their potential impact on ML model performance.…”