The Effect of Using Data Pre-Processing by Imputations in Handling Missing Values

Abstract

The evolution of big data analytics through machine learning and artificial intelligence techniques has caused organizations in a wide range of sectors including health, manufacturing, e-commerce, governance, and social welfare to realize the value of massive volumes of data accumulating on web-based repositories daily. This has led to the adoption of data-driven decision models; for example, through sentiment analysis in marketing where produces leverage customer feedback and reviews to develop customer-oriented products. However, the data generated in real-world activities is subject to errors resulting from inaccurate measurements or fault input devices, which may result in the loss of some values. Missing attribute/variable values make data unsuitable for decision analytics due to noises and inconsistencies that create bias. The objective of this paper was to explore the problem of missing data and develop an advanced imputation model based on Machine Learning and implemented on K-Nearest Neighbor (KNN) algorithm in R programming language as an approach to handle missing values. The methodology used in this paper relied on the applying advanced machine learning algorithms with high-level accuracy in pattern detection and predictive analytics on the existing imputation techniques, which handle missing values by random replacement or deletion.. According to the results, advanced imputation technique based on machine learning models replaced missing values from a dataset with 89.5% accuracy. The experimental results showed that pre-processing by imputation delivers high-level performance efficiency in handling missing data values. These findings are consistent with the key idea of paper, which is to explore alternative imputation techniques for handling missing values to improve the accuracy and reliability of decision insights extracted from datasets.

Pancreatic Cancer Classification Using Missing Data Imputation And Cluster-Based Undersampling Methods: A Comparative Analysis With Multiple Machine Learning Algorithms

“…These findings are consistent with other studies that have shown that imputation methods, including kNN, are reliable methods for missing values estimation and can help improve the classification performance [14] [35] [36] . A study realized by [36] randomly inserted missing values in an existing dataset to evaluate kNN imputation accuracy and obtained a 89.5% accuracy rate using this method.…”

“…These findings are consistent with other studies that have shown that imputation methods, including kNN, are reliable methods for missing values estimation and can help improve the classification performance [14] [35] [36] . A study realized by [36] randomly inserted missing values in an existing dataset to evaluate kNN imputation accuracy and obtained a 89.5% accuracy rate using this method. Reference [14] compared different imputation techniques in a breast cancer dataset and kNN presented the highest accuracy averages to 4 out of 7 classifiers analyzed when compared to other imputation methods.…”

Leveraging AI for sustainable accounting: Developing models for environmental impact assessment and reporting

“…This step involves resolving inconsistencies, such as differing units of measurement and temporal or spatial resolutions. Removing or correcting errors, such as missing values, outliers, and duplicates (Karrar et al, 2022). Techniques such as imputation, interpolation, and anomaly detection can be used to address these issues.…”

Application of Regularized Logistic Regression and Artificial Neural Network Model for Ozone Classification across El Paso County, Texas, United States

“…o Preprocessing and normalization: Proper preprocessing steps, such as handling missing values, scaling features, and handling outliers, can impact the accuracy of both LR and ANN models. Different preprocessing techniques may be more suitable for different models, and their proper application can enhance accuracy [22].…”