A Comparative Study of Various Methods for Handling Missing Data in UNSODA

Abstract

UNSODA, a free international soil database, is very popular and has been used in many fields. However, missing soil property data have limited the utility of this dataset, especially for data-driven models. Here, three machine learning-based methods, i.e., random forest (RF) regression, support vector (SVR) regression, and artificial neural network (ANN) regression, and two statistics-based methods, i.e., mean and multiple imputation (MI), were used to impute the missing soil property data, including pH, saturated hydraulic conductivity (SHC), organic matter content (OMC), porosity (PO), and particle density (PD). The missing upper depths (DU) and lower depths (DL) for the sampling locations were also imputed. Before imputing the missing values in UNSODA, a missing value simulation was performed and evaluated quantitatively. Next, nonparametric tests and multiple linear regression were performed to qualitatively evaluate the reliability of these five imputation methods. Results showed that RMSEs and MAEs of all features fluctuated within acceptable ranges. RF imputation and MI presented the lowest RMSEs and MAEs; both methods are good at explaining the variability of data. The standard error, coefficient of variance, and standard deviation decreased significantly after imputation, and there were no significant differences before and after imputation. Together, DU, pH, SHC, OMC, PO, and PD explained 91.0%, 63.9%, 88.5%, 59.4%, and 90.2% of the variation in BD using RF, SVR, ANN, mean, and MI, respectively; and this value was 99.8% when missing values were discarded. This study suggests that the RF and MI methods may be better for imputing the missing data in UNSODA.

Long‐term effects of management intensity and bioclimatic variables on leatherjacket (Tipula paludosa Meigen) populations at farm scale

“…The number of fields sampled per farm were not uniform throughout, and the managerial variables were inconsistently recorded each year. As is common practice within this research area (Fu et al., 2021), data with complete sets of values were used for statistical analysis, carried out using R 4.0.2 (R Core Team, 2021). The dataset finally used for management intensity analysis therefore included 16 farms, 43 fields and a total of 188 observations across the 10‐year timespan (Table S1).…”

Comparison of Classification and Regression Model Approaches on the Main Causes of Stroke with Symbolic Regression Feyn Qlattice

“…The dataset is subsequently processed to optimize the performance of the machine learning model. The initial stage of data processing involves removing all null or NaN data, a process commonly referred to as "handling missing data" [26]. The next step is converting categorical data into numerical data.…”

Systematic Review of Using Machine Learning in Imputing Missing Values

“…For instance, the K-nearest neighbor is widely used because of its simplicity and high performance. However, K-NN performs poorly in large datasets and high-dimensional data contexts [47]. Additionally, many researchers stumble while linking different features in a dataset.…”

“…Hence, clinical researchers should be aware not to exceed this threshold while simulating missing values. Furthermore, outliers can also affect the imputation performance by causing a large RMSE if not handled correctly by deleting or replacing them [47].…”

“…Another important factor is the dataset, which is specified by its size, type, and level of correlation between variables. For instance, deep learning techniques work better with large datasets, while support vector machines and K-NN achieve high performance in small datasets [47]. Different datasets have different data types, such as time-series datasets, which work better with recurrent neural networks [108].…”