A Method of Pruning and Random Replacing of Known Values for Comparing Missing Data Imputation Models for Incomplete Air Quality Time Series

Abstract

The data obtained from air quality monitoring stations, which are used to carry out studies using data mining techniques, present the problem of missing values. This paper describes a research work on missing data imputation. Among the most common methods, the method that best imputes values to the available data set is analysed. It uses an algorithm that randomly replaces all known values in a dataset once with imputed values and compares them with the actual known values, forming several subsets. Data from seven stations in the Silesian region (Poland) were analyzed for hourly concentrations of four pollutants: nitrogen dioxide (NO2), nitrogen oxides (NOx), particles of 10 μm or less (PM10) and sulphur dioxide (SO2) for five years. Imputations were performed using linear imputation (LI), predictive mean matching (PMM), random forest (RF), k-nearest neighbours (k-NN) and imputation by Kalman smoothing on structural time series (Kalman) methods and performance evaluations were performed. Once the comparison method was validated, it was determine that, in general, Kalman structural smoothing and the linear imputation methods best fitted the imputed values to the data pattern. It was observed that each imputation method behaves in an analogous way for the different stations The variables with the best results are NO2 and SO2. The UMI method is the worst imputer for missing values in the data sets.

Handling missing data in the time-series data from wearables

“…Kalman prediction is used for various types of physiological data like heart rate variability and body weight variability [69][70][71] as well as for other time series data [72]. Luis Alfonso et al [73] used data imputation techniques on air quality data and evaluated the results using RMSE values. The study compared the performance of the Kalman smoothing algorithm with other imputation methods such as kNN and RF (Random Forest).…”

Imputation of missing microclimate data of coffee-pine agroforestry with machine learning

“…Additionally, this research introduces a random pruning and replacement method for known values to compare missing data imputation models. This approach offers a practical means to evaluate and choose appropriate imputation models for incomplete datasets, potentially proving valuable in the context of recovering lost microclimate data in the UB Forest [12].…”

“…Anomaly detection frameworks for wearable device data, emphasizing the increasing use of wearable devices in clinical studies, demonstrate the need for robust imputation methods in this domain [15]. Furthermore, proposes a random pruning and replacement method of known values to compare missing data imputation models, providing insight into the behavior of different imputation methods for incomplete air quality time series [12]. These studies collectively underscore the need for efficient, accurate, and domain-specific imputation methods to address missing data across various fields, from proteomics to health records and environmental sensing.…”

Air Quality—Meteorology Correlation Modeling Using Random Forest and Neural Network

“…They compared and analyzed the prediction performance of three ML algorithms and discovered that the RF algorithm was the most accurate in predicting meteorological factors and air pollutants. Menéndez García et al [21] designed an RF model to forecast the air quality of Swiss meteorological factors and air pollutants. They proved that RF had excellent accuracy in air quality prediction.…”