A context-intensive approach to imputation of missing values in data sets from networks of environmental monitors

Abstract

Although networks of environmental monitors are constantly improving through advances in technology and management, instances of missing data still occur. Many methods of imputing values for missing data are available, but they are often difficult to use or produce unsatisfactory results. I-Bot (short for "Imputation Robot") is a context-intensive approach to the imputation of missing data in data sets from networks of environmental monitors. I-Bot is easy to use and routinely produces imputed values that are highly reliable. I-Bot is described and demonstrated using more than 10 years of California data for daily maximum 8-hr ozone, 24-hr PM 2.5 (particulate matter with an aerodynamic diameter <2.5 μm), mid-day average surface temperature, and midday average wind speed. I-Bot performance is evaluated by imputing values for observed data as if they were missing, and then comparing the imputed values with the observed values. In many cases, I-Bot is able to impute values for long periods with missing data, such as a week, a month, a year, or even longer. Qualitative visual methods and standard quantitative metrics demonstrate the effectiveness of the I-Bot methodology.Implications: Many resources are expended every year to analyze and interpret data sets from networks of environmental monitors. A large fraction of those resources is used to cope with difficulties due to the presence of missing data. The I-Bot method of imputing values for such missing data may help convert incomplete data sets into virtually complete data sets that facilitate the analysis and reliable interpretation of vital environmental data. PAPER HISTORY

A Novel Method for Air Quality Data Imputation by Nuclear Norm Minimization

“…The proposed algorithm can be implemented on specific hardware, such as graphic processing unit (GPU) consisting of thousands of cores, to further improve the computational efficiency. The comparison of our method with other air quality data 9 Journal of Sensors imputation methods [32][33][34] is also an interesting research topic in the future.…”

Spatial statistical modelling of urban particulate matter

“…This would entail consideration of other models like the generalized additive and extreme value models (De Jong et al, 2016). Air quality datasets for each station are large and implementing our multiple imputation method for several stations simultaneously presented a challenge in terms of memory in R. Therefore further work is required on improving computational efficiency if a system like I-BOT is to be built to assist air quality officers with imputations using our method (Larsen and Shah, 2016). Another area of further research concerns visual diagnostics for multivariate imputations an efficient way to check whether the imputed values are sensible when one has several stations to consider at a time (Abayomi et al, 2008).…”

“…Our novel approach to this problem was to build a sequence of regression models starting with regressing meteorological data from air quality stations with data from neighbouring weather stations, then imputing missing gaseous pollutant values using the completed meteorological data and eventually imputing missing particulate matter values. The closest in concept to our approach is the Imputation Robot system (I-BOT) of the California Air Resources Board which considers variable-by-variable single imputation through linear regression of missing values at a target air quality station using values from a donor station which is chosen based on correlation strength and maximum distance of 100 km (Larsen and Shah, 2016). Our approach differs from I-BOT in being a multiple imputation through bootstrap method where multivariate regression models are developed instead of univariate regressions.…”