Missing-value Imputation of Environment Sensors Using Multilayer Stacking with Scoring Method

Abstract

While analyzing environmental data, missing data reduce the predictive power of the model, and imputing missing data to the algorithm biases the parameter estimation, thereby increasing the uncertainty of the results. This study introduces a missing data handling model based on multilayer stacking by synthesizing the characteristics of missing data handling. The model utilizes an ensemble technique to integrate the advantages of existing models, and the final meta learner of the ensemble model is augmented with data fusion data using a Kalman filter for training data and add the features of sensor fusion for multiple identical characteristics to the model. In a situation that includes the types of missing data used in the study, the method of generating new learning data by collecting weights based on the scoring method and weighting the existing learning data has the effect of matching the measurement environment. The performance of the model improved by 20% compared with existing models utilized as nodes in an environment where normal values and various types of defects were combined. In addition, the performance improved by 30% compared with the multiple imputation by chain equations (MICE) and the single center imputation from multiple chained equations (SICE) models, which are commonly used in other sensor data with defects within a sensor group, and stable results were obtained. This shows that the proposed model reduces the cost of determining the model for various errors that may occur in environmental sensors, and, by checking how sensitive the model is to different patterns of missing data, it can be applied in various environments and improved using advanced node models in the future.