A Review of Missing Sensor Data Imputation Methods

Faizin, Rahmat Nur; Riasetiawan, Mardhani; Ashari, Ahmad

doi:10.1109/icst47872.2019.9166287

Cited by 8 publications

(3 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Missing values are divided into three mechanisms, as de ned by Little and Rubin (1987): MCAR, missing at random (MAR), and missing not at random (MNAR). Each type is characterized as follows [22]. MCAR is unrelated to the nature of the variable.…”

Section: Missing Sensor Data Typementioning

confidence: 99%

Missing-value Imputation of Environment Sensors Using Multilayer Stacking with Scoring Method

Chun

Cho

2023

Preprint

View full text Add to dashboard Cite

While analyzing environmental data, missing data reduce the predictive power of the model, and imputing missing data to the algorithm biases the parameter estimation, thereby increasing the uncertainty of the results. This study introduces a missing data handling model based on multilayer stacking by synthesizing the characteristics of missing data handling. The model utilizes an ensemble technique to integrate the advantages of existing models, and the final meta learner of the ensemble model is augmented with data fusion data using a Kalman filter for training data and add the features of sensor fusion for multiple identical characteristics to the model. In a situation that includes the types of missing data used in the study, the method of generating new learning data by collecting weights based on the scoring method and weighting the existing learning data has the effect of matching the measurement environment. The performance of the model improved by 20% compared with existing models utilized as nodes in an environment where normal values and various types of defects were combined. In addition, the performance improved by 30% compared with the multiple imputation by chain equations (MICE) and the single center imputation from multiple chained equations (SICE) models, which are commonly used in other sensor data with defects within a sensor group, and stable results were obtained. This shows that the proposed model reduces the cost of determining the model for various errors that may occur in environmental sensors, and, by checking how sensitive the model is to different patterns of missing data, it can be applied in various environments and improved using advanced node models in the future.

show abstract

Section: Missing Sensor Data Typementioning

confidence: 99%

Missing-value Imputation of Environment Sensors Using Multilayer Stacking with Scoring Method

Chun

Cho

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…However, in real livestock environments, IoT (Internet of Things) sensor devices are used to collect odor substance data. Due to the nature of data collection using sensors, data being missing due to various causes is a frequent problem [36]. Therefore, we consider that it is applicable to the actual livestock environment that we used data with missing values.…”

Section: Data Availability Statementmentioning

confidence: 99%

Evaluation of Odor Prediction Model Performance and Variable Importance according to Various Missing Imputation Methods

et al. 2022

View full text Add to dashboard Cite

The aim of this study is to ascertain the most suitable model for predicting complex odors using odor substance data that has a small number of data and a large number of missing data. First, we compared the data removal and imputation methods, and the method of imputing missing data was found to be more effective. Then, in order to recommend a suitable model, we created a total of 126 models (missing imputation: single imputation, multiple imputations, K-nearest neighbor imputation; data preprocessing: standardization, principal component analysis, partial least square; and predictive method: multiple regression, machine learning, deep learning) and compared them using R2 and mean absolute error (MAE) values. Finally, we investigated variable importance using the best prediction model. The results identified the best model as a combination of multivariate imputation using Bayesian ridge as the missing imputation method, standardization for data preprocessing, and an extremely randomized tree as the predictive method. Among the odor compounds, Methyl mercaptan, acetic acid, and dimethyl sulfide were identified as the most important odor compounds in predicting complex odors.

show abstract

“…Rubin et al [17], stated that there are three mechanisms by which missing data typically occur: 1) missing completely at random (MCAR), 2) missing at random (MAR), and 3) missing not at random (MNAR). Being able to associate the missing data structure to one of these mechanisms is highly valuable because it guides the users to the best technique to properly handle the particularity of the data [18], [19], i.e., if we are able to correlate the situation at which the missing data happens or the structure of the time series including the missing data segments with one of the mentioned three missing data mechanisms. To understand the different mechanisms underlying missing data, we first need to define a mathematical reference to rely on.…”

Section: ) Statistical Mechanisms Underlying Missing Datamentioning

confidence: 99%

Long Gaps Missing IoT Sensors Time Series Data Imputation: A Bayesian Gaussian Approach

et al. 2022

View full text Add to dashboard Cite

Missing sensor data is a common problem associated with the Internet of Things (IoT) ecosystems, which affect the accuracy of the associated services such as adequate medical intervention for older adults living at home. This problem is caused by many factors, power down is one of them, communication failure and sensor failure are another two reasons. Multiple missing data imputation methods have been developed to solve this issue. However, irregular temporal missing data locations is challenging to handle, due to lack of knowledge of their occurrence probability and their random temporal location. In this paper, we propose a Bayesian Gaussian Process based imputation technique that accounts for temporal forcing to fill in the missing sensor data. Our approach; Bayesian Gaussian Process (BGaP); can impute missing data efficiently at any missing rate and for any temporal location using prior knowledge gathered on past observations. We illustrated how our approach performs using real data collected from sensors deployed in the residence of 10 older adults over a two-year period. Using our novel approach, we were able to impute all the missing data which allowed us to observe long-term behavior changes that we would not have been able to observe otherwise.INDEX TERMS Missing Sensor Data, Missing Data Imputation, Bayesian Gaussian Process, Long-term Older adults Behavior Monitoring.

show abstract

A Review of Missing Sensor Data Imputation Methods

Cited by 8 publications

References 17 publications

Missing-value Imputation of Environment Sensors Using Multilayer Stacking with Scoring Method

Missing-value Imputation of Environment Sensors Using Multilayer Stacking with Scoring Method

Evaluation of Odor Prediction Model Performance and Variable Importance according to Various Missing Imputation Methods

Long Gaps Missing IoT Sensors Time Series Data Imputation: A Bayesian Gaussian Approach

Contact Info

Product

Resources

About