A Comparison of Methods for Missing Data Treatment in Building Sensor Data

Pazhoohesh, Mehdi; Pourmirza, Zoya; Walker, Sara

doi:10.1109/sege.2019.8859963

Cited by 15 publications

(15 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Data preprocessing is a fundamental step in ML and DL models to improve the model's accuracy and performance [12]. In the scope of this study, five data preprocessing steps were used.…”

Section: B Data Preprocessingmentioning

confidence: 99%

Transfer Learning Approach for Occupancy Prediction in Smart Buildings

Khalil

McGough

Pourmirza

et al. 2021

2021 12th International Renewable Engineering Conference (IREC)

Self Cite

View full text Add to dashboard Cite

Accurate occupancy prediction in smart buildings is a key element to reduce building energy consumption and control HVAC systems (Heating -Ventilation and-Air Conditioning) efficiently, resulting in an increment of human comfort. This work focuses on the problem of occupancy prediction modelling (occupied / unoccupied) in smart buildings using environmental sensor data. A novel transfer learning approach was used to enhance occupancy prediction accuracy when the amounts of historical training data are limited. The proposed approach and models are applied to a case study of three office rooms in an educational building. The data sets used in this work are actual data collected from the Urban Sciences Building (USB) in Newcastle University. The results of the proposed transfer learning approach have been compared with the models from Support Vector Machine and Random Forest algorithms. The final results demonstrate that the most accurate model in this study to predict occupancy status was produced by stacked Long-Short-Term-Memory with a transfer learning framework.

show abstract

“…Data preprocessing is a fundamental step in ML and DL models to improve the model's accuracy and performance [12]. In the scope of this study, five data preprocessing steps were used.…”

Section: B Data Preprocessingmentioning

confidence: 99%

Transfer Learning Approach for Occupancy Prediction in Smart Buildings

Khalil

McGough

Pourmirza

et al. 2021

2021 12th International Renewable Engineering Conference (IREC)

Self Cite

View full text Add to dashboard Cite

show abstract

“…During the first stage, three data reduced versions of the original dataset were prepared by methodically removing 10%, F I G U R E 2 Flow chart of a simple rule-based EMS of an ESS providing peak shaving service 20% and 30% of the data following the missing completely at random (MCAR) mechanism [22]. The logic behind the data removal process was inspired by studies [22,24] that classed 10% reduction as small, 20% as medium and 30% reduction as large amounts of missing data reflecting real life scenarios.…”

Section: Data Framingmentioning

confidence: 99%

Investigating the impact of missing data imputation techniques on battery energy management system

et al. 2021

Self Cite

View full text Add to dashboard Cite

Effective control of energy storage system (ESS), supplying an ancillary service to a grid, requires effective and critical calculation of state-of-charge (SoC). Charging and discharging values from battery operations are essential in calculating the efficiency and performance of a storage system. This information can also be a key to understand and forecast peak demand performance. Missing data is a real problem in any operations system, and it appears to be more common within powers systems due to sensor and/or network malfunctioning problems. Missing data imputation techniques have evolved in power systems research using smart meter data, but little research has gone into understanding how missing data can be best handled within storage management systems. This paper builds on a year's worth of charging and discharging data collected from a real 6MW/10MWh lithium-ion storage battery deployed on the distribution network at Leighton Buzzard, UK. Using R Studio version (1.3.959-1) open-source software, eight selected imputation techniques were applied in identifying the best suited technique in replacing various missing data amounts and patterns. Findings from the study open up avenues for discussion and debate in identifying an appropriate imputation technique within the storage management context. The study also provides a pioneering lead in understanding the importance of decomposition in evaluating the right imputation technique. This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited.

show abstract

“…A non-parametric approach used to impute missing data by averaging its neighbouring observed data (9).…”

Section: K-nearest Neighbour (Knn)mentioning

confidence: 99%

Comparison of Imputation Methods on Retrospective Breast Cancer Data in Tanzania: A Case Study of Muhimbili and Ocean Road Hospitals

Abassi

Msengwa

Akarro

2021

Preprint

View full text Add to dashboard Cite

Background Clinical data are at risk of having missing or incomplete values for several reasons including patients’ failure to attend clinical measurements, wrong interpretations of measurements, and measurement recorder’s defects. Missing data can significantly affect the analysis and results might be doubtful due to bias caused by omission of missed observation during statistical analysis especially if a dataset is considerably small. The objective of this study is to compare several imputation methods in terms of efficiency in filling-in the missing data so as to increase the prediction and classification accuracy in breast cancer dataset. Methods Five imputation methods namely series mean, k-nearest neighbour, hot deck, predictive mean matching, and multiple imputations were applied to replace the missing values to the real breast cancer dataset. The efficiency of imputation methods was compared by using the Root Mean Square Errors and Mean Absolute Errors to obtain a suitable complete dataset. Binary logistic regression and linear discrimination classifiers were applied to the imputed dataset to compare their efficacy on classification and discrimination. Results The evaluation of imputation methods revealed that the predictive mean matching method was better off compared to other imputation methods. In addition, the binary logistic regression and linear discriminant analyses yield almost similar values on overall classification rates, sensitivity and specificity. Conclusion The predictive mean matching imputation showed higher accuracy in estimating and replacing missing/incomplete data values in a real breast cancer dataset under the study. It is a more effective and good method to handle missing data in this scenario. We recommend to replace missing data by using predictive mean matching since it is a plausible approach toward multiple imputations for numerical variables, as it improves estimation and prediction accuracy over the use complete-case analysis especially when percentage of missing data is not very small.

show abstract

A Comparison of Methods for Missing Data Treatment in Building Sensor Data

Cited by 15 publications

References 14 publications

Transfer Learning Approach for Occupancy Prediction in Smart Buildings

Transfer Learning Approach for Occupancy Prediction in Smart Buildings

Investigating the impact of missing data imputation techniques on battery energy management system

Comparison of Imputation Methods on Retrospective Breast Cancer Data in Tanzania: A Case Study of Muhimbili and Ocean Road Hospitals

Contact Info

Product

Resources

About