Handling bad or missing smart meter data through advanced data imputation

Peppanen, Jouni; Zhang, Xiaochen; Grijalva, Santiago; Reno, Matthew J.

doi:10.1109/isgt.2016.7781213

Cited by 63 publications

(56 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition, as they also refer to some collected data after the missing data, it is difficult to apply this approach to instant imputation for real-time applications in an energy system. Thus, the previously proposed work in [12] still has shortcomings that should be addressed, and this is the inspiration for the present work.…”

Section: Introductionmentioning

confidence: 91%

“…We evaluate the proposed LAI and eLAI by comparing them to existing methods, LI and OWA [12], which are commonly used or suggested for missing power data imputation. In addition to two comparison methods, the probabilistic principle component analysis (PPCA)-based imputation method, which had been proven to be one of the most effective imputing methods in traffic data [20,21], is also compared.…”

Section: Performance Evaluationmentioning

confidence: 99%

“…Nevertheless, most previous studies related to power data have concentrated on designing a demand forecasting mechanism or robust methods to missing data [8][9][10]. The limitations of these previous research efforts have recently motivated studies focused on handling missing data in energy systems [11,12].…”

Section: Introductionmentioning

confidence: 99%

“…Accordingly, method selection criteria based only on fixed missing interval lengths without considering each load pattern cannot optimally be adapted to unpredictable variations in the energy system. Recently, to resolve this issue, Peppanen et al in [12] suggested an optimally-weighted average (OWA) data imputation method. The work in [12] imputes missing data with a weighted sum of LI and their own HA.…”

Section: Introductionmentioning

confidence: 99%

“…Recently, to resolve this issue, Peppanen et al in [12] suggested an optimally-weighted average (OWA) data imputation method. The work in [12] imputes missing data with a weighted sum of LI and their own HA. In addition, this method employs an optimal weight factor to enhance the imputation performance.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Learning-Based Adaptive Imputation Methodwith kNN Algorithm for Missing Power Data

et al. 2017

View full text Add to dashboard Cite

This paper proposes a learning-based adaptive imputation method (LAI) for imputing missing power data in an energy system. This method estimates the missing power data by using the pattern that appears in the collected data. Here, in order to capture the patterns from past power data, we newly model a feature vector by using past data and its variations. The proposed LAI then learns the optimal length of the feature vector and the optimal historical length, which are significant hyper parameters of the proposed method, by utilizing intentional missing data. Based on a weighted distance between feature vectors representing a missing situation and past situation, missing power data are estimated by referring to the k most similar past situations in the optimal historical length. We further extend the proposed LAI to alleviate the effect of unexpected variation in power data and refer to this new approach as the extended LAI method (eLAI). The eLAI selects a method between linear interpolation (LI) and the proposed LAI to improve accuracy under unexpected variations. Finally, from a simulation under various energy consumption profiles, we verify that the proposed eLAI achieves about a 74% reduction of the average imputation error in an energy system, compared to the existing imputation methods.

show abstract

Section: Introductionmentioning

confidence: 91%

Section: Performance Evaluationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Learning-Based Adaptive Imputation Methodwith kNN Algorithm for Missing Power Data

et al. 2017

View full text Add to dashboard Cite

show abstract

Data processing and quality verification for improved photovoltaic performance and reliability analytics

Livera

Theristis

Koumpli³

et al. 2020

Progress in Photovoltaics

View full text Add to dashboard Cite

Data integrity is crucial for the performance and reliability analysis of photovoltaic (PV) systems, since actual in‐field measurements commonly exhibit invalid data caused by outages and component failures. The scope of this paper is to present a complete methodology for PV data processing and quality verification in order to ensure improved PV performance and reliability analyses. Data quality routines (DQRs) were developed to ensure data fidelity by detecting and reconstructing invalid data through a sequence of filtering stages and inference techniques. The obtained results verified that PV performance and reliability analyses are sensitive to the fidelity of data and, therefore, time series reconstruction should be handled appropriately. To mitigate the bias effects of 10% or less invalid data, the listwise deletion technique provided accurate results for performance analytics (exhibited a maximum absolute percentage error of 0.92%). When missing data rates exceed 10%, data inference techniques yield more accurate results. The evaluation of missing power measurements demonstrated that time series reconstruction by applying the Sandia PV Array Performance Model yielded the lowest error among the investigated data inference techniques for PV performance analysis, with an absolute percentage error less than 0.71%, even at 40% missing data rate levels. The verification of the routines was performed on historical datasets from two different locations (desert and steppe climates). The proposed methodology provides a set of standardized analytical procedures to ensure the validity of performance and reliability evaluations that are performed over the lifetime of PV systems.

show abstract

Data Quality Management Framework for Smart Grid Systems

Chren

Rossi

et al. 2019

Business Information Systems

View full text Add to dashboard Cite

Handling bad or missing smart meter data through advanced data imputation

Cited by 63 publications

References 9 publications

Learning-Based Adaptive Imputation Methodwith kNN Algorithm for Missing Power Data

Learning-Based Adaptive Imputation Methodwith kNN Algorithm for Missing Power Data

Data processing and quality verification for improved photovoltaic performance and reliability analytics

Data Quality Management Framework for Smart Grid Systems

Contact Info

Product

Resources

About