Time series data cleaning

Zhang, Aoqian; Song, Shaoxu; Wang, Jianmin; Yu, Philip S.

doi:10.14778/3115404.3115410

Cited by 99 publications

(21 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…To remove errors, data cleaning is required: defined as the "process of detecting, diagnosing, and editing faulty data" [6]. Ideally, data cleaning methods should prioritise data repair over data removal [7] and use computer programs to improve reproducibility [8].…”

Section: Introductionmentioning

confidence: 99%

Is it time to stop sweeping data cleaning under the carpet? A novel algorithm for outlier management in growth data

et al. 2020

View full text Add to dashboard Cite

All data are prone to error and require data cleaning prior to analysis. An important example is longitudinal growth data, for which there are no universally agreed standard methods for identifying and removing implausible values and many existing methods have limitations that restrict their usage across different domains. A decision-making algorithm that modified or deleted growth measurements based on a combination of pre-defined cutoffs and logic rules was designed. Five data cleaning methods for growth were tested with and without the addition of the algorithm and applied to five different longitudinal growth datasets: four uncleaned canine weight or height datasets and one pre-cleaned human weight dataset with randomly simulated errors. Prior to the addition of the algorithm, data cleaning based on non-linear mixed effects models was the most effective in all datasets and had on average a minimum of 26.00% higher sensitivity and 0.12% higher specificity than other methods. Data cleaning methods using the algorithm had improved data preservation and were capable of correcting simulated errors according to the gold standard; returning a value to its original state prior to error simulation. The algorithm improved the performance of all data cleaning methods and increased the average sensitivity and specificity of the non-linear mixed effects model method by 7.68% and 0.42% respectively. Using non-linear mixed effects models combined with the algorithm to clean data allows individual growth trajectories to vary from the population by using repeated longitudinal measurements, identifies consecutive errors or those within the first data entry, avoids the requirement for a minimum number of data entries, preserves data where possible by correcting errors rather than deleting them and removes duplications intelligently. This algorithm is broadly applicable to data cleaning anthropometric data in different mammalian species and could be adapted for use in a range of other domains.

show abstract

Section: Introductionmentioning

confidence: 99%

Is it time to stop sweeping data cleaning under the carpet? A novel algorithm for outlier management in growth data

et al. 2020

View full text Add to dashboard Cite

show abstract

“…(3) Consistency: The evaluation needs to assess whether the data items in a data set contradict with each other. It is often known as anomaly [46] or outlier [16] in time series data. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page.…”

Section: Introductionmentioning

confidence: 99%

IoT Data Quality

Song

Zhang

2020

Proceedings of the 29th ACM International Conference on Information &Amp; Knowledge Management

Self Cite

View full text Add to dashboard Cite

Data quality issues have been widely recognized in IoT data, and prevent the downstream applications. In this tutorial, we review the state-of-the-art techniques for IoT data quality management. In particular, we discuss how the dedicated approaches improve various data quality dimensions, including validity, completeness and consistency. Among others, we further highlight the recent advances by deep learning techniques for IoT data quality. Finally, we indicate the open problems in IoT data quality management, such as benchmark or interpretation of data quality issues.

show abstract

“…Most of the existing faulty data detection techniques are designed for offline applications where the entire data set is available. Moreover, the methods which are proposed for online data cleaning are mainly effective for repairing spike errors [9].…”

Section: Introductionmentioning

confidence: 99%

“…Smoothing techniques are widely used for online data cleaning to eliminate noisy data [9]. Moving average (MA) [10], weighted moving average (WMA) [11], exponentially weighted moving average (EWMA) [11], and sliding window bottom-up (SWAB) [12] are examples of smoothing methods.…”

Section: Introductionmentioning

confidence: 99%

“…The detection and replacement of faulty data only based on the change in a recent sensor reading is a useful method when dealing with spike errors. However, in the case of sequences of faulty data (e.g., several hours), this method fails [9]. In this case, the replaced values for the faulty data gradually increases/decreases and eventually causes a big offset error after several inputs.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Stream Data Cleaning for Dynamic Line Rating Application

et al. 2018

View full text Add to dashboard Cite

The maximum current that an overhead transmission line can continuously carry depends on external weather conditions, most commonly obtained from real-time streaming weather sensors. The accuracy of the sensor data is very important in order to avoid problems such as overheating. Furthermore, faulty sensor readings may cause operators to limit or even stop the energy production from renewable sources in radial networks. This paper presents a method for detecting and replacing sequences of consecutive faulty data originating from streaming weather sensors. The method is based on a combination of (a) a set of constraints obtained from derivatives in consecutive data, and (b) association rules that are automatically generated from historical data. In smart grids, a large amount of historical data from different weather stations are available but rarely used. In this work, we show that mining and analyzing this historical data provides valuable information that can be used for detecting and replacing faulty sensor readings. We compare the result of the proposed method against the exponentially weighted moving average and vector autoregression models. Experiments on data sets with real and synthetic errors demonstrate the good performance of the proposed method for monitoring weather sensors.

show abstract

Time series data cleaning

Cited by 99 publications

References 20 publications

Is it time to stop sweeping data cleaning under the carpet? A novel algorithm for outlier management in growth data

Is it time to stop sweeping data cleaning under the carpet? A novel algorithm for outlier management in growth data

IoT Data Quality

Stream Data Cleaning for Dynamic Line Rating Application

Contact Info

Product

Resources

About