An Adaptive Outlier Detection and Processing Approach Towards Time Series Sensor Data

Zhang, Minghu; Li, Xin; Wang, Lili

doi:10.1109/access.2019.2957602

Cited by 38 publications

(22 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the first step of the data-driven approach for anomaly detection, the MF algorithm is used to preprocess anomalies in the data stream [ 21 , 22 , 23 ]. First, if input data with anomalies are imported into a prediction model, inaccurate predictions will be generated.…”

Section: Framework and Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Data-Driven Anomaly Detection Approach for Time-Series Streaming Data

Zhang

Guo

et al. 2020

Sensors

Self Cite

View full text Add to dashboard Cite

Recently, wireless sensor networks (WSNs) have been extensively deployed to monitor environments. Sensor nodes are susceptible to fault generation due to hardware and software failures in harsh environments. Anomaly detection for the time-series streaming data of sensor nodes is a challenging but critical fault diagnosis task, particularly in large-scale WSNs. The data-driven approach is becoming essential for the goal of improving the reliability and stability of WSNs. We propose a data-driven anomaly detection approach in this paper, named median filter (MF)-stacked long short-term memory-exponentially weighted moving average (LSTM-EWMA), for time-series status data, including the operating voltage and panel temperature recorded by a sensor node deployed in the field. These status data can be used to diagnose device anomalies. First, a median filter (MF) is introduced as a preprocessor to preprocess obvious anomalies in input data. Then, stacked long short-term memory (LSTM) is employed for prediction. Finally, the exponentially weighted moving average (EWMA) control chart is employed as a detector for recognizing anomalies. We evaluate the proposed approach for the panel temperature and operating voltage of time-series streaming data recorded by wireless node devices deployed in harsh field conditions for environmental monitoring. Extensive experiments were conducted on real time-series status data. The results demonstrate that compared to other approaches, the MF-stacked LSTM-EWMA approach can significantly improve the detection rate (DR) and false rate (FR). The average DR and FR values with the proposed approach are 95.46% and 4.42%, respectively. MF-stacked LSTM-EWMA anomaly detection also achieves a better F2 score than that achieved by other methods. The proposed approach provides valuable insights for anomaly detection in WSNs by detecting anomalies in the time-series status data recorded by wireless sensor nodes.

show abstract

Section: Framework and Methodsmentioning

confidence: 99%

“…We classify an observation value as anomalous or non-anomalous by calculating the residuals between the predicted value and the actual observation at a specific time through the EWMA control chart. We set

based on a confidence level of 90% [ 23 ].…”

Section: Framework and Methodsmentioning

confidence: 99%

Data-Driven Anomaly Detection Approach for Time-Series Streaming Data

Zhang

Guo

et al. 2020

Sensors

Self Cite

View full text Add to dashboard Cite

show abstract

“…The problem of time series prediction is considered as the most important problem in machine learning, with a large number of practical applications such as stock price trend prediction [ 39 ], housing price prediction [ 40 ], sensor data analysis [ 41 ], and water price prediction [ 42 ]. LSTMs are the most popular specialized model of recurrent neural networks (RNNs) for the time series forecasting problem.…”

Section: Related Workmentioning

confidence: 99%

Multiple Electric Energy Consumption Forecasting Using a Cluster-Based Strategy for Transfer Learning in Smart Building

Kieu

et al. 2020

Sensors

View full text Add to dashboard Cite

Electric energy consumption forecasting is an interesting, challenging, and important issue in energy management and equipment efficiency improvement. Existing approaches are predictive models that have the ability to predict for a specific profile, i.e., a time series of a whole building or an individual household in a smart building. In practice, there are many profiles in each smart building, which leads to time-consuming and expensive system resources. Therefore, this study develops a robust framework for the Multiple Electric Energy Consumption forecasting (MEC) of a smart building using Transfer Learning and Long Short-Term Memory (TLL), the so-called MEC-TLL framework. In this framework, we first employ a k-means clustering algorithm to cluster the daily load demand of many profiles in the training set. In this phase, we also perform Silhouette analysis to specify the optimal number of clusters for the experimental datasets. Next, this study develops the MEC training algorithm, which utilizes a cluster-based strategy for transfer learning the Long Short-Term Memory models to reduce the computational time. Finally, extensive experiments are conducted to compare the computational time and different performance metrics for multiple electric energy consumption forecasting on two smart buildings in South Korea. The experimental results indicate that our proposed approach is capable of economical overheads while achieving superior performances. Therefore, the proposed approach can be applied effectively for intelligent energy management in smart buildings.

show abstract

“…• The × 100% of the sample that doesn't follow the distribution of the rest of the sample lies outside the region [L, U] where L = µ + Zl×σ, U = µ + Zu×σ, Zu > Zl and Zl, Zu are known. where the values of µU, µL, σU, and σL can be found using the equations (4), (5), (6), (7), and (8) in subsection III-A. Note: the equations in subsection III-A require the values of M * and MADM * of the population.…”

Section: B Estimation Proceduresmentioning

confidence: 99%

“…Robust estimators are the current tools in modern statistics for estimating the parameters of a distribution in the presence of errors. [1], [2], [3], [4], [5], [6] and [7] presented methods for applying robust estimators in statistical inference and its applications. In particular, [3] developed a modified version of the Z-score method that depends on robust estimators for outlier detection in normally distributed samples.…”

Section: Introductionmentioning

confidence: 99%

Asymptotically Unbiased Estimation of Mean and Standard Deviation in the Presence of Outlying Errors

Joudah

Abbas

2020

IEEE Access

View full text Add to dashboard Cite

In this work, we develop a new method for estimating the mean and standard deviation of normally distributed populations when errors exist in the sample. Our estimators are asymptotically unbiased if all errors are outlying errors (i.e. all errors lie outside a region defined by the parameters of normal distribution). However, the proposed method requires having an upper bound for the percentage of errors in the sample. The proposed method is compared with the already existing and commonly used estimators in terms of bias, mean square error, and Pitman closeness criterion. The proposed method is found to be superior to the others when the sample size is not too small. A simulation study is designed to test the performance of our estimators when most but not all errors are outlying errors. The findings of the simulation study also indicate the superiority of the proposed method when the sample size is moderately large. As an application, we use our method in Phase I of designing a control chart to improve its performance. We apply the method on a dataset where the robustness of our proposed method is tested (and compared with the other estimators) against the presence of outlying errors in Phase I data. In the findings of the application, we notice that our estimators were the only ones that identified out of control data points in Phase II when Phase I samples are contaminated with the outliers.

show abstract

An Adaptive Outlier Detection and Processing Approach Towards Time Series Sensor Data

Cited by 38 publications

References 50 publications

Data-Driven Anomaly Detection Approach for Time-Series Streaming Data

Data-Driven Anomaly Detection Approach for Time-Series Streaming Data

Multiple Electric Energy Consumption Forecasting Using a Cluster-Based Strategy for Transfer Learning in Smart Building

Asymptotically Unbiased Estimation of Mean and Standard Deviation in the Presence of Outlying Errors

Contact Info

Product

Resources

About