Incremental Principal Component Analysis Based Outlier Detection Methods for Spatiotemporal Data Streams

Bhushan, Alka; Sharker, Monir H.; Karimi, Hassan A.

doi:10.5194/isprsannals-ii-4-w2-67-2015

Cited by 6 publications

(2 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The essay addresses the challenge of outliers in spatiotemporal data streams from geographically dispersed sensor networks, which can distort future analyses. The study proposes two novel IPCA-based outlier detection methods, compares them with existing techniques, and provides insights into IPCA's applicability for real-time applications such as image analysis, pattern recognition, and credit card fraud detection [4,28]. This paper focuses on the air quality index (AQI), a numerical measure affected by human activities, industrial operations, and weather conditions, and uses regression models to estimate AQI based on data from a monitoring station in Chennai, India.…”

Section: Literature Reviewmentioning

confidence: 99%

Enhancing Outlier Detection in Air Quality Index Data Using a Stacked Machine Learning Model

Diallo,

Nderu,

Malenje

et al. 2023

Preprint

View full text Add to dashboard Cite

Air quality is an important part of environmental health, having serious consequences for human health and well-being. The Air Quality Index (AQI) is a frequently used metric for assessing air quality in various areas and at different times. However, AQI data, like many other types of environmental data, can contain outliers - data points that deviate significantly from other observations, indicating exceptionally good or poor air quality, a critical step in identifying and understanding extreme pollution episodes that can have serious environmental and public health consequences. These outliers can be caused by a variety of variables, including measurement mistakes, odd meteorological circumstances, and pollution occurrences. While outliers can occasionally give useful information about these unusual conditions, they can also skew studies and models if they are not adequately accounted for. This paper describes a hybrid method for detecting outliers in data, AQI data are used in this study. The model uses a stacked machine learning model that incorporates K-means clustering, Random Forest (RF), and Gradient Boosting Classifier (GBC). K-means is used for initial categorization, followed by RF model training, and ultimately, the RF output is used as input for the GBC to generate the final classification. The performance of this stacked machine learning model is examined and compared to single models using the Accuracy measure. The findings show that the suggested technique is efficient, with an accuracy of 0.99, showing its potential for effective outlier detection in data.

show abstract

Section: Literature Reviewmentioning

confidence: 99%

Enhancing Outlier Detection in Air Quality Index Data Using a Stacked Machine Learning Model

Diallo,

Nderu,

Malenje

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…to ignore the old data pattern and to capture the recent pattern for spatio-temporal streaming data analysis [29]. For detecting rare anomalies from a traffic management platform, the low rank recovery problem was considered, and robust PCA with regularized reconstruction showed good performance [30].…”

Section: Related Workmentioning

confidence: 99%

Unsupervised Abnormal Sensor Signal Detection With Channelwise Reconstruction Errors

Kwak

Kim

2021

IEEE Access

View full text Add to dashboard Cite

Detecting an anomaly in multichannel signal data is a challenging task in various domains. It should take into account the cross-channel relationship and temporal relationship within each channel. Moreover, the signal data is high-dimensional and making it difficult to gather sufficient abnormal labels. Consequently, unsupervised reconstruction-based anomaly detection methods have been applied successfully in many studies. However, they lose valuable channel information inherent in the reconstruction errors by merely averaging the errors for both the channel and time, then consider the average value as an anomaly score. In this study, we propose a method to explicitly employ channelwise reconstruction errors as a feature to detect abnormal signals. After a convolutional autoencoder produces the channelwise reconstruction errors, a machine learning anomaly detection model aggregates the errors as an anomaly score. To demonstrate the effectiveness and applicability of the proposed model, we conduct experiments using simulated data and real-world automobile data. The results show that the proposed method remarkably enhances the detectability compared to the simple average of the reconstruction errors. The reconstruction errors of abnormal and normal channels are shown to be different; therefore, it can be considered as an appropriate feature for anomaly detection. The best performance is obtained by using local outlier factors in the following anomaly detection model.

show abstract

Enhancing outlier detection in air quality index data using a stacked machine learning model

Diallo,

Nderu,

Malenje

et al. 2024

Engineering Reports

View full text Add to dashboard Cite

The air quality index (AQI) is a commonly employed metric for evaluating air quality across diverse locations and temporal spans. Similar to other environmental datasets, AQI data can exhibit outliers data points markedly divergent from the norm, signifying instances of exceptionally favorable or adverse air quality. This becomes crucial in identifying and comprehending severe pollution episodes with far‐reaching environmental and public health implications. This study utilizes air quality data from January 1, 2014, to January 31, 2021, collected at daily intervals in Shanghai City, China, as the experimental dataset. The dataset includes daily AQI measurements, along with six pollutant concentrations: particulate matter (PM2.5 and PM10), sulfur dioxide (SO2), nitrogen dioxide (NO2), ozone (O3), and carbon monoxide (CO). Each pollutant's concentration is measured in micrograms per cubic meter (g/m). The dataset is then preprocessed by cleaning and normalizing it before using K‐means clustering to discover different patterns. A stacked ensemble machine learning model that incorporates K‐means clustering, random forest (RF) and gradient boosting classifier (GBC) is developed and compared to decision tree, support vector machine, K‐nearest neighbor and Naive Bayes algorithms to evaluate its performance in identifying outliers using accuracy, precision, recall, and F1‐score. The stacked model outperformed all other established models based on the accuracy, precision, recall, and F1‐score of 0.99, 0.99, 0.97, and 0.99, respectively.

show abstract

Incremental Principal Component Analysis Based Outlier Detection Methods for Spatiotemporal Data Streams

Cited by 6 publications

References 24 publications

Enhancing Outlier Detection in Air Quality Index Data Using a Stacked Machine Learning Model

Enhancing Outlier Detection in Air Quality Index Data Using a Stacked Machine Learning Model

Unsupervised Abnormal Sensor Signal Detection With Channelwise Reconstruction Errors

Enhancing outlier detection in air quality index data using a stacked machine learning model

Contact Info

Product

Resources

About