Observing the unobservable : distributed online outlier detection in wireless sensor networks

Abstract

This research was conducted within the EU projects e-SENSE and SENSEI. Copyright c 2010 by Yang Zhang, Enschede, The Netherlands. All rights reserved. No part of this book may be reproduced or transmitted, in any form or by any means, electronic or mechanical, including photocopying, microfilming, and recording, or by any information storage or retrieval system, without the prior written permission of the author.Printed by Wöhrmann Print Service. AbstractThe generation of wireless sensor networks (WSNs) makes human beings observe and reason about the physical environment better, easier, and faster. The wireless sensor nodes equipped with sensing, processing, wireless communication and actuation capabilities can be densely deployed in a wide geographical area and measure various parameters continuously from the physical world. Compared with traditional environmental sensing technologies, such densely deployed WSNs enable collection of fine-grained high spatial and temporal resolution data with less installation, maintenance, and operation costs.However, raw sensor observations often have low data quality and reliability due to both internal and external factors including low quality of cheap sensors, dynamicity of network conditions, and harshness of the deployment environment. Use of low quality sensor data in any data analysis and decision making process will not only negatively impact analysis results and decisions made but also waste huge amount of valuable and limited network resources such as energy, as many incorrect values are transmitted. Low quality sensor data also prevents WSNs to fulfill their promises in terms of reliable real-time situation-awareness, as the low quality sensor data may generate large number of false alarms.Motivated by the need to improve quality of data analysis and decision making, enhance efficiency of using WSNs resources by preventing unnecessary transmission of erroneous sensor observations, and increase effectiveness of monitoring and situation-awareness capabilities of the WSNs, in this thesis we focus on online identification of outliers whenever and wherever they occur. Outliers in WSNs are those observations that represent erroneous values (errors) or indicate particular phenomenal changes (events). Our outlier detection techniques, which are based on distributed in-network data processing, identify sensor observations that do not conform to normal behavior of sensor data without using a pre-defined threshold or triggering conditions.Our main research objective is to design and implement effective and efficient outlier detection techniques for WSNs to identify outliers in an online and disv tributed manner and distinguish between errors and events with high accuracy and low false alarm, while maintaining the communication, computation and memory complexity low. Main contributions of this thesis can be summarized as: 3. Statistical-Based outlier detection techniques for WSNs. We take two approaches in designing our outlier detection techniques. One approach originates fr...

One-class support vector machines: analysis of outlier detection for wireless sensor networks in harsh environments

“…The purpose of an outlier and event detection technique for WSNs deployed in harsh environments should be three-fold: outlier detection, event detection and event identification Zhang et al 2010;Zhang 2010).…”

Wireless Sensor Networks and Fusion of Contextual Information for Weather Outlier Detection

“…An instance sensor data is labelled as an outlier since the probability that the data could be generated by the model is very low (Chandola et al, 2007). These techniques assume that majority of sensor data includes normal observations (Zhang, 2010) since WSNs imperfect sensors can break the assumption especially in the presence of the faulty nodes.…”

Modeling Anomalies Prevalent in Sensor Network Deployments: A Representative Ground Truth

“…In addition, inspectors are sometimes uncertain if a suspicious reading is in fact an anomaly unless they are experts in their domain. Although uncommon, algorithm designers may use metrics such as distance, density, or a running average to label anomalies in the original data [6]. However, these labeling techniques are themselves anomaly detection mechanisms that have their own deficiencies.…”