A Feature‐Based Procedure for Detecting Technical Outliers in Water‐Quality Data From In Situ Sensors

Abstract

Outliers due to technical errors in water‐quality data from in situ sensors can reduce data quality and have a direct impact on inference drawn from subsequent data analysis. However, outlier detection through manual monitoring is infeasible given the volume and velocity of data the sensors produce. Here we introduce an automated procedure, named oddwater, that provides early detection of outliers in water‐quality data from in situ sensors caused by technical issues. Our oddwater procedure is used to first identify the data features that differentiate outlying instances from typical behaviors. Then, statistical transformations are applied to make the outlying instances stand out in a transformed data space. Unsupervised outlier scoring techniques are applied to the transformed data space, and an approach based on extreme value theory is used to calculate a threshold for each potential outlier. Using two data sets obtained from in situ sensors in rivers flowing into the Great Barrier Reef lagoon, Australia, we show that oddwater successfully identifies outliers involving abrupt changes in turbidity, conductivity, and river level, including sudden spikes, sudden isolated drops, and level shifts, while maintaining very low false detection rates. We have implemented this oddwater procedure in the open source R package oddwater.

Toward automating post processing of aquatic sensor data

“…Feature based methods comprise another class of anomaly detectors commonly used for discrete data (Tan et al, 2019), which some authors have applied to environmental time series (Leigh et al, 2018;Russo et al, 2020;Talagala et al, 2019). Unlike regression methods, feature based methods do not make a prediction of the observation.…”

“…Particularly for data with temporal correlation, it is not obvious which features should be selected, and complex feature engineering may be required (Christ et al, 2018). Another challenge is selecting an appropriate data transformation, a preprocessing step (e.g., taking the first derivative of the data) to highlight outlying points (Leigh et al, 2018;Talagala et al, 2019).…”

“…Almost any feature based machine learning method may be applied to anomaly detection problems, and approaches described in the literature include principal components analysis, support vector machines (Tran et al, 2019), HDOutliers (Leigh et al, 2018), k-nearest neighbor (Russo et al, 2020;Talagala et al, 2019), clustering (Hill and Minsker, 2010), random forest (Russo et al, 2020), xgboost, and isolated forest (Smolyakov et al, 2019). The success of feature based techniques in detecting anomalies from environmental sensor data is mixed (Hill and Minsker, 2010;Leigh et al, 2018;Russo et al, 2020).…”

“…Without reusable code, the specifics of the algorithms as implemented with environmental data cannot be examined, further tested, or applied to other datasets. Recent work in outlier detection was encapsulated in an R package (Talagala et al, 2019); however, a lack of documentation made it difficult to know how to install the package and apply the methods to our datasets. Provenance of data from raw field observations to quality controlled data products is vitally important yet rarely described in sufficient detail that the process used to arrive at final data products could be repeated (Horsburgh et al, 2015).…”

“…As a result, there is a recognized need for automating and improving quality control post processing for high frequency in situ sensor data. In this vein, automated, data driven techniques to detect anomalies in streaming sensor data are documented in the realm of research (Hill and Minsker, 2010;Leigh et al, 2018;Russo et al, 2020;Talagala et al, 2019); however, they are unfamiliar to practitioners, generally lack robust and accessible software implementations, and are not typically reproducible. Furthermore, while basic checks and more complex algorithms may identify and flag potentially erroneous values (e.g., Dereszynski and Dietterich, 2007;Hill et al, 2009;Taylor and Loescher, 2013), these procedures are generally not capable of applying corrective actions.…”